Oct 30 16:17:57.704915 ip-10-0-129-65 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 16:17:57.704928 ip-10-0-129-65 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 16:17:57.704935 ip-10-0-129-65 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 16:17:57.705193 ip-10-0-129-65 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 16:18:09.047164 ip-10-0-129-65 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 16:18:09.047182 ip-10-0-129-65 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 76abe132bc1642d785d936ba7b516edd -- Oct 30 16:20:13.592088 ip-10-0-129-65 systemd[1]: Starting Kubernetes Kubelet... Oct 30 16:20:14.022593 ip-10-0-129-65 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:20:14.022593 ip-10-0-129-65 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 16:20:14.022593 ip-10-0-129-65 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:20:14.022593 ip-10-0-129-65 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:20:14.029490 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029402 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:14.029490 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029485 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:14.029490 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029489 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:14.029490 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029492 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:14.029490 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029495 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:14.029490 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029498 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:14.029490 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029501 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029503 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029506 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029509 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029511 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029514 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029517 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029519 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029522 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029527 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029531 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029534 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029536 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029540 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029542 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029545 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029548 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029551 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029553 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:14.029724 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029563 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029566 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029568 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029572 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029575 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029577 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029580 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029583 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029585 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029588 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029591 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029593 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029596 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029598 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029601 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029605 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029607 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029610 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029612 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029615 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:14.030185 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029618 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029620 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029623 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029625 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029628 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029631 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029636 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029639 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029642 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029644 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029646 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029649 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029652 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029654 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029657 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029659 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029662 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029664 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029668 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:14.030655 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029670 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029673 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029675 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029678 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029680 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029683 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029685 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029688 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029691 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029694 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029696 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029699 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029701 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029704 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029706 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029709 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029712 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029715 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029717 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:14.031115 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029721 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029724 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029726 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029729 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029731 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029734 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029737 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029739 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029742 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029744 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029747 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029749 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029752 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029754 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029757 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029760 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029778 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029781 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029784 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029786 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:14.031588 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029789 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029793 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.029796 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030573 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030580 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030583 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030585 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030588 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030591 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030594 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030596 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030599 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030601 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030604 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030607 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030609 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030612 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030614 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030617 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:14.032175 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030620 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030622 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030625 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030627 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030630 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030632 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030635 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030637 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030640 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030642 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030645 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030648 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030650 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030654 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030657 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030660 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030662 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030665 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030667 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:14.032623 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030670 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030672 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030675 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030677 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030680 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030682 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030685 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030687 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030690 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030693 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030695 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030698 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030700 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030703 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030705 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030707 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030710 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030713 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030715 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030717 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:14.033094 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030720 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030722 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030725 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030727 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030730 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030732 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030735 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030738 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030740 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030743 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030745 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030748 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030750 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030753 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030755 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030757 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030760 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030781 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030785 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030788 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:14.033556 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030791 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030794 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030797 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030799 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030802 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030804 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030807 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030811 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030813 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030816 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030820 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030825 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030829 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030840 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030843 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030846 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030848 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030851 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030855 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:14.034041 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030857 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030860 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030864 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030867 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030870 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030873 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030875 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030878 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030881 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030883 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030886 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.030888 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031380 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031384 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031388 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031391 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031393 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031397 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031400 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031402 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:14.034484 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031405 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031407 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031410 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031412 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031415 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031418 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031420 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031422 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031425 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031427 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031430 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031433 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031436 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031438 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031441 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031443 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031446 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031449 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031451 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031453 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:14.034969 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031456 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031459 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031461 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031463 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031466 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031468 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031470 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031474 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031476 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031479 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031481 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031484 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031486 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031488 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031491 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031493 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031495 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031506 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031508 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:14.035440 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031511 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031513 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031516 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031518 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031523 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031526 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031529 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031531 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031534 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031536 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031538 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031541 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031543 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031546 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031548 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031551 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031553 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031556 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031558 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031561 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:14.035913 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031564 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031569 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031572 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031576 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031579 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031581 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031584 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031587 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031589 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031592 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031594 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031597 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031599 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031602 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031604 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031607 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031610 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031613 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031615 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031618 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:14.036414 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031620 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031623 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031625 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031628 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031630 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031632 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031635 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031637 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031641 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031645 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031648 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031651 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031653 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031657 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031661 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031664 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031667 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031670 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.031672 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:14.036890 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033798 2566 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033809 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033815 2566 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033820 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033825 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033828 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033832 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033837 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033841 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033845 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033848 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033852 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033855 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033858 2566 flags.go:64] FLAG: --cgroup-root="" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033861 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033864 2566 flags.go:64] FLAG: --client-ca-file="" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033867 2566 flags.go:64] FLAG: --cloud-provider="external" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033870 2566 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033875 2566 flags.go:64] FLAG: --cluster-domain="" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033878 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033881 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033884 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033888 2566 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033897 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 16:20:14.037338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033900 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033903 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033906 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033910 2566 flags.go:64] FLAG: --contention-profiling="false" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033913 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033916 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033920 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033922 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033927 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033930 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033933 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033935 2566 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033938 2566 flags.go:64] FLAG: --enable-server="true" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033941 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033946 2566 flags.go:64] FLAG: --event-burst="100" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033949 2566 flags.go:64] FLAG: --event-qps="50" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033952 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033955 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033958 2566 flags.go:64] FLAG: --eviction-hard="" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033963 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033966 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033969 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033972 2566 flags.go:64] FLAG: --eviction-soft="" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033975 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033978 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 16:20:14.037913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033981 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033984 2566 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033987 2566 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033989 2566 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033992 2566 flags.go:64] FLAG: --feature-gates="" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033996 2566 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.033999 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034002 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034005 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034008 2566 flags.go:64] FLAG: --healthz-port="10248" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034011 2566 flags.go:64] FLAG: --help="false" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034014 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-129-65.ec2.internal" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034017 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034020 2566 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034023 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034027 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034030 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034033 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034035 2566 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034038 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034041 2566 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034044 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034047 2566 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034050 2566 flags.go:64] FLAG: --kube-reserved="" Oct 30 16:20:14.038522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034054 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034057 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034060 2566 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034063 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034065 2566 flags.go:64] FLAG: --lock-file="" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034068 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034071 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034074 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034080 2566 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034083 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034086 2566 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034089 2566 flags.go:64] FLAG: --logging-format="text" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034091 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034095 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034097 2566 flags.go:64] FLAG: --manifest-url="" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034100 2566 flags.go:64] FLAG: --manifest-url-header="" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034104 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034107 2566 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034111 2566 flags.go:64] FLAG: --max-pods="110" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034114 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034117 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034120 2566 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034123 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034126 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034129 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 16:20:14.039121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034131 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034139 2566 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034141 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034144 2566 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034147 2566 flags.go:64] FLAG: --pod-cidr="" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034150 2566 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034155 2566 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034159 2566 flags.go:64] FLAG: --pods-per-core="0" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034162 2566 flags.go:64] FLAG: --port="10250" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034165 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034168 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05b5b6dea8bd51d43" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034171 2566 flags.go:64] FLAG: --qos-reserved="" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034174 2566 flags.go:64] FLAG: --read-only-port="10255" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034177 2566 flags.go:64] FLAG: --register-node="true" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034180 2566 flags.go:64] FLAG: --register-with-taints="" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034184 2566 flags.go:64] FLAG: --registry-burst="10" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034187 2566 flags.go:64] FLAG: --registry-qps="5" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034189 2566 flags.go:64] FLAG: --reserved-cpus="" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034192 2566 flags.go:64] FLAG: --reserved-memory="" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034196 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034199 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034202 2566 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034205 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034208 2566 flags.go:64] FLAG: --runonce="false" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034211 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034214 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034216 2566 flags.go:64] FLAG: --seccomp-default="false" Oct 30 16:20:14.039747 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034219 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034222 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034226 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034229 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034232 2566 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034234 2566 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034237 2566 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034240 2566 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034243 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034246 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034248 2566 flags.go:64] FLAG: --system-cgroups="" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034253 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034258 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034261 2566 flags.go:64] FLAG: --tls-cert-file="" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034264 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034268 2566 flags.go:64] FLAG: --tls-min-version="" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034271 2566 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034274 2566 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034276 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034280 2566 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034283 2566 flags.go:64] FLAG: --v="2" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034287 2566 flags.go:64] FLAG: --version="false" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034291 2566 flags.go:64] FLAG: --vmodule="" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034295 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034298 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 16:20:14.040395 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034428 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034431 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034434 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034437 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034439 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034442 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034445 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034447 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034450 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034452 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034455 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034457 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034460 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034463 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034465 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034468 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034470 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034472 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034477 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034479 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:14.041425 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034482 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034485 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034487 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034490 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034493 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034495 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034498 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034500 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034503 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034505 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034508 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034510 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034513 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034515 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034517 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034520 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034522 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034525 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034527 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034530 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:14.042284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034532 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034534 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034537 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034540 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034542 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034544 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034547 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034549 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034551 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034554 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034559 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034562 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034564 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034567 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034570 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034572 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034575 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034577 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034580 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034582 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:14.043072 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034584 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034587 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034589 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034592 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034594 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034597 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034599 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034602 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034604 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034606 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034609 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034611 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034614 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034618 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034621 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034625 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034629 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034632 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034635 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:14.043606 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034638 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034641 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034643 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034648 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034650 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034653 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034655 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034658 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034660 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034663 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034665 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034667 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034670 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034672 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034675 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034677 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034680 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034682 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034685 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034687 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:14.044437 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034690 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:14.045340 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034692 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:14.045340 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034695 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:14.045340 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034698 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:14.045340 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034700 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:14.045340 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034703 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:14.045340 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034705 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:14.045340 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.034731 2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:20:14.045340 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034782 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:14.045340 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034789 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:14.045340 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034793 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:14.045340 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034796 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:14.045340 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034798 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:14.045340 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034801 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:14.045340 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034803 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:14.045340 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034807 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:14.045340 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034811 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034813 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034816 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034818 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034821 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034823 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034826 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034829 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034831 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034840 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034844 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034846 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034849 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034852 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034854 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034857 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034859 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034862 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034864 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034867 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:14.045891 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034869 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034872 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034874 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034877 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034880 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034882 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034885 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034896 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034899 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034901 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034904 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034911 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034915 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034917 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034920 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034922 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034925 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034927 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034930 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:14.046472 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034933 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034935 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034938 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034940 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034943 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034945 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034948 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034951 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034955 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034959 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034963 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034966 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034968 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034971 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034974 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034977 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034980 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034983 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034985 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:14.046961 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034988 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034990 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034993 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034996 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.034998 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035002 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035005 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035007 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035010 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035012 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035015 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035018 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035020 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035023 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035025 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035028 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035030 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035033 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035035 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:14.047625 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035038 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035040 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035043 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035045 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035048 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035050 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035052 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035055 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035057 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035060 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035062 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035064 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035067 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035069 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035072 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035074 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035077 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035079 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035083 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035085 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:14.048596 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.035089 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.042912 2566 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.042931 2566 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.043516 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.043527 2566 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043612 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043621 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043626 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043630 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043635 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043639 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043643 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043647 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043652 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043656 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043661 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043665 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043669 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043674 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:14.049336 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043678 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043682 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043686 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043691 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043695 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043699 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043704 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043708 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043713 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043718 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043722 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043726 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043730 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043734 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043739 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043746 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043753 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043758 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043786 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:14.049804 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043792 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043796 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043800 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043804 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043808 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043812 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043816 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043820 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043824 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043828 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043832 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043837 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043841 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043845 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043850 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043854 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043858 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043862 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043866 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:14.050253 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043870 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043874 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043878 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043882 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043886 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043891 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043895 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043901 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043907 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043913 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043919 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043923 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043927 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043932 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043936 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043940 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043944 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043948 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043953 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:14.050740 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043958 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043962 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043966 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043970 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043974 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043978 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043982 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043986 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043991 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.043995 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044000 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044004 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044008 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044012 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044016 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044020 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044024 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044028 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044032 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044037 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:14.051213 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044041 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:14.051738 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044046 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:14.051738 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044050 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:14.051738 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044055 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:14.051738 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044060 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:14.051738 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044064 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:14.051738 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044068 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:14.051738 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044072 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:14.051738 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044076 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:14.051738 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044080 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:14.051738 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044084 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:14.051738 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044089 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:14.051738 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044093 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:14.051738 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044097 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:14.051738 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044101 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:14.051738 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.044144 2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:20:14.051738 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044408 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044418 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044425 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044433 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044437 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044442 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044445 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044450 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044453 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044458 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044462 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044466 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044470 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044474 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044478 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044483 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044487 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044491 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044495 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:14.052134 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044499 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044503 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044508 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044512 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044517 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044523 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044530 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044535 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044539 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044544 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044548 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044552 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044556 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044560 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044564 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044568 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044573 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044578 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044582 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:14.052847 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044586 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044591 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044595 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044599 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044604 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044609 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044613 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044617 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044621 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044625 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044629 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044634 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044639 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044643 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044648 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044652 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044657 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044661 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044665 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044669 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:14.053346 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044673 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044677 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044682 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044686 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044690 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044694 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044698 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044702 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044706 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044711 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044715 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044720 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044724 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044728 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044733 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044737 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044741 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044745 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044750 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044755 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:14.054114 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044759 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044781 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044785 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044789 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044794 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044798 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044802 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044806 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044810 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044815 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044819 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044823 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044827 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044831 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044835 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044839 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044842 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044846 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044850 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:14.054802 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044855 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:14.055321 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044859 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:14.055321 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044862 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:14.055321 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044867 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:14.055321 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044871 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:14.055321 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044875 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:14.055321 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044879 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:14.055321 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044883 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:14.055321 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044888 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:14.055321 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.044930 2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:20:14.055321 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044981 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:14.055321 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044988 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:14.055321 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044992 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:14.055321 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.044998 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:14.055321 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045003 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:14.055321 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045008 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:14.055321 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045014 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045018 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045023 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045027 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045030 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045035 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045061 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045069 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045074 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045078 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045082 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045086 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045090 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045094 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045098 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045102 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045106 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045111 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045115 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045119 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:14.055716 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045123 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045128 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045132 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045136 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045140 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045144 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045148 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045152 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045156 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045160 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045164 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045168 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045172 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045176 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045180 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045184 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045188 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045192 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045196 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:14.056202 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045200 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045208 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045212 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045217 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045221 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045225 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045229 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045233 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045238 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045242 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045246 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045252 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045258 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045263 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045267 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045271 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045276 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045280 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045284 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:14.056618 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045288 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045292 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045296 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045300 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045304 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045308 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045313 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045317 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045321 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045325 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045329 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045333 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045336 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045340 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045347 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045354 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045358 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045363 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045367 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:14.057108 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045371 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045375 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045380 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045384 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045388 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045392 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045396 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045404 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045408 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045412 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045417 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045421 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045425 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045429 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045434 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045438 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045442 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045445 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045450 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045454 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:14.057549 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045460 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:14.058043 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045464 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:14.058043 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:14.045468 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:14.058043 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.045543 2566 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 16:20:14.058043 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.049067 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 16:20:14.058043 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.050277 2566 server.go:1014] "Starting client certificate rotation" Oct 30 16:20:14.058043 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.050381 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:20:14.058043 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.051400 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:20:14.058043 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.055016 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:20:14.058043 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.056523 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:20:14.068929 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.068914 2566 log.go:25] "Validated CRI v1 runtime API" Oct 30 16:20:14.072643 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.072629 2566 log.go:25] "Validated CRI v1 image API" Oct 30 16:20:14.073456 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.073428 2566 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 16:20:14.079179 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.079158 2566 fs.go:135] Filesystem UUIDs: map[4ce666f9-2d08-4baf-9d89-b699101e4f8e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d486a53b-5a12-4a19-b01e-8659668cb2ef:/dev/nvme0n1p3] Oct 30 16:20:14.079250 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.079177 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 16:20:14.083359 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.083241 2566 manager.go:217] Machine: {Timestamp:2026-10-30 16:20:14.082715147 +0000 UTC m=+0.365245035 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099440 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec274d11ebf734d657f9c790077b8c7c SystemUUID:ec274d11-ebf7-34d6-57f9-c790077b8c7c BootID:76abe132-bc16-42d7-85d9-36ba7b516edd Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d1:56:5b:39 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d1:56:5b:39 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:65:af:29:43:87 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 16:20:14.083359 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.083355 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 16:20:14.083472 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.083449 2566 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 16:20:14.086683 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.086659 2566 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 16:20:14.086824 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.086684 2566 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-65.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 16:20:14.086871 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.086827 2566 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 16:20:14.086871 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.086841 2566 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 16:20:14.086871 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.086852 2566 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:20:14.087761 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.087751 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:20:14.088620 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.088611 2566 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 16:20:14.092428 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.092416 2566 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 16:20:14.092540 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.092531 2566 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 16:20:14.093289 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.093279 2566 kubelet.go:493] "Attempting to sync node with API server" Oct 30 16:20:14.093324 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.093294 2566 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 16:20:14.093324 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.093304 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 16:20:14.093324 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.093312 2566 kubelet.go:405] "Adding apiserver pod source" Oct 30 16:20:14.093324 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.093320 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 16:20:14.093700 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.093681 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:20:14.097070 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.097034 2566 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 16:20:14.102038 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.102022 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:20:14.102916 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.102903 2566 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 16:20:14.102963 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.102935 2566 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 16:20:14.103122 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.103112 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 16:20:14.103151 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.103130 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 16:20:14.103151 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.103136 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 16:20:14.103151 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.103143 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 16:20:14.103221 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.103152 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 16:20:14.103221 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.103159 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 16:20:14.103805 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.103796 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 16:20:14.103838 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.103806 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 16:20:14.103838 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.103813 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 16:20:14.103838 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.103819 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 16:20:14.103838 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.103829 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 16:20:14.103838 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.103838 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 16:20:14.104753 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.104744 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 16:20:14.104799 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.104757 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 16:20:14.108667 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.108656 2566 server.go:1263] "Started kubelet" Oct 30 16:20:14.108836 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.108795 2566 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 16:20:14.108932 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.108824 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 16:20:14.108932 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.108899 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 16:20:14.110058 ip-10-0-129-65 systemd[1]: Started Kubernetes Kubelet. Oct 30 16:20:14.111418 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.111400 2566 server.go:317] "Adding debug handlers to kubelet server" Oct 30 16:20:14.112821 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.112807 2566 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 16:20:14.114696 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:14.114676 2566 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 16:20:14.116802 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.116786 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 16:20:14.117239 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.117222 2566 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 16:20:14.117704 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:14.117381 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-65.ec2.internal\" not found" Oct 30 16:20:14.118011 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.117990 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 16:20:14.118011 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.117991 2566 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 16:20:14.118135 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.118018 2566 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 16:20:14.118184 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.118140 2566 reconstruct.go:97] "Volume reconstruction finished" Oct 30 16:20:14.118184 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.118148 2566 reconciler.go:29] "Reconciler: start to sync state" Oct 30 16:20:14.118281 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.118257 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 16:20:14.118340 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.118284 2566 factory.go:55] Registering systemd factory Oct 30 16:20:14.118340 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.118293 2566 factory.go:223] Registration of the systemd container factory successfully Oct 30 16:20:14.118506 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.118493 2566 factory.go:153] Registering CRI-O factory Oct 30 16:20:14.118506 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.118507 2566 factory.go:223] Registration of the crio container factory successfully Oct 30 16:20:14.118596 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.118520 2566 factory.go:103] Registering Raw factory Oct 30 16:20:14.118596 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.118532 2566 manager.go:1199] Started watching for new ooms in manager Oct 30 16:20:14.119451 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.119436 2566 manager.go:319] Starting recovery of all containers Oct 30 16:20:14.131011 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.130991 2566 manager.go:324] Recovery completed Oct 30 16:20:14.135415 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.135401 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:20:14.136947 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.136928 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:20:14.137017 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.136960 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:20:14.137017 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.136972 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:20:14.137464 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.137447 2566 cpu_manager.go:226] "Starting" policy="none" Oct 30 16:20:14.137464 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.137463 2566 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 16:20:14.137591 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.137482 2566 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 16:20:14.137591 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.137452 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:20:14.138191 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.138177 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:20:14.138254 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.138202 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:20:14.138254 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.138211 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:20:14.139734 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.139722 2566 policy_none.go:50] "Start" Oct 30 16:20:14.139786 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.139749 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 16:20:14.139828 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.139811 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 16:20:14.141840 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.141828 2566 policy_none.go:44] "Start" Oct 30 16:20:14.149894 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.149877 2566 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 16:20:14.149955 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.149915 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zsg2w" Oct 30 16:20:14.158887 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.158868 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zsg2w" Oct 30 16:20:14.159022 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.159008 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-65.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 16:20:14.159117 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:14.159007 2566 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-65.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 16:20:14.159468 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:14.159449 2566 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 16:20:14.173201 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:14.171718 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-65.ec2.internal.18e35b1313397aa1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-65.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-65.ec2.internal,},FirstTimestamp:2026-10-30 16:20:14.108629665 +0000 UTC m=+0.391159551,LastTimestamp:2026-10-30 16:20:14.108629665 +0000 UTC m=+0.391159551,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-65.ec2.internal,}" Oct 30 16:20:14.175335 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.175308 2566 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 16:20:14.177609 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.177598 2566 manager.go:341] "Starting Device Plugin manager" Oct 30 16:20:14.178116 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:14.178103 2566 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 16:20:14.178153 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.178117 2566 server.go:87] "Starting device plugin registration server" Oct 30 16:20:14.178305 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.178295 2566 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 16:20:14.178338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.178309 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 16:20:14.178438 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.178423 2566 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 16:20:14.178504 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.178495 2566 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 16:20:14.178555 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.178506 2566 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 16:20:14.179034 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:14.179014 2566 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 16:20:14.179130 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:14.179063 2566 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-65.ec2.internal\" not found" Oct 30 16:20:14.188160 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.188145 2566 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:14.226964 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.211845 2566 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:14.249820 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.249791 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 16:20:14.251083 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.251061 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 16:20:14.251159 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.251089 2566 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 16:20:14.251159 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.251110 2566 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 16:20:14.251159 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:14.251147 2566 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 16:20:14.265548 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.265529 2566 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:14.278654 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.278618 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:20:14.279415 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.279398 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:20:14.279492 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.279431 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:20:14.279492 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.279441 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:20:14.279492 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.279472 2566 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-65.ec2.internal" Oct 30 16:20:14.299597 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.299573 2566 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-65.ec2.internal" Oct 30 16:20:14.351259 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.351241 2566 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal"] Oct 30 16:20:14.353433 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.353418 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:20:14.355138 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.355126 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Oct 30 16:20:14.376886 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.376871 2566 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:20:14.378798 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.378783 2566 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Oct 30 16:20:14.401138 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.400916 2566 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:20:14.401138 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.401085 2566 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:20:14.519088 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.519042 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/829ec3bbd4f4a9d896bac875cd5fa87a-config\") pod \"kube-apiserver-proxy-ip-10-0-129-65.ec2.internal\" (UID: \"829ec3bbd4f4a9d896bac875cd5fa87a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Oct 30 16:20:14.519088 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.519094 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7ecc0c6cd8524d6054106d923cd48af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" (UID: \"d7ecc0c6cd8524d6054106d923cd48af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:20:14.519264 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.519179 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ecc0c6cd8524d6054106d923cd48af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" (UID: \"d7ecc0c6cd8524d6054106d923cd48af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:20:14.620168 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.620135 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/829ec3bbd4f4a9d896bac875cd5fa87a-config\") pod \"kube-apiserver-proxy-ip-10-0-129-65.ec2.internal\" (UID: \"829ec3bbd4f4a9d896bac875cd5fa87a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Oct 30 16:20:14.620168 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.620173 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7ecc0c6cd8524d6054106d923cd48af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" (UID: \"d7ecc0c6cd8524d6054106d923cd48af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:20:14.620330 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.620188 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ecc0c6cd8524d6054106d923cd48af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" (UID: \"d7ecc0c6cd8524d6054106d923cd48af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:20:14.620330 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.620243 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7ecc0c6cd8524d6054106d923cd48af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" (UID: \"d7ecc0c6cd8524d6054106d923cd48af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:20:14.620330 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.620248 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/829ec3bbd4f4a9d896bac875cd5fa87a-config\") pod \"kube-apiserver-proxy-ip-10-0-129-65.ec2.internal\" (UID: \"829ec3bbd4f4a9d896bac875cd5fa87a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Oct 30 16:20:14.620330 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.620303 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ecc0c6cd8524d6054106d923cd48af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" (UID: \"d7ecc0c6cd8524d6054106d923cd48af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:20:14.706104 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.706084 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Oct 30 16:20:14.707156 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:14.707139 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:20:15.049600 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.049513 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 16:20:15.050086 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.049699 2566 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 16:20:15.050086 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.049700 2566 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 16:20:15.094182 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.094155 2566 apiserver.go:52] "Watching apiserver" Oct 30 16:20:15.117252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.117226 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 16:20:15.145463 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.145440 2566 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 16:20:15.145797 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.145758 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal","openshift-image-registry/node-ca-7w7js","openshift-network-diagnostics/network-check-target-xchk6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2","openshift-multus/multus-99l8n","openshift-multus/multus-additional-cni-plugins-m72dq","openshift-multus/network-metrics-daemon-wtk5g","kube-system/global-pull-secret-syncer-k4k2w","kube-system/konnectivity-agent-4n84w","openshift-network-operator/iptables-alerter-p694c","openshift-ovn-kubernetes/ovnkube-node-6sprd","kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal","openshift-cluster-node-tuning-operator/tuned-868tl","openshift-dns/node-resolver-lr2cd"] Oct 30 16:20:15.148353 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.148331 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7w7js" Oct 30 16:20:15.151079 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.151061 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:15.151172 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.151132 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:15.151237 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.151175 2566 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:15.153838 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.153824 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.156474 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.156460 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.159192 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.159176 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.162161 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.162145 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:15.162254 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.162200 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:15.164047 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.164023 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:15:14 +0000 UTC" deadline="2028-05-24 01:38:12.360821572 +0000 UTC" Oct 30 16:20:15.164109 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.164047 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13713h17m57.196778419s" Oct 30 16:20:15.164599 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.164580 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:15.164696 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.164648 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:15.165649 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.165622 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fpzw7\"" Oct 30 16:20:15.165834 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.165811 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 16:20:15.165921 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.165818 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 16:20:15.165921 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.165868 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 16:20:15.168953 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.168931 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4n84w" Oct 30 16:20:15.172352 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.171071 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p694c" Oct 30 16:20:15.172514 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.172481 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n7jsz\"" Oct 30 16:20:15.175760 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.174440 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.176656 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.176632 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.177751 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.177731 2566 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 16:20:15.177900 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.177884 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jszml" Oct 30 16:20:15.179614 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.179597 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lr2cd" Oct 30 16:20:15.186601 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.186580 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jszml" Oct 30 16:20:15.219056 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.219031 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 16:20:15.222794 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.222755 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-var-lib-cni-bin\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.222883 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.222806 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-etc-kubernetes\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.222883 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.222875 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znx2b\" (UniqueName: \"kubernetes.io/projected/17726533-46eb-4d51-bc1d-b4e9959d7ba5-kube-api-access-znx2b\") pod \"node-ca-7w7js\" (UID: \"17726533-46eb-4d51-bc1d-b4e9959d7ba5\") " pod="openshift-image-registry/node-ca-7w7js" Oct 30 16:20:15.222973 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.222899 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc-hosts-file\") pod \"node-resolver-lr2cd\" (UID: \"ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc\") " pod="openshift-dns/node-resolver-lr2cd" Oct 30 16:20:15.222973 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.222916 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-var-lib-cni-multus\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.222973 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.222936 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-run-multus-certs\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.222973 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.222952 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-run-systemd\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.223141 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.222993 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmxkq\" (UniqueName: \"kubernetes.io/projected/25906571-b2ad-4134-9674-e67842de154d-kube-api-access-tmxkq\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.223141 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223027 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-kubernetes\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.223141 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223046 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-systemd\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.223141 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223087 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-socket-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.223141 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223124 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-etc-selinux\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.223332 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223159 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffdd3593-6091-4874-b55e-4049aa26ab60-dbus\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:15.223332 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223212 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32528057-07fa-4eb9-882f-a43c9cd36b31-konnectivity-ca\") pod \"konnectivity-agent-4n84w\" (UID: \"32528057-07fa-4eb9-882f-a43c9cd36b31\") " pod="kube-system/konnectivity-agent-4n84w" Oct 30 16:20:15.223332 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223256 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/25906571-b2ad-4134-9674-e67842de154d-env-overrides\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.223332 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223286 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-tuned\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.223332 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223308 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-daemon-config\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.223497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223337 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-run-netns\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.223497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223354 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-device-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.223497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223406 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4512b7bb-6729-4045-aac9-4a1ece176016-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.223497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223433 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-systemd-units\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.223497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223473 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.223693 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223508 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-run-netns\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.223693 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223538 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32528057-07fa-4eb9-882f-a43c9cd36b31-agent-certs\") pod \"konnectivity-agent-4n84w\" (UID: \"32528057-07fa-4eb9-882f-a43c9cd36b31\") " pod="kube-system/konnectivity-agent-4n84w" Oct 30 16:20:15.223693 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223572 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-var-lib-openvswitch\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.223693 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223656 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-var-lib-kubelet\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.223889 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223700 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4512b7bb-6729-4045-aac9-4a1ece176016-cnibin\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.223889 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223720 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4512b7bb-6729-4045-aac9-4a1ece176016-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.223889 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223784 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-cni-netd\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.223889 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223817 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-lib-modules\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.223889 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223840 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-host\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.223889 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223878 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-kubelet\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.224066 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223908 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-etc-openvswitch\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.224066 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223932 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-sysctl-d\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.224066 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223964 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7b597f3-5dc9-45e6-840f-b492ba06c241-tmp\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.224066 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.223985 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.224066 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224020 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2p9c\" (UniqueName: \"kubernetes.io/projected/c1bbf041-5696-4a2c-94a8-69ab0e995096-kube-api-access-c2p9c\") pod \"iptables-alerter-p694c\" (UID: \"c1bbf041-5696-4a2c-94a8-69ab0e995096\") " pod="openshift-network-operator/iptables-alerter-p694c" Oct 30 16:20:15.224260 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224067 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17726533-46eb-4d51-bc1d-b4e9959d7ba5-host\") pod \"node-ca-7w7js\" (UID: \"17726533-46eb-4d51-bc1d-b4e9959d7ba5\") " pod="openshift-image-registry/node-ca-7w7js" Oct 30 16:20:15.224260 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224101 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/25906571-b2ad-4134-9674-e67842de154d-ovn-node-metrics-cert\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.224260 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224145 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-sysconfig\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.224260 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224179 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsrqq\" (UniqueName: \"kubernetes.io/projected/42aa85ec-676a-4f13-90f4-0fb13e94b22f-kube-api-access-dsrqq\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.224260 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224198 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-slash\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.224260 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224229 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/25906571-b2ad-4134-9674-e67842de154d-ovnkube-script-lib\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.224427 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224262 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-modprobe-d\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.224427 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224298 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc-tmp-dir\") pod \"node-resolver-lr2cd\" (UID: \"ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc\") " pod="openshift-dns/node-resolver-lr2cd" Oct 30 16:20:15.224427 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224328 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mpwt\" (UniqueName: \"kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt\") pod \"network-check-target-xchk6\" (UID: \"73012827-44d9-4cec-9f8d-40b2055c0c69\") " pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:15.224427 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224352 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-765lm\" (UniqueName: \"kubernetes.io/projected/b17ec352-725f-4bea-aaee-a3ca414fde6e-kube-api-access-765lm\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.224427 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224367 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1bbf041-5696-4a2c-94a8-69ab0e995096-host-slash\") pod \"iptables-alerter-p694c\" (UID: \"c1bbf041-5696-4a2c-94a8-69ab0e995096\") " pod="openshift-network-operator/iptables-alerter-p694c" Oct 30 16:20:15.224427 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224387 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-sys\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.224427 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224409 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6r8b\" (UniqueName: \"kubernetes.io/projected/e7b597f3-5dc9-45e6-840f-b492ba06c241-kube-api-access-z6r8b\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.224654 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224442 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-hostroot\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.224654 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224473 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4512b7bb-6729-4045-aac9-4a1ece176016-system-cni-dir\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.224654 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224494 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4512b7bb-6729-4045-aac9-4a1ece176016-cni-binary-copy\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.224654 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224510 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17726533-46eb-4d51-bc1d-b4e9959d7ba5-serviceca\") pod \"node-ca-7w7js\" (UID: \"17726533-46eb-4d51-bc1d-b4e9959d7ba5\") " pod="openshift-image-registry/node-ca-7w7js" Oct 30 16:20:15.224654 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224532 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wslqp\" (UniqueName: \"kubernetes.io/projected/35d777f2-4005-45e4-af04-2a61c0567fc0-kube-api-access-wslqp\") pod \"network-metrics-daemon-wtk5g\" (UID: \"35d777f2-4005-45e4-af04-2a61c0567fc0\") " pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:15.224654 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224561 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:15.224654 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224580 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs\") pod \"network-metrics-daemon-wtk5g\" (UID: \"35d777f2-4005-45e4-af04-2a61c0567fc0\") " pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:15.224654 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224612 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.224654 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224631 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/25906571-b2ad-4134-9674-e67842de154d-ovnkube-config\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.224925 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224671 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-var-lib-kubelet\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.224925 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224694 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-socket-dir-parent\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.224925 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224708 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-run-k8s-cni-cncf-io\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.224925 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224741 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4512b7bb-6729-4045-aac9-4a1ece176016-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.224925 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224787 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghbq5\" (UniqueName: \"kubernetes.io/projected/4512b7bb-6729-4045-aac9-4a1ece176016-kube-api-access-ghbq5\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.224925 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224815 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-run-ovn\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.224925 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224832 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-node-log\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.224925 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224865 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-run\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.224925 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224886 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k2rm\" (UniqueName: \"kubernetes.io/projected/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc-kube-api-access-5k2rm\") pod \"node-resolver-lr2cd\" (UID: \"ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc\") " pod="openshift-dns/node-resolver-lr2cd" Oct 30 16:20:15.224925 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224903 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-cnibin\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.225211 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224938 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-system-cni-dir\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.225211 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224968 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-cni-dir\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.225211 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.224999 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1bbf041-5696-4a2c-94a8-69ab0e995096-iptables-alerter-script\") pod \"iptables-alerter-p694c\" (UID: \"c1bbf041-5696-4a2c-94a8-69ab0e995096\") " pod="openshift-network-operator/iptables-alerter-p694c" Oct 30 16:20:15.225211 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.225024 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-run-openvswitch\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.225211 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.225040 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-registration-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.225211 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.225066 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-sys-fs\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.225211 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.225088 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-conf-dir\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.225211 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.225105 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffdd3593-6091-4874-b55e-4049aa26ab60-kubelet-config\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:15.225211 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.225124 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-log-socket\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.225211 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.225154 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-cni-bin\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.225211 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.225175 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-sysctl-conf\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.225211 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.225195 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b17ec352-725f-4bea-aaee-a3ca414fde6e-cni-binary-copy\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.265659 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.265635 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 16:20:15.266037 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266015 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l28xz\"" Oct 30 16:20:15.266037 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266024 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gfrzf\"" Oct 30 16:20:15.266176 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266073 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 16:20:15.266176 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266120 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hxhjs\"" Oct 30 16:20:15.266176 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266141 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 16:20:15.266425 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266407 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5kvnn\"" Oct 30 16:20:15.266524 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266436 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:15.266524 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266452 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 16:20:15.266524 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266474 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 16:20:15.266524 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266495 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 16:20:15.266721 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266656 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 16:20:15.266790 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266735 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 16:20:15.266790 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266744 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 16:20:15.266790 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266745 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 16:20:15.266946 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266804 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 16:20:15.266946 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266814 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 16:20:15.266946 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266823 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:15.266946 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266854 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lngbh\"" Oct 30 16:20:15.266946 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266856 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 16:20:15.266946 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266887 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 16:20:15.266946 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.266896 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jrplp\"" Oct 30 16:20:15.279839 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.279823 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:15.280288 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.280269 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 16:20:15.280350 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.280329 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 16:20:15.280636 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.280623 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 16:20:15.280702 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.280691 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 16:20:15.280777 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.280745 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sg8tz\"" Oct 30 16:20:15.280880 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.280753 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 16:20:15.280933 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.280811 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:15.280933 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.280840 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 16:20:15.303585 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:15.303533 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ecc0c6cd8524d6054106d923cd48af.slice/crio-2ef5e9a5a1ba2ce970fe79131d48f63f2821bfef48f192fc8ec0fb5cc54415e3 WatchSource:0}: Error finding container 2ef5e9a5a1ba2ce970fe79131d48f63f2821bfef48f192fc8ec0fb5cc54415e3: Status 404 returned error can't find the container with id 2ef5e9a5a1ba2ce970fe79131d48f63f2821bfef48f192fc8ec0fb5cc54415e3 Oct 30 16:20:15.304502 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:15.304475 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829ec3bbd4f4a9d896bac875cd5fa87a.slice/crio-7ae24ea35dd910421df6922ed257e7e1ebbae16c456a86eade15e12bf9bb4cf1 WatchSource:0}: Error finding container 7ae24ea35dd910421df6922ed257e7e1ebbae16c456a86eade15e12bf9bb4cf1: Status 404 returned error can't find the container with id 7ae24ea35dd910421df6922ed257e7e1ebbae16c456a86eade15e12bf9bb4cf1 Oct 30 16:20:15.309912 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.309899 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:20:15.326364 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326337 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-var-lib-cni-multus\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.326453 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326374 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-run-multus-certs\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.326453 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326397 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-run-systemd\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.326453 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326424 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmxkq\" (UniqueName: \"kubernetes.io/projected/25906571-b2ad-4134-9674-e67842de154d-kube-api-access-tmxkq\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.326558 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326449 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-var-lib-cni-multus\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.326558 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326454 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-run-multus-certs\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.326558 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326470 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-run-systemd\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.326558 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326492 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-kubernetes\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.326558 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326532 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-systemd\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.326558 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326539 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-kubernetes\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.326812 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326565 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-socket-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.326812 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326586 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-etc-selinux\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.326812 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326601 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffdd3593-6091-4874-b55e-4049aa26ab60-dbus\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:15.326812 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326683 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-systemd\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.326812 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326705 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-socket-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.326812 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326709 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-etc-selinux\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.326812 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326739 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32528057-07fa-4eb9-882f-a43c9cd36b31-konnectivity-ca\") pod \"konnectivity-agent-4n84w\" (UID: \"32528057-07fa-4eb9-882f-a43c9cd36b31\") " pod="kube-system/konnectivity-agent-4n84w" Oct 30 16:20:15.326812 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326788 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/25906571-b2ad-4134-9674-e67842de154d-env-overrides\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.326812 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326799 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffdd3593-6091-4874-b55e-4049aa26ab60-dbus\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:15.326812 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326812 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-tuned\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.327204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326829 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-daemon-config\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.327204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326861 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-run-netns\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.327204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326903 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-run-netns\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.327204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326933 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-device-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.327204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326956 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4512b7bb-6729-4045-aac9-4a1ece176016-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.327204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.326980 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-systemd-units\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.327204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327004 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.327204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327004 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-device-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.327204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327046 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-run-netns\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.327204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327076 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32528057-07fa-4eb9-882f-a43c9cd36b31-agent-certs\") pod \"konnectivity-agent-4n84w\" (UID: \"32528057-07fa-4eb9-882f-a43c9cd36b31\") " pod="kube-system/konnectivity-agent-4n84w" Oct 30 16:20:15.327204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327123 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-systemd-units\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.327204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327136 2566 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 16:20:15.327204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327100 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-var-lib-openvswitch\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.327204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327166 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4512b7bb-6729-4045-aac9-4a1ece176016-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.327204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327170 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-run-netns\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.327204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327191 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-var-lib-kubelet\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.327204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327049 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.328008 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327219 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4512b7bb-6729-4045-aac9-4a1ece176016-cnibin\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.328008 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327241 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-var-lib-kubelet\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.328008 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327229 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-var-lib-openvswitch\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.328008 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327258 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4512b7bb-6729-4045-aac9-4a1ece176016-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.328008 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327266 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4512b7bb-6729-4045-aac9-4a1ece176016-cnibin\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.328008 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327284 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-cni-netd\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.328008 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327337 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-lib-modules\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.328008 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327360 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-cni-netd\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.328008 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327362 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-host\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.328008 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327389 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-kubelet\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.328008 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327404 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-etc-openvswitch\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.328008 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327420 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-host\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.328008 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327421 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-sysctl-d\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.328008 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327450 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7b597f3-5dc9-45e6-840f-b492ba06c241-tmp\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.328008 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327469 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-kubelet\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.328008 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327475 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.328008 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327493 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-sysctl-d\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.328733 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327518 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2p9c\" (UniqueName: \"kubernetes.io/projected/c1bbf041-5696-4a2c-94a8-69ab0e995096-kube-api-access-c2p9c\") pod \"iptables-alerter-p694c\" (UID: \"c1bbf041-5696-4a2c-94a8-69ab0e995096\") " pod="openshift-network-operator/iptables-alerter-p694c" Oct 30 16:20:15.328733 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327531 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-etc-openvswitch\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.328733 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327519 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.328733 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327543 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17726533-46eb-4d51-bc1d-b4e9959d7ba5-host\") pod \"node-ca-7w7js\" (UID: \"17726533-46eb-4d51-bc1d-b4e9959d7ba5\") " pod="openshift-image-registry/node-ca-7w7js" Oct 30 16:20:15.328733 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327568 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17726533-46eb-4d51-bc1d-b4e9959d7ba5-host\") pod \"node-ca-7w7js\" (UID: \"17726533-46eb-4d51-bc1d-b4e9959d7ba5\") " pod="openshift-image-registry/node-ca-7w7js" Oct 30 16:20:15.328733 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327568 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/25906571-b2ad-4134-9674-e67842de154d-ovn-node-metrics-cert\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.328733 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327517 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-lib-modules\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.328733 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327591 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-sysconfig\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.328733 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327607 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsrqq\" (UniqueName: \"kubernetes.io/projected/42aa85ec-676a-4f13-90f4-0fb13e94b22f-kube-api-access-dsrqq\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.328733 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327621 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-slash\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.328733 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327642 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/25906571-b2ad-4134-9674-e67842de154d-ovnkube-script-lib\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.328733 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327665 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-modprobe-d\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.328733 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327710 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc-tmp-dir\") pod \"node-resolver-lr2cd\" (UID: \"ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc\") " pod="openshift-dns/node-resolver-lr2cd" Oct 30 16:20:15.328733 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327737 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpwt\" (UniqueName: \"kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt\") pod \"network-check-target-xchk6\" (UID: \"73012827-44d9-4cec-9f8d-40b2055c0c69\") " pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:15.328733 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327784 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-765lm\" (UniqueName: \"kubernetes.io/projected/b17ec352-725f-4bea-aaee-a3ca414fde6e-kube-api-access-765lm\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.328733 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327808 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1bbf041-5696-4a2c-94a8-69ab0e995096-host-slash\") pod \"iptables-alerter-p694c\" (UID: \"c1bbf041-5696-4a2c-94a8-69ab0e995096\") " pod="openshift-network-operator/iptables-alerter-p694c" Oct 30 16:20:15.328733 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327829 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-sys\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.330041 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327834 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-modprobe-d\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.330041 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327853 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6r8b\" (UniqueName: \"kubernetes.io/projected/e7b597f3-5dc9-45e6-840f-b492ba06c241-kube-api-access-z6r8b\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.330041 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327880 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-hostroot\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.330041 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327894 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-sysconfig\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.330041 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327900 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4512b7bb-6729-4045-aac9-4a1ece176016-system-cni-dir\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.330041 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327925 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4512b7bb-6729-4045-aac9-4a1ece176016-cni-binary-copy\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.330041 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327948 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17726533-46eb-4d51-bc1d-b4e9959d7ba5-serviceca\") pod \"node-ca-7w7js\" (UID: \"17726533-46eb-4d51-bc1d-b4e9959d7ba5\") " pod="openshift-image-registry/node-ca-7w7js" Oct 30 16:20:15.330041 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327970 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wslqp\" (UniqueName: \"kubernetes.io/projected/35d777f2-4005-45e4-af04-2a61c0567fc0-kube-api-access-wslqp\") pod \"network-metrics-daemon-wtk5g\" (UID: \"35d777f2-4005-45e4-af04-2a61c0567fc0\") " pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:15.330041 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327994 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:15.330041 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.327999 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32528057-07fa-4eb9-882f-a43c9cd36b31-konnectivity-ca\") pod \"konnectivity-agent-4n84w\" (UID: \"32528057-07fa-4eb9-882f-a43c9cd36b31\") " pod="kube-system/konnectivity-agent-4n84w" Oct 30 16:20:15.330041 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328015 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs\") pod \"network-metrics-daemon-wtk5g\" (UID: \"35d777f2-4005-45e4-af04-2a61c0567fc0\") " pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:15.330041 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328041 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.330041 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328069 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1bbf041-5696-4a2c-94a8-69ab0e995096-host-slash\") pod \"iptables-alerter-p694c\" (UID: \"c1bbf041-5696-4a2c-94a8-69ab0e995096\") " pod="openshift-network-operator/iptables-alerter-p694c" Oct 30 16:20:15.330041 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328066 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/25906571-b2ad-4134-9674-e67842de154d-ovnkube-config\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.330041 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328112 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-var-lib-kubelet\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.330041 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328135 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-socket-dir-parent\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.330041 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328157 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-run-k8s-cni-cncf-io\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.330754 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328182 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4512b7bb-6729-4045-aac9-4a1ece176016-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.330754 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328208 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghbq5\" (UniqueName: \"kubernetes.io/projected/4512b7bb-6729-4045-aac9-4a1ece176016-kube-api-access-ghbq5\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.330754 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328207 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc-tmp-dir\") pod \"node-resolver-lr2cd\" (UID: \"ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc\") " pod="openshift-dns/node-resolver-lr2cd" Oct 30 16:20:15.330754 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328237 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-run-ovn\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.330754 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328569 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-node-log\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.330754 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328607 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-run\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.330754 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328638 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k2rm\" (UniqueName: \"kubernetes.io/projected/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc-kube-api-access-5k2rm\") pod \"node-resolver-lr2cd\" (UID: \"ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc\") " pod="openshift-dns/node-resolver-lr2cd" Oct 30 16:20:15.330754 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328672 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-cnibin\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.330754 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328739 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4512b7bb-6729-4045-aac9-4a1ece176016-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.330754 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328748 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-cnibin\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.330754 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328893 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-system-cni-dir\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.330754 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328956 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-cni-dir\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.330754 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.329036 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-cni-dir\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.330754 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.329064 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:15.330754 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.329091 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1bbf041-5696-4a2c-94a8-69ab0e995096-iptables-alerter-script\") pod \"iptables-alerter-p694c\" (UID: \"c1bbf041-5696-4a2c-94a8-69ab0e995096\") " pod="openshift-network-operator/iptables-alerter-p694c" Oct 30 16:20:15.330754 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.329128 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-run-openvswitch\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.330754 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.329152 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs podName:35d777f2-4005-45e4-af04-2a61c0567fc0 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:15.82913074 +0000 UTC m=+2.111660638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs") pod "network-metrics-daemon-wtk5g" (UID: "35d777f2-4005-45e4-af04-2a61c0567fc0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:15.331497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.329178 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-run-openvswitch\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.331497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.329198 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-registration-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.331497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.329237 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-sys-fs\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.331497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.329244 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-system-cni-dir\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.331497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.329256 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4512b7bb-6729-4045-aac9-4a1ece176016-cni-binary-copy\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.331497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.329280 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-conf-dir\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.331497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.329302 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.331497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.329667 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17726533-46eb-4d51-bc1d-b4e9959d7ba5-serviceca\") pod \"node-ca-7w7js\" (UID: \"17726533-46eb-4d51-bc1d-b4e9959d7ba5\") " pod="openshift-image-registry/node-ca-7w7js" Oct 30 16:20:15.331497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.329782 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffdd3593-6091-4874-b55e-4049aa26ab60-kubelet-config\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:15.331497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.329816 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-log-socket\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.331497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.329847 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-cni-bin\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.331497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.329880 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-sysctl-conf\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.331497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.329935 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b17ec352-725f-4bea-aaee-a3ca414fde6e-cni-binary-copy\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.331497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.329969 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-var-lib-cni-bin\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.331497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.330005 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-etc-kubernetes\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.331497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.330040 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znx2b\" (UniqueName: \"kubernetes.io/projected/17726533-46eb-4d51-bc1d-b4e9959d7ba5-kube-api-access-znx2b\") pod \"node-ca-7w7js\" (UID: \"17726533-46eb-4d51-bc1d-b4e9959d7ba5\") " pod="openshift-image-registry/node-ca-7w7js" Oct 30 16:20:15.331497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.330078 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc-hosts-file\") pod \"node-resolver-lr2cd\" (UID: \"ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc\") " pod="openshift-dns/node-resolver-lr2cd" Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328112 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-sys\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.330211 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc-hosts-file\") pod \"node-resolver-lr2cd\" (UID: \"ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc\") " pod="openshift-dns/node-resolver-lr2cd" Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.330249 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/25906571-b2ad-4134-9674-e67842de154d-env-overrides\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.328111 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-daemon-config\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.330815 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/25906571-b2ad-4134-9674-e67842de154d-ovnkube-script-lib\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.330190 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/25906571-b2ad-4134-9674-e67842de154d-ovnkube-config\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.331205 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-var-lib-kubelet\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.331254 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.331270 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-sys-fs\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.331299 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffdd3593-6091-4874-b55e-4049aa26ab60-kubelet-config\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.331344 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-socket-dir-parent\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.331260 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-conf-dir\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.331376 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-log-socket\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.331418 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-run-k8s-cni-cncf-io\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.331444 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-cni-bin\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.331609 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-sysctl-conf\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.331830 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-var-lib-cni-bin\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.331878 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-run-ovn\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.332252 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.332012 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-node-log\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.333066 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.332073 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-etc-kubernetes\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.333066 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.332128 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-run\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.333066 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.332249 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4512b7bb-6729-4045-aac9-4a1ece176016-system-cni-dir\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.333066 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.332315 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-slash\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.333066 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.332376 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-hostroot\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.333066 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.332428 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-registration-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.333391 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.333375 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret podName:ffdd3593-6091-4874-b55e-4049aa26ab60 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:15.833353164 +0000 UTC m=+2.115883051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret") pod "global-pull-secret-syncer-k4k2w" (UID: "ffdd3593-6091-4874-b55e-4049aa26ab60") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:15.333936 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.333912 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-tuned\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.334006 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.333931 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32528057-07fa-4eb9-882f-a43c9cd36b31-agent-certs\") pod \"konnectivity-agent-4n84w\" (UID: \"32528057-07fa-4eb9-882f-a43c9cd36b31\") " pod="kube-system/konnectivity-agent-4n84w" Oct 30 16:20:15.334006 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.333976 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/25906571-b2ad-4134-9674-e67842de154d-ovn-node-metrics-cert\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.334076 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.334048 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1bbf041-5696-4a2c-94a8-69ab0e995096-iptables-alerter-script\") pod \"iptables-alerter-p694c\" (UID: \"c1bbf041-5696-4a2c-94a8-69ab0e995096\") " pod="openshift-network-operator/iptables-alerter-p694c" Oct 30 16:20:15.334146 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.334127 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b17ec352-725f-4bea-aaee-a3ca414fde6e-cni-binary-copy\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.334207 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.334146 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4512b7bb-6729-4045-aac9-4a1ece176016-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.334911 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.334894 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7b597f3-5dc9-45e6-840f-b492ba06c241-tmp\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.351243 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.351215 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmxkq\" (UniqueName: \"kubernetes.io/projected/25906571-b2ad-4134-9674-e67842de154d-kube-api-access-tmxkq\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.351551 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.351535 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2p9c\" (UniqueName: \"kubernetes.io/projected/c1bbf041-5696-4a2c-94a8-69ab0e995096-kube-api-access-c2p9c\") pod \"iptables-alerter-p694c\" (UID: \"c1bbf041-5696-4a2c-94a8-69ab0e995096\") " pod="openshift-network-operator/iptables-alerter-p694c" Oct 30 16:20:15.353017 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.353000 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsrqq\" (UniqueName: \"kubernetes.io/projected/42aa85ec-676a-4f13-90f4-0fb13e94b22f-kube-api-access-dsrqq\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.354847 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.354797 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wslqp\" (UniqueName: \"kubernetes.io/projected/35d777f2-4005-45e4-af04-2a61c0567fc0-kube-api-access-wslqp\") pod \"network-metrics-daemon-wtk5g\" (UID: \"35d777f2-4005-45e4-af04-2a61c0567fc0\") " pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:15.354847 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.354822 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:15.354847 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.354849 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:15.355051 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.354863 2566 projected.go:196] Error preparing data for projected volume kube-api-access-5mpwt for pod openshift-network-diagnostics/network-check-target-xchk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:15.355051 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.354917 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt podName:73012827-44d9-4cec-9f8d-40b2055c0c69 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:15.854899844 +0000 UTC m=+2.137429735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5mpwt" (UniqueName: "kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt") pod "network-check-target-xchk6" (UID: "73012827-44d9-4cec-9f8d-40b2055c0c69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:15.355051 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.354941 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-765lm\" (UniqueName: \"kubernetes.io/projected/b17ec352-725f-4bea-aaee-a3ca414fde6e-kube-api-access-765lm\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.356349 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.356323 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6r8b\" (UniqueName: \"kubernetes.io/projected/e7b597f3-5dc9-45e6-840f-b492ba06c241-kube-api-access-z6r8b\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.356752 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.356732 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghbq5\" (UniqueName: \"kubernetes.io/projected/4512b7bb-6729-4045-aac9-4a1ece176016-kube-api-access-ghbq5\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.357719 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.357698 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znx2b\" (UniqueName: \"kubernetes.io/projected/17726533-46eb-4d51-bc1d-b4e9959d7ba5-kube-api-access-znx2b\") pod \"node-ca-7w7js\" (UID: \"17726533-46eb-4d51-bc1d-b4e9959d7ba5\") " pod="openshift-image-registry/node-ca-7w7js" Oct 30 16:20:15.357775 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.357697 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k2rm\" (UniqueName: \"kubernetes.io/projected/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc-kube-api-access-5k2rm\") pod \"node-resolver-lr2cd\" (UID: \"ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc\") " pod="openshift-dns/node-resolver-lr2cd" Oct 30 16:20:15.475687 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.475650 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7w7js" Oct 30 16:20:15.485862 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:15.485837 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17726533_46eb_4d51_bc1d_b4e9959d7ba5.slice/crio-8a74a378aac3d83f9a93fc7212e59dc36ec5df65b30c05569341922dd6f0b669 WatchSource:0}: Error finding container 8a74a378aac3d83f9a93fc7212e59dc36ec5df65b30c05569341922dd6f0b669: Status 404 returned error can't find the container with id 8a74a378aac3d83f9a93fc7212e59dc36ec5df65b30c05569341922dd6f0b669 Oct 30 16:20:15.498372 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.498348 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:20:15.507398 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:15.507374 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42aa85ec_676a_4f13_90f4_0fb13e94b22f.slice/crio-7af113a1c1ae3ca9d88081baddb267079c3c8eee8a2e653bf0bc7c654c50da47 WatchSource:0}: Error finding container 7af113a1c1ae3ca9d88081baddb267079c3c8eee8a2e653bf0bc7c654c50da47: Status 404 returned error can't find the container with id 7af113a1c1ae3ca9d88081baddb267079c3c8eee8a2e653bf0bc7c654c50da47 Oct 30 16:20:15.511750 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.511734 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-99l8n" Oct 30 16:20:15.520118 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:15.520092 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb17ec352_725f_4bea_aaee_a3ca414fde6e.slice/crio-9d3d29cc7e093b1be18697db482c2e1f86cca240144f5b21745dc05d7271928a WatchSource:0}: Error finding container 9d3d29cc7e093b1be18697db482c2e1f86cca240144f5b21745dc05d7271928a: Status 404 returned error can't find the container with id 9d3d29cc7e093b1be18697db482c2e1f86cca240144f5b21745dc05d7271928a Oct 30 16:20:15.532972 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.532955 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:20:15.535123 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.535102 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4n84w" Oct 30 16:20:15.543604 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:15.543581 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4512b7bb_6729_4045_aac9_4a1ece176016.slice/crio-eaa7ac44907717dfd5e76801c725ac94f8101ae4512b52404e04893350cbf434 WatchSource:0}: Error finding container eaa7ac44907717dfd5e76801c725ac94f8101ae4512b52404e04893350cbf434: Status 404 returned error can't find the container with id eaa7ac44907717dfd5e76801c725ac94f8101ae4512b52404e04893350cbf434 Oct 30 16:20:15.544529 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:15.544503 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32528057_07fa_4eb9_882f_a43c9cd36b31.slice/crio-2d0aa1ec165a544356719160f8a7c8b019d8db79dcc1309892994e71a1bdcede WatchSource:0}: Error finding container 2d0aa1ec165a544356719160f8a7c8b019d8db79dcc1309892994e71a1bdcede: Status 404 returned error can't find the container with id 2d0aa1ec165a544356719160f8a7c8b019d8db79dcc1309892994e71a1bdcede Oct 30 16:20:15.552040 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.552020 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p694c" Oct 30 16:20:15.561394 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:15.561374 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bbf041_5696_4a2c_94a8_69ab0e995096.slice/crio-1f31f459ebbe6e8f2965c2fd4d8ff1b385c2cfd2342cc2417bf8e1ca7cb21da7 WatchSource:0}: Error finding container 1f31f459ebbe6e8f2965c2fd4d8ff1b385c2cfd2342cc2417bf8e1ca7cb21da7: Status 404 returned error can't find the container with id 1f31f459ebbe6e8f2965c2fd4d8ff1b385c2cfd2342cc2417bf8e1ca7cb21da7 Oct 30 16:20:15.583230 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.583210 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:15.585852 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.585835 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:20:15.590958 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.590943 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lr2cd" Oct 30 16:20:15.593278 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:15.593261 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25906571_b2ad_4134_9674_e67842de154d.slice/crio-25905daee0d773c71b579005e239a140a15a514e6aa7a146fa1ba5eabbc9cb79 WatchSource:0}: Error finding container 25905daee0d773c71b579005e239a140a15a514e6aa7a146fa1ba5eabbc9cb79: Status 404 returned error can't find the container with id 25905daee0d773c71b579005e239a140a15a514e6aa7a146fa1ba5eabbc9cb79 Oct 30 16:20:15.596205 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:15.596166 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b597f3_5dc9_45e6_840f_b492ba06c241.slice/crio-7d7ac62b232520032246b8700261c5399744e8e5d8e1d55a85f51ff008fbfc3d WatchSource:0}: Error finding container 7d7ac62b232520032246b8700261c5399744e8e5d8e1d55a85f51ff008fbfc3d: Status 404 returned error can't find the container with id 7d7ac62b232520032246b8700261c5399744e8e5d8e1d55a85f51ff008fbfc3d Oct 30 16:20:15.603211 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:20:15.603192 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4b38a9_e2b5_43dd_ad2b_6ecf9a080ffc.slice/crio-dc3bd5082efb84b9694faee499d28f189117d32c69e400ca7377b315d0b1058b WatchSource:0}: Error finding container dc3bd5082efb84b9694faee499d28f189117d32c69e400ca7377b315d0b1058b: Status 404 returned error can't find the container with id dc3bd5082efb84b9694faee499d28f189117d32c69e400ca7377b315d0b1058b Oct 30 16:20:15.833811 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.833777 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:15.833967 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.833828 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs\") pod \"network-metrics-daemon-wtk5g\" (UID: \"35d777f2-4005-45e4-af04-2a61c0567fc0\") " pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:15.833967 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.833876 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:15.833967 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.833945 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret podName:ffdd3593-6091-4874-b55e-4049aa26ab60 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:16.833924935 +0000 UTC m=+3.116454825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret") pod "global-pull-secret-syncer-k4k2w" (UID: "ffdd3593-6091-4874-b55e-4049aa26ab60") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:15.834139 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.834000 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:15.834139 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.834040 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs podName:35d777f2-4005-45e4-af04-2a61c0567fc0 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:16.834028517 +0000 UTC m=+3.116558408 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs") pod "network-metrics-daemon-wtk5g" (UID: "35d777f2-4005-45e4-af04-2a61c0567fc0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:15.934491 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:15.934450 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpwt\" (UniqueName: \"kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt\") pod \"network-check-target-xchk6\" (UID: \"73012827-44d9-4cec-9f8d-40b2055c0c69\") " pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:15.934659 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.934641 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:15.934659 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.934660 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:15.934783 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.934672 2566 projected.go:196] Error preparing data for projected volume kube-api-access-5mpwt for pod openshift-network-diagnostics/network-check-target-xchk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:15.934783 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:15.934730 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt podName:73012827-44d9-4cec-9f8d-40b2055c0c69 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:16.934712423 +0000 UTC m=+3.217242299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mpwt" (UniqueName: "kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt") pod "network-check-target-xchk6" (UID: "73012827-44d9-4cec-9f8d-40b2055c0c69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:16.189346 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:16.189180 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:15:15 +0000 UTC" deadline="2028-07-22 22:57:43.270561728 +0000 UTC" Oct 30 16:20:16.189346 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:16.189241 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15150h37m27.081325683s" Oct 30 16:20:16.263875 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:16.263789 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lr2cd" event={"ID":"ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc","Type":"ContainerStarted","Data":"dc3bd5082efb84b9694faee499d28f189117d32c69e400ca7377b315d0b1058b"} Oct 30 16:20:16.280191 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:16.280144 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" event={"ID":"25906571-b2ad-4134-9674-e67842de154d","Type":"ContainerStarted","Data":"25905daee0d773c71b579005e239a140a15a514e6aa7a146fa1ba5eabbc9cb79"} Oct 30 16:20:16.288207 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:16.288175 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p694c" event={"ID":"c1bbf041-5696-4a2c-94a8-69ab0e995096","Type":"ContainerStarted","Data":"1f31f459ebbe6e8f2965c2fd4d8ff1b385c2cfd2342cc2417bf8e1ca7cb21da7"} Oct 30 16:20:16.290357 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:16.290325 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-99l8n" event={"ID":"b17ec352-725f-4bea-aaee-a3ca414fde6e","Type":"ContainerStarted","Data":"9d3d29cc7e093b1be18697db482c2e1f86cca240144f5b21745dc05d7271928a"} Oct 30 16:20:16.297073 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:16.297041 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" event={"ID":"42aa85ec-676a-4f13-90f4-0fb13e94b22f","Type":"ContainerStarted","Data":"7af113a1c1ae3ca9d88081baddb267079c3c8eee8a2e653bf0bc7c654c50da47"} Oct 30 16:20:16.300183 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:16.300149 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7w7js" event={"ID":"17726533-46eb-4d51-bc1d-b4e9959d7ba5","Type":"ContainerStarted","Data":"8a74a378aac3d83f9a93fc7212e59dc36ec5df65b30c05569341922dd6f0b669"} Oct 30 16:20:16.307062 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:16.307033 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-868tl" event={"ID":"e7b597f3-5dc9-45e6-840f-b492ba06c241","Type":"ContainerStarted","Data":"7d7ac62b232520032246b8700261c5399744e8e5d8e1d55a85f51ff008fbfc3d"} Oct 30 16:20:16.311658 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:16.311626 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4n84w" event={"ID":"32528057-07fa-4eb9-882f-a43c9cd36b31","Type":"ContainerStarted","Data":"2d0aa1ec165a544356719160f8a7c8b019d8db79dcc1309892994e71a1bdcede"} Oct 30 16:20:16.317423 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:16.317373 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m72dq" event={"ID":"4512b7bb-6729-4045-aac9-4a1ece176016","Type":"ContainerStarted","Data":"eaa7ac44907717dfd5e76801c725ac94f8101ae4512b52404e04893350cbf434"} Oct 30 16:20:16.324221 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:16.324184 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" event={"ID":"829ec3bbd4f4a9d896bac875cd5fa87a","Type":"ContainerStarted","Data":"7ae24ea35dd910421df6922ed257e7e1ebbae16c456a86eade15e12bf9bb4cf1"} Oct 30 16:20:16.334172 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:16.334149 2566 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:16.345449 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:16.345416 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" event={"ID":"d7ecc0c6cd8524d6054106d923cd48af","Type":"ContainerStarted","Data":"2ef5e9a5a1ba2ce970fe79131d48f63f2821bfef48f192fc8ec0fb5cc54415e3"} Oct 30 16:20:16.446427 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:16.446341 2566 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:16.843946 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:16.842990 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:16.843946 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:16.843057 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs\") pod \"network-metrics-daemon-wtk5g\" (UID: \"35d777f2-4005-45e4-af04-2a61c0567fc0\") " pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:16.843946 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:16.843256 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:16.843946 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:16.843325 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs podName:35d777f2-4005-45e4-af04-2a61c0567fc0 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:18.843304129 +0000 UTC m=+5.125834015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs") pod "network-metrics-daemon-wtk5g" (UID: "35d777f2-4005-45e4-af04-2a61c0567fc0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:16.843946 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:16.843828 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:16.843946 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:16.843908 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret podName:ffdd3593-6091-4874-b55e-4049aa26ab60 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:18.843887903 +0000 UTC m=+5.126417804 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret") pod "global-pull-secret-syncer-k4k2w" (UID: "ffdd3593-6091-4874-b55e-4049aa26ab60") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:16.944090 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:16.944046 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpwt\" (UniqueName: \"kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt\") pod \"network-check-target-xchk6\" (UID: \"73012827-44d9-4cec-9f8d-40b2055c0c69\") " pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:16.944279 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:16.944238 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:16.944279 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:16.944263 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:16.944279 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:16.944277 2566 projected.go:196] Error preparing data for projected volume kube-api-access-5mpwt for pod openshift-network-diagnostics/network-check-target-xchk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:16.944469 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:16.944331 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt podName:73012827-44d9-4cec-9f8d-40b2055c0c69 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:18.944313463 +0000 UTC m=+5.226843341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mpwt" (UniqueName: "kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt") pod "network-check-target-xchk6" (UID: "73012827-44d9-4cec-9f8d-40b2055c0c69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:17.190143 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:17.190027 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:15:15 +0000 UTC" deadline="2028-05-26 13:07:16.788280928 +0000 UTC" Oct 30 16:20:17.190143 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:17.190067 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13772h46m59.59821902s" Oct 30 16:20:17.251590 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:17.251555 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:17.251779 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:17.251720 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:17.252184 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:17.252162 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:17.252293 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:17.252272 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:17.252364 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:17.252350 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:17.252453 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:17.252436 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:18.865175 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:18.863833 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:18.865175 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:18.863882 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs\") pod \"network-metrics-daemon-wtk5g\" (UID: \"35d777f2-4005-45e4-af04-2a61c0567fc0\") " pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:18.865175 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:18.864031 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:18.865175 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:18.864101 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs podName:35d777f2-4005-45e4-af04-2a61c0567fc0 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:22.864081155 +0000 UTC m=+9.146611033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs") pod "network-metrics-daemon-wtk5g" (UID: "35d777f2-4005-45e4-af04-2a61c0567fc0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:18.865175 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:18.864170 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:18.865175 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:18.864205 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret podName:ffdd3593-6091-4874-b55e-4049aa26ab60 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:22.864193823 +0000 UTC m=+9.146723705 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret") pod "global-pull-secret-syncer-k4k2w" (UID: "ffdd3593-6091-4874-b55e-4049aa26ab60") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:18.965494 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:18.964758 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpwt\" (UniqueName: \"kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt\") pod \"network-check-target-xchk6\" (UID: \"73012827-44d9-4cec-9f8d-40b2055c0c69\") " pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:18.965494 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:18.965017 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:18.965494 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:18.965038 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:18.965494 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:18.965053 2566 projected.go:196] Error preparing data for projected volume kube-api-access-5mpwt for pod openshift-network-diagnostics/network-check-target-xchk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:18.965494 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:18.965114 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt podName:73012827-44d9-4cec-9f8d-40b2055c0c69 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:22.965094243 +0000 UTC m=+9.247624125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mpwt" (UniqueName: "kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt") pod "network-check-target-xchk6" (UID: "73012827-44d9-4cec-9f8d-40b2055c0c69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:19.253572 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:19.252732 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:19.253572 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:19.252915 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:19.253572 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:19.253310 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:19.253572 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:19.253397 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:19.253572 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:19.253446 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:19.253572 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:19.253517 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:20.377901 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:20.377605 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4n84w" event={"ID":"32528057-07fa-4eb9-882f-a43c9cd36b31","Type":"ContainerStarted","Data":"c74d5923ed44d9a51b6f80ed33f7e5363cffa20d5fada5fcd8b717f1a536b710"} Oct 30 16:20:20.382188 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:20.382155 2566 generic.go:356] "Generic (PLEG): container finished" podID="4512b7bb-6729-4045-aac9-4a1ece176016" containerID="6888dc52e4bc627bcc56a608c369543d2dc3b8a6926020a8932ee87b47b8ea46" exitCode=0 Oct 30 16:20:20.382314 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:20.382252 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m72dq" event={"ID":"4512b7bb-6729-4045-aac9-4a1ece176016","Type":"ContainerDied","Data":"6888dc52e4bc627bcc56a608c369543d2dc3b8a6926020a8932ee87b47b8ea46"} Oct 30 16:20:20.387165 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:20.387138 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" event={"ID":"829ec3bbd4f4a9d896bac875cd5fa87a","Type":"ContainerStarted","Data":"a6103ccba7696cfa758e65f3829b41c4a1a814617135af748c6b8f2222494f7b"} Oct 30 16:20:20.390511 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:20.390484 2566 generic.go:356] "Generic (PLEG): container finished" podID="d7ecc0c6cd8524d6054106d923cd48af" containerID="a157ffac181ba6670e958e074630632719958af6aa11121031653652ddcaa994" exitCode=0 Oct 30 16:20:20.390612 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:20.390529 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" event={"ID":"d7ecc0c6cd8524d6054106d923cd48af","Type":"ContainerDied","Data":"a157ffac181ba6670e958e074630632719958af6aa11121031653652ddcaa994"} Oct 30 16:20:20.478401 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:20.478320 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4n84w" podStartSLOduration=0.951006569 podStartE2EDuration="5.478297279s" totalImagesPullingTime="4.52729071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:20.427209324 +0000 UTC m=+6.709739223" watchObservedRunningTime="2026-10-30 16:20:20.478297279 +0000 UTC m=+6.760827180" Oct 30 16:20:20.570559 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:20.570498 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" podStartSLOduration=5.5704763790000005 podStartE2EDuration="5.570476379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:20.569691176 +0000 UTC m=+6.852221078" watchObservedRunningTime="2026-10-30 16:20:20.570476379 +0000 UTC m=+6.853006281" Oct 30 16:20:21.252550 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:21.251580 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:21.252550 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:21.251734 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:21.252550 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:21.252186 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:21.252550 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:21.252311 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:21.252550 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:21.252379 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:21.252550 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:21.252472 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:21.398593 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:21.398548 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" event={"ID":"d7ecc0c6cd8524d6054106d923cd48af","Type":"ContainerStarted","Data":"c9afdce4e6fbfa1edf648258f1ad56686eb1d959ecae672bb763c1baf7d889c6"} Oct 30 16:20:22.403953 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:22.403916 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7w7js" event={"ID":"17726533-46eb-4d51-bc1d-b4e9959d7ba5","Type":"ContainerStarted","Data":"31c9600f117f09fc436dd2255098d5f2d92b7c5e427081eabbe08020f2089148"} Oct 30 16:20:22.452204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:22.452036 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" podStartSLOduration=7.452016136 podStartE2EDuration="7.452016136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:21.450999051 +0000 UTC m=+7.733528952" watchObservedRunningTime="2026-10-30 16:20:22.452016136 +0000 UTC m=+8.734546039" Oct 30 16:20:22.452204 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:22.452139 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7w7js" podStartSLOduration=1.3956216559999999 podStartE2EDuration="7.452132925s" totalImagesPullingTime="6.056511269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:22.452011924 +0000 UTC m=+8.734541822" watchObservedRunningTime="2026-10-30 16:20:22.452132925 +0000 UTC m=+8.734662824" Oct 30 16:20:22.905973 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:22.905923 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:22.906144 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:22.905984 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs\") pod \"network-metrics-daemon-wtk5g\" (UID: \"35d777f2-4005-45e4-af04-2a61c0567fc0\") " pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:22.906236 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:22.906184 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:22.906297 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:22.906254 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs podName:35d777f2-4005-45e4-af04-2a61c0567fc0 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:30.90623323 +0000 UTC m=+17.188763122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs") pod "network-metrics-daemon-wtk5g" (UID: "35d777f2-4005-45e4-af04-2a61c0567fc0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:22.906723 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:22.906700 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:22.906864 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:22.906766 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret podName:ffdd3593-6091-4874-b55e-4049aa26ab60 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:30.906748557 +0000 UTC m=+17.189278433 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret") pod "global-pull-secret-syncer-k4k2w" (UID: "ffdd3593-6091-4874-b55e-4049aa26ab60") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:22.957501 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:22.957251 2566 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4n84w" Oct 30 16:20:22.958708 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:22.958396 2566 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4n84w" Oct 30 16:20:23.008398 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:23.007129 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpwt\" (UniqueName: \"kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt\") pod \"network-check-target-xchk6\" (UID: \"73012827-44d9-4cec-9f8d-40b2055c0c69\") " pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:23.008398 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:23.007335 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:23.008398 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:23.007354 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:23.008398 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:23.007367 2566 projected.go:196] Error preparing data for projected volume kube-api-access-5mpwt for pod openshift-network-diagnostics/network-check-target-xchk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:23.008398 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:23.007424 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt podName:73012827-44d9-4cec-9f8d-40b2055c0c69 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:31.007405459 +0000 UTC m=+17.289935337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mpwt" (UniqueName: "kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt") pod "network-check-target-xchk6" (UID: "73012827-44d9-4cec-9f8d-40b2055c0c69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:23.252784 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:23.251932 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:23.252784 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:23.252038 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:23.252784 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:23.252100 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:23.252784 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:23.252158 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:23.252784 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:23.252199 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:23.252784 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:23.252262 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:23.410275 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:23.409650 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" event={"ID":"42aa85ec-676a-4f13-90f4-0fb13e94b22f","Type":"ContainerStarted","Data":"97c387bb9c6386419596791f60f6545195cbb1c84a6f9a7674d2ef9161a0b4c5"} Oct 30 16:20:24.025339 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:24.025237 2566 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 16:20:24.185097 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:24.184993 2566 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:20:24.025260338Z","UUID":"54599f4d-9a60-4611-bbdb-3b74d9f7ea03","Handler":null,"Name":"","Endpoint":""} Oct 30 16:20:24.192966 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:24.192948 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 16:20:24.193076 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:24.192976 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 16:20:24.414351 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:24.414305 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-99l8n" event={"ID":"b17ec352-725f-4bea-aaee-a3ca414fde6e","Type":"ContainerStarted","Data":"bc853149d3abe221dbccae797dab652454df431941d3594617e5ed053ce5013b"} Oct 30 16:20:24.419387 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:24.419307 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" event={"ID":"42aa85ec-676a-4f13-90f4-0fb13e94b22f","Type":"ContainerStarted","Data":"deff7b6becf8c0162ecb5face20ff1d47d919c07fd991edcd509b0e77700e4f1"} Oct 30 16:20:25.252512 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:25.252314 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:25.252680 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:25.252378 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:25.252680 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:25.252601 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:25.252829 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:25.252708 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:25.252829 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:25.252390 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:25.252924 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:25.252863 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:25.423911 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:25.423816 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" event={"ID":"42aa85ec-676a-4f13-90f4-0fb13e94b22f","Type":"ContainerStarted","Data":"8675684aa4f2084342bbdf4f4327102504a4c159bd18f9e702044e40491541c7"} Oct 30 16:20:25.425985 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:25.425947 2566 generic.go:356] "Generic (PLEG): container finished" podID="4512b7bb-6729-4045-aac9-4a1ece176016" containerID="e5ed0c0a4778077480bf47e0fa2d282b68cc9d9e616ec1738574b035160bf988" exitCode=0 Oct 30 16:20:25.426119 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:25.426041 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m72dq" event={"ID":"4512b7bb-6729-4045-aac9-4a1ece176016","Type":"ContainerDied","Data":"e5ed0c0a4778077480bf47e0fa2d282b68cc9d9e616ec1738574b035160bf988"} Oct 30 16:20:25.428120 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:25.428098 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lr2cd" event={"ID":"ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc","Type":"ContainerStarted","Data":"51a630209c4c6397b7db1f4c25a9200dff9803e29d7d2a3f7ceced66a71b814d"} Oct 30 16:20:25.470357 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:25.470311 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" podStartSLOduration=1.283721701 podStartE2EDuration="10.470297481s" totalImagesPullingTime="9.18657578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:25.470042418 +0000 UTC m=+11.752572317" watchObservedRunningTime="2026-10-30 16:20:25.470297481 +0000 UTC m=+11.752827380" Oct 30 16:20:25.470508 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:25.470426 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-99l8n" podStartSLOduration=1.8561767470000001 podStartE2EDuration="10.470421893s" totalImagesPullingTime="8.614245146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:24.462372387 +0000 UTC m=+10.744902299" watchObservedRunningTime="2026-10-30 16:20:25.470421893 +0000 UTC m=+11.752951792" Oct 30 16:20:25.561578 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:25.561512 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-lr2cd" podStartSLOduration=1.9428765380000002 podStartE2EDuration="10.561491389s" totalImagesPullingTime="8.618614851s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:25.560896529 +0000 UTC m=+11.843426427" watchObservedRunningTime="2026-10-30 16:20:25.561491389 +0000 UTC m=+11.844021290" Oct 30 16:20:26.433491 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:26.433412 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p694c" event={"ID":"c1bbf041-5696-4a2c-94a8-69ab0e995096","Type":"ContainerStarted","Data":"2805a8d5fe2ea1fb4a7ac091ac39d97cd38e1be5ad106c61a8f42d459d3cd31e"} Oct 30 16:20:26.435811 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:26.435786 2566 generic.go:356] "Generic (PLEG): container finished" podID="4512b7bb-6729-4045-aac9-4a1ece176016" containerID="aefdfe161a3994246f1b18c4d79933e24c86207aaddb0d078cf06396c40a149b" exitCode=0 Oct 30 16:20:26.435925 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:26.435852 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m72dq" event={"ID":"4512b7bb-6729-4045-aac9-4a1ece176016","Type":"ContainerDied","Data":"aefdfe161a3994246f1b18c4d79933e24c86207aaddb0d078cf06396c40a149b"} Oct 30 16:20:26.475713 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:26.475663 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-p694c" podStartSLOduration=2.599955799 podStartE2EDuration="11.47564714s" totalImagesPullingTime="8.875691341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:26.47524941 +0000 UTC m=+12.757779305" watchObservedRunningTime="2026-10-30 16:20:26.47564714 +0000 UTC m=+12.758177038" Oct 30 16:20:27.251889 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:27.251854 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:27.252027 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:27.251895 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:27.252027 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:27.252002 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:27.252111 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:27.252057 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:27.252159 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:27.252117 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:27.252213 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:27.252196 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:27.440595 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:27.440509 2566 generic.go:356] "Generic (PLEG): container finished" podID="4512b7bb-6729-4045-aac9-4a1ece176016" containerID="c86b03b18b60610ee9b894a0cf19f0c743462ace4b95c4544bcb27a617a5e282" exitCode=0 Oct 30 16:20:27.440595 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:27.440549 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m72dq" event={"ID":"4512b7bb-6729-4045-aac9-4a1ece176016","Type":"ContainerDied","Data":"c86b03b18b60610ee9b894a0cf19f0c743462ace4b95c4544bcb27a617a5e282"} Oct 30 16:20:27.442907 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:27.442883 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-868tl" event={"ID":"e7b597f3-5dc9-45e6-840f-b492ba06c241","Type":"ContainerStarted","Data":"ecadeaa5bde66cfd330a7a045591f5cd3b34e3b5197da752fedac2ab5ac67acc"} Oct 30 16:20:27.587846 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:27.587799 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-868tl" podStartSLOduration=1.5876053319999999 podStartE2EDuration="12.587785551s" totalImagesPullingTime="11.000180219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:27.58667087 +0000 UTC m=+13.869200777" watchObservedRunningTime="2026-10-30 16:20:27.587785551 +0000 UTC m=+13.870315443" Oct 30 16:20:29.251787 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:29.251735 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:29.252293 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:29.251755 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:29.252293 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:29.251889 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:29.252293 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:29.251899 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:29.252293 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:29.251980 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:29.252293 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:29.252079 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:29.451535 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:29.451354 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m72dq" event={"ID":"4512b7bb-6729-4045-aac9-4a1ece176016","Type":"ContainerStarted","Data":"e07268ef7214a1ba6541b92b70500c8c440dc1bbbd58fb1e51f1bf32e8de0693"} Oct 30 16:20:30.457442 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:30.457404 2566 generic.go:356] "Generic (PLEG): container finished" podID="4512b7bb-6729-4045-aac9-4a1ece176016" containerID="e07268ef7214a1ba6541b92b70500c8c440dc1bbbd58fb1e51f1bf32e8de0693" exitCode=0 Oct 30 16:20:30.457869 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:30.457459 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m72dq" event={"ID":"4512b7bb-6729-4045-aac9-4a1ece176016","Type":"ContainerDied","Data":"e07268ef7214a1ba6541b92b70500c8c440dc1bbbd58fb1e51f1bf32e8de0693"} Oct 30 16:20:30.974089 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:30.974046 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:30.974089 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:30.974101 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs\") pod \"network-metrics-daemon-wtk5g\" (UID: \"35d777f2-4005-45e4-af04-2a61c0567fc0\") " pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:30.974299 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:30.974196 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:30.974299 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:30.974196 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:30.974299 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:30.974262 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs podName:35d777f2-4005-45e4-af04-2a61c0567fc0 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:46.974245109 +0000 UTC m=+33.256775006 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs") pod "network-metrics-daemon-wtk5g" (UID: "35d777f2-4005-45e4-af04-2a61c0567fc0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:30.974299 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:30.974281 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret podName:ffdd3593-6091-4874-b55e-4049aa26ab60 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:46.974272314 +0000 UTC m=+33.256802190 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret") pod "global-pull-secret-syncer-k4k2w" (UID: "ffdd3593-6091-4874-b55e-4049aa26ab60") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:31.075091 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:31.075056 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpwt\" (UniqueName: \"kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt\") pod \"network-check-target-xchk6\" (UID: \"73012827-44d9-4cec-9f8d-40b2055c0c69\") " pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:31.075263 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:31.075242 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:31.075323 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:31.075270 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:31.075323 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:31.075282 2566 projected.go:196] Error preparing data for projected volume kube-api-access-5mpwt for pod openshift-network-diagnostics/network-check-target-xchk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:31.075405 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:31.075347 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt podName:73012827-44d9-4cec-9f8d-40b2055c0c69 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:47.075328836 +0000 UTC m=+33.357858727 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mpwt" (UniqueName: "kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt") pod "network-check-target-xchk6" (UID: "73012827-44d9-4cec-9f8d-40b2055c0c69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:31.251450 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:31.251378 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:31.251450 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:31.251409 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:31.251641 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:31.251527 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:31.251641 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:31.251528 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:31.251641 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:31.251610 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:31.251758 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:31.251655 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:31.462002 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:31.461969 2566 generic.go:356] "Generic (PLEG): container finished" podID="4512b7bb-6729-4045-aac9-4a1ece176016" containerID="958ab40ebef67e47dce9b9671eb69a276b14b2b4b8429d80ab012ed43e75e815" exitCode=0 Oct 30 16:20:31.462002 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:31.462003 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m72dq" event={"ID":"4512b7bb-6729-4045-aac9-4a1ece176016","Type":"ContainerDied","Data":"958ab40ebef67e47dce9b9671eb69a276b14b2b4b8429d80ab012ed43e75e815"} Oct 30 16:20:32.469970 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:32.469640 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m72dq" event={"ID":"4512b7bb-6729-4045-aac9-4a1ece176016","Type":"ContainerStarted","Data":"ce9e112db6d2cbe599dce9b0c36ba1420b50ce6c822709a6ca946f867cc63c56"} Oct 30 16:20:32.472907 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:32.472883 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" event={"ID":"25906571-b2ad-4134-9674-e67842de154d","Type":"ContainerStarted","Data":"9d3376ed3088da1a668d2b5dd68217340173e52d67a5405b7c0d57f53fbb269b"} Oct 30 16:20:32.473010 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:32.472944 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" event={"ID":"25906571-b2ad-4134-9674-e67842de154d","Type":"ContainerStarted","Data":"712fa8342a3e07c1da244267e1e45c75141646f0335de9f0c08bd14ffeaa7fac"} Oct 30 16:20:32.473010 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:32.472972 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" event={"ID":"25906571-b2ad-4134-9674-e67842de154d","Type":"ContainerStarted","Data":"fcc4ef2ef5b13153baa120e552d62616fadf09a0a65c273173ee3aa0963c9578"} Oct 30 16:20:32.473010 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:32.472997 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" event={"ID":"25906571-b2ad-4134-9674-e67842de154d","Type":"ContainerStarted","Data":"59d0cc12169b08558d0b44c75a9897ef59f31607f1472dc4ce84fd3f7e0512bf"} Oct 30 16:20:32.473096 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:32.473022 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" event={"ID":"25906571-b2ad-4134-9674-e67842de154d","Type":"ContainerStarted","Data":"13c3f48d4186c27a5176a77d87d255adddad1a110f47dc3d1f6506a3aa0a5e2c"} Oct 30 16:20:32.473096 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:32.473048 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" event={"ID":"25906571-b2ad-4134-9674-e67842de154d","Type":"ContainerStarted","Data":"ebd55ac7016a9da2aecd794459dab92aca995e415677e1f93a2ccc4aa9bace54"} Oct 30 16:20:32.534945 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:32.534900 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m72dq" podStartSLOduration=6.363846863 podStartE2EDuration="17.534886796s" totalImagesPullingTime="11.171039933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:32.534101354 +0000 UTC m=+18.816631249" watchObservedRunningTime="2026-10-30 16:20:32.534886796 +0000 UTC m=+18.817416691" Oct 30 16:20:33.117839 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:33.117804 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4n84w" Oct 30 16:20:33.117982 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:33.117936 2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:20:33.118344 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:33.118327 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4n84w" Oct 30 16:20:33.251896 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:33.251863 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:33.251896 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:33.251888 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:33.252065 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:33.251955 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:33.252065 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:33.251989 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:33.252120 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:33.252066 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:33.252153 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:33.252139 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:33.731243 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:33.731209 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:33.885171 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:33.885148 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:34.054635 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:34.054554 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:34.478995 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:34.478966 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" event={"ID":"25906571-b2ad-4134-9674-e67842de154d","Type":"ContainerStarted","Data":"fbbdf25ad01d54f18c2119a2321b2060f2555030305be8423c222d63c90a1961"} Oct 30 16:20:34.841096 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:34.841059 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:35.251894 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:35.251813 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:35.251894 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:35.251826 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:35.252084 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:35.251935 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:35.252084 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:35.251937 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:35.252084 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:35.252033 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:35.252185 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:35.252106 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:35.521036 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:35.520953 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:36.486890 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:36.486855 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" event={"ID":"25906571-b2ad-4134-9674-e67842de154d","Type":"ContainerStarted","Data":"ca384d2de9a0aa58e5e2341da92c628ed5d45113a220554653135a1495664ad7"} Oct 30 16:20:36.490222 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:36.487550 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:36.506102 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:36.506049 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:36.585946 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:36.585897 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" podStartSLOduration=5.357147865 podStartE2EDuration="21.585885141s" totalImagesPullingTime="16.228737276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:36.58570703 +0000 UTC m=+22.868236923" watchObservedRunningTime="2026-10-30 16:20:36.585885141 +0000 UTC m=+22.868415038" Oct 30 16:20:37.251685 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:37.251645 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:37.251947 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:37.251645 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:37.251947 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:37.251758 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:37.251947 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:37.251660 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:37.251947 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:37.251886 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:37.251947 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:37.251944 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:37.489728 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:37.489700 2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:20:37.490296 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:37.490151 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:37.514595 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:37.514523 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:37.690485 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:37.690450 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:37.798295 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:37.798221 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:38.380941 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:38.380667 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xchk6"] Oct 30 16:20:38.381089 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:38.380942 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:38.381089 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:38.381030 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:38.383911 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:38.383654 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k4k2w"] Oct 30 16:20:38.383911 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:38.383759 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:38.383911 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:38.383881 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:38.384331 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:38.384313 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wtk5g"] Oct 30 16:20:38.384401 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:38.384391 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:38.384485 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:38.384469 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:38.491836 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:38.491815 2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:20:38.978185 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:38.978154 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:39.086846 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:39.086816 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:39.365712 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:39.365683 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:39.493951 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:39.493916 2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:20:39.500850 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:39.500789 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:39.639344 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:39.639246 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:39.835598 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:39.835564 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:39.964886 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:39.964752 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:40.252039 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:40.251952 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:40.252039 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:40.251986 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:40.252236 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:40.252069 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:40.252236 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:40.251954 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:40.252236 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:40.252177 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:40.252372 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:40.252260 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:40.691057 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:40.691022 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:40.847423 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:40.847381 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:41.290375 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:41.290331 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:41.677888 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:41.677851 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:20:41.678071 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:41.678047 2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:20:41.689107 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:41.689072 2566 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" podUID="25906571-b2ad-4134-9674-e67842de154d" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 16:20:41.697738 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:41.697714 2566 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" podUID="25906571-b2ad-4134-9674-e67842de154d" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 16:20:42.251705 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:42.251677 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:42.251901 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:42.251676 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:42.251901 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:42.251808 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:42.251901 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:42.251676 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:42.251901 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:42.251861 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:42.252121 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:42.251934 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:42.292822 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:42.292799 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lr2cd_ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc/dns-node-resolver/0.log" Oct 30 16:20:42.447338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:42.447306 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:42.587275 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:42.587181 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:42.719811 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:42.719776 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:43.099694 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:43.099674 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7w7js_17726533-46eb-4d51-bc1d-b4e9959d7ba5/node-ca/0.log" Oct 30 16:20:43.104177 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:43.104154 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:43.870262 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:43.870229 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:43.982709 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:43.982677 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:44.252526 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:44.252447 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:44.252681 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:44.252570 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:44.252681 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:44.252608 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:44.252681 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:44.252648 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:44.252867 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:44.252724 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:44.252867 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:44.252802 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:44.302317 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:44.302287 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:44.448249 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:44.448218 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:44.565882 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:44.565808 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:44.681936 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:44.681906 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:45.095716 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:45.095685 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:45.216491 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:45.216461 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:45.875525 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:45.875490 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:45.993933 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:45.993901 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:46.110209 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:46.110181 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:46.252148 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:46.252066 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:46.252313 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:46.252178 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:46.252313 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:46.252189 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:46.252313 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:46.252266 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:46.252313 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:46.252289 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:46.252502 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:46.252444 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:46.285117 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:46.285085 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:46.427336 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:46.427309 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:46.569933 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:46.569855 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:46.703090 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:46.703057 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:47.000926 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:47.000891 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:47.000926 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:47.000927 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs\") pod \"network-metrics-daemon-wtk5g\" (UID: \"35d777f2-4005-45e4-af04-2a61c0567fc0\") " pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:47.001172 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:47.001045 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:47.001172 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:47.001045 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:47.001172 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:47.001106 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret podName:ffdd3593-6091-4874-b55e-4049aa26ab60 nodeName:}" failed. No retries permitted until 2026-10-30 16:21:19.001091014 +0000 UTC m=+65.283620898 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret") pod "global-pull-secret-syncer-k4k2w" (UID: "ffdd3593-6091-4874-b55e-4049aa26ab60") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:47.001172 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:47.001119 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs podName:35d777f2-4005-45e4-af04-2a61c0567fc0 nodeName:}" failed. No retries permitted until 2026-10-30 16:21:19.001112936 +0000 UTC m=+65.283642810 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs") pod "network-metrics-daemon-wtk5g" (UID: "35d777f2-4005-45e4-af04-2a61c0567fc0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:47.072889 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:47.072846 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:47.101950 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:47.101925 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpwt\" (UniqueName: \"kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt\") pod \"network-check-target-xchk6\" (UID: \"73012827-44d9-4cec-9f8d-40b2055c0c69\") " pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:47.102083 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:47.102071 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:47.102126 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:47.102088 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:47.102126 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:47.102097 2566 projected.go:196] Error preparing data for projected volume kube-api-access-5mpwt for pod openshift-network-diagnostics/network-check-target-xchk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:47.102195 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:47.102140 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt podName:73012827-44d9-4cec-9f8d-40b2055c0c69 nodeName:}" failed. No retries permitted until 2026-10-30 16:21:19.102129532 +0000 UTC m=+65.384659406 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mpwt" (UniqueName: "kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt") pod "network-check-target-xchk6" (UID: "73012827-44d9-4cec-9f8d-40b2055c0c69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:47.871445 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:47.871415 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:48.251500 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:48.251418 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:48.251644 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:48.251519 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:48.251644 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:48.251546 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:48.251644 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:48.251428 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:48.251747 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:48.251650 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:48.251747 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:48.251722 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:48.272105 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:48.272054 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:48.503656 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:48.503576 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:48.663584 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:48.663546 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:48.780812 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:48.780727 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:49.076805 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:49.076758 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:49.188288 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:49.188256 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:49.870016 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:49.869984 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:50.252277 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:50.252196 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:50.252277 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:50.252202 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:50.252804 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:50.252297 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:50.252804 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:50.252366 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:50.252804 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:50.252399 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:50.252804 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:50.252457 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:50.269804 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:50.269780 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:50.468600 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:50.468565 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:50.672025 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:50.671991 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:51.074296 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:51.074266 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:51.867595 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:51.867563 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:52.252080 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:52.252007 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:52.252080 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:52.252026 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:52.252265 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:52.252118 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:52.252265 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:52.252168 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:52.252339 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:52.252260 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:52.252339 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:52.252297 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:52.303896 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:52.303866 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:52.474610 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:52.474582 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:52.588868 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:52.588831 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:52.698358 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:52.698329 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:52.908161 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:52.908059 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:53.021410 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:53.021338 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:53.130473 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:53.130438 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:53.269555 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:53.269466 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:54.252093 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:54.252066 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:54.252541 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:54.252183 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:54.252541 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:54.252194 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:54.252541 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:54.252280 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:54.252541 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:54.252347 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:54.252541 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:54.252379 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:56.252183 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:56.252147 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:56.252183 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:56.252180 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:56.252730 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:56.252266 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:56.252730 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:56.252333 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:56.252730 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:56.252447 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:56.252730 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:56.252537 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:57.332043 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:57.332009 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:57.441717 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:57.441690 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:57.552831 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:57.552799 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:57.884019 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:57.883986 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:58.031398 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:58.031365 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:58.251476 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:58.251397 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:20:58.251476 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:58.251413 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:20:58.251688 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:58.251403 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:20:58.251688 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:58.251503 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:20:58.251688 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:58.251566 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:20:58.251688 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:20:58.251639 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:20:59.158415 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:59.158384 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:20:59.267784 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:20:59.267734 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:00.252068 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:00.252034 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:21:00.252553 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:00.252072 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:21:00.252553 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:21:00.252120 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:21:00.252553 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:21:00.252169 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:21:00.252553 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:00.252226 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:21:00.252553 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:21:00.252288 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:21:01.925539 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:01.925502 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:02.040456 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:02.040417 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:02.252307 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:02.252223 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:21:02.252307 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:02.252249 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:21:02.252515 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:02.252228 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:21:02.252515 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:21:02.252326 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtk5g" podUID="35d777f2-4005-45e4-af04-2a61c0567fc0" Oct 30 16:21:02.252515 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:21:02.252396 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xchk6" podUID="73012827-44d9-4cec-9f8d-40b2055c0c69" Oct 30 16:21:02.252515 ip-10-0-129-65 kubenswrapper[2566]: E1030 16:21:02.252455 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k4k2w" podUID="ffdd3593-6091-4874-b55e-4049aa26ab60" Oct 30 16:21:02.978552 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:02.978522 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeReady" Oct 30 16:21:02.979040 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:02.978646 2566 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 16:21:03.046354 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.046317 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xrskd"] Oct 30 16:21:03.048203 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.048186 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ckz2r"] Oct 30 16:21:03.048339 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.048322 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:03.049738 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.049719 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ckz2r" Oct 30 16:21:03.065585 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.065566 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s8kck\"" Oct 30 16:21:03.068990 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.068967 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fd5ww"] Oct 30 16:21:03.070608 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.070593 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:03.075142 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.071884 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 16:21:03.075142 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.072219 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 16:21:03.075142 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.075135 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ckz2r"] Oct 30 16:21:03.075605 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.075584 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 16:21:03.076964 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.076906 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xrskd"] Oct 30 16:21:03.101545 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.101516 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fd5ww"] Oct 30 16:21:03.116677 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.116655 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-data-volume\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:03.116801 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.116684 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:03.116801 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.116702 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pjpq\" (UniqueName: \"kubernetes.io/projected/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-kube-api-access-5pjpq\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:03.116801 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.116731 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86634960-264a-48a7-969a-5fbe6e58447e-metrics-tls\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:03.116964 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.116889 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86634960-264a-48a7-969a-5fbe6e58447e-config-volume\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:03.116964 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.116927 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-crio-socket\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:03.117029 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.116987 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86634960-264a-48a7-969a-5fbe6e58447e-tmp-dir\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:03.117029 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.117009 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd7dac20-991b-4048-acd5-08dcbdb57892-cert\") pod \"ingress-canary-ckz2r\" (UID: \"cd7dac20-991b-4048-acd5-08dcbdb57892\") " pod="openshift-ingress-canary/ingress-canary-ckz2r" Oct 30 16:21:03.117100 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.117032 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:03.117100 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.117052 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f28s\" (UniqueName: \"kubernetes.io/projected/86634960-264a-48a7-969a-5fbe6e58447e-kube-api-access-5f28s\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:03.117100 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.117068 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd4m6\" (UniqueName: \"kubernetes.io/projected/cd7dac20-991b-4048-acd5-08dcbdb57892-kube-api-access-vd4m6\") pod \"ingress-canary-ckz2r\" (UID: \"cd7dac20-991b-4048-acd5-08dcbdb57892\") " pod="openshift-ingress-canary/ingress-canary-ckz2r" Oct 30 16:21:03.166830 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.166801 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 16:21:03.166830 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.166820 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 16:21:03.166966 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.166864 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-m9dmv\"" Oct 30 16:21:03.166966 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.166879 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 16:21:03.166966 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.166890 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6q4zp\"" Oct 30 16:21:03.171997 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.171980 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 16:21:03.172072 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.172038 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 16:21:03.172108 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.172096 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 16:21:03.217363 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.217334 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-data-volume\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:03.217482 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.217369 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:03.217482 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.217391 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pjpq\" (UniqueName: \"kubernetes.io/projected/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-kube-api-access-5pjpq\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:03.217650 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.217546 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86634960-264a-48a7-969a-5fbe6e58447e-metrics-tls\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:03.217650 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.217607 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86634960-264a-48a7-969a-5fbe6e58447e-config-volume\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:03.217650 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.217641 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-crio-socket\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:03.217836 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.217667 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86634960-264a-48a7-969a-5fbe6e58447e-tmp-dir\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:03.217836 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.217694 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd7dac20-991b-4048-acd5-08dcbdb57892-cert\") pod \"ingress-canary-ckz2r\" (UID: \"cd7dac20-991b-4048-acd5-08dcbdb57892\") " pod="openshift-ingress-canary/ingress-canary-ckz2r" Oct 30 16:21:03.217836 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.217787 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:03.217836 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.217817 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-data-volume\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:03.217836 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.217833 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f28s\" (UniqueName: \"kubernetes.io/projected/86634960-264a-48a7-969a-5fbe6e58447e-kube-api-access-5f28s\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:03.218010 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.217863 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd4m6\" (UniqueName: \"kubernetes.io/projected/cd7dac20-991b-4048-acd5-08dcbdb57892-kube-api-access-vd4m6\") pod \"ingress-canary-ckz2r\" (UID: \"cd7dac20-991b-4048-acd5-08dcbdb57892\") " pod="openshift-ingress-canary/ingress-canary-ckz2r" Oct 30 16:21:03.218010 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.217907 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-crio-socket\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:03.219685 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.219661 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86634960-264a-48a7-969a-5fbe6e58447e-tmp-dir\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:03.221194 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.221176 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86634960-264a-48a7-969a-5fbe6e58447e-config-volume\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:03.221194 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.221185 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:03.221902 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.221887 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86634960-264a-48a7-969a-5fbe6e58447e-metrics-tls\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:03.221951 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.221887 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:03.223079 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.223054 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd7dac20-991b-4048-acd5-08dcbdb57892-cert\") pod \"ingress-canary-ckz2r\" (UID: \"cd7dac20-991b-4048-acd5-08dcbdb57892\") " pod="openshift-ingress-canary/ingress-canary-ckz2r" Oct 30 16:21:03.242590 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.242535 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f28s\" (UniqueName: \"kubernetes.io/projected/86634960-264a-48a7-969a-5fbe6e58447e-kube-api-access-5f28s\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:03.242666 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.242626 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pjpq\" (UniqueName: \"kubernetes.io/projected/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-kube-api-access-5pjpq\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:03.242779 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.242750 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd4m6\" (UniqueName: \"kubernetes.io/projected/cd7dac20-991b-4048-acd5-08dcbdb57892-kube-api-access-vd4m6\") pod \"ingress-canary-ckz2r\" (UID: \"cd7dac20-991b-4048-acd5-08dcbdb57892\") " pod="openshift-ingress-canary/ingress-canary-ckz2r" Oct 30 16:21:03.361068 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.361034 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:03.363833 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.363818 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ckz2r" Oct 30 16:21:03.386601 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.386572 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:03.664577 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.664549 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:03.676412 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.676391 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xrskd"] Oct 30 16:21:03.676997 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:21:03.676973 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86634960_264a_48a7_969a_5fbe6e58447e.slice/crio-092b4c87f40344e29f6960c03f2d7fdbc986bc2222db5324fde6d7c135ac2683 WatchSource:0}: Error finding container 092b4c87f40344e29f6960c03f2d7fdbc986bc2222db5324fde6d7c135ac2683: Status 404 returned error can't find the container with id 092b4c87f40344e29f6960c03f2d7fdbc986bc2222db5324fde6d7c135ac2683 Oct 30 16:21:03.684379 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.684359 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ckz2r"] Oct 30 16:21:03.685664 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:21:03.685639 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7dac20_991b_4048_acd5_08dcbdb57892.slice/crio-e181cbd1daf8abf4db197ac2ae288fa1833186a1c601945425f04c61bac34840 WatchSource:0}: Error finding container e181cbd1daf8abf4db197ac2ae288fa1833186a1c601945425f04c61bac34840: Status 404 returned error can't find the container with id e181cbd1daf8abf4db197ac2ae288fa1833186a1c601945425f04c61bac34840 Oct 30 16:21:03.698751 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:21:03.698724 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfe68e7_95cd_46c9_ae4a_e2207e61b9c5.slice/crio-80f44efa402a758d3adac4a4a32036485bd2b2370cac6cfb091fc80ea6a343d8 WatchSource:0}: Error finding container 80f44efa402a758d3adac4a4a32036485bd2b2370cac6cfb091fc80ea6a343d8: Status 404 returned error can't find the container with id 80f44efa402a758d3adac4a4a32036485bd2b2370cac6cfb091fc80ea6a343d8 Oct 30 16:21:03.699078 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.699062 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fd5ww"] Oct 30 16:21:03.781352 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:03.781326 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:04.253091 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:04.253061 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:21:04.254175 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:04.253416 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:21:04.254175 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:04.253605 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:21:04.270724 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:04.270701 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:21:04.270724 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:04.270703 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d8rcz\"" Oct 30 16:21:04.270913 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:04.270753 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:21:04.284322 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:04.284299 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-m64dw\"" Oct 30 16:21:04.284479 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:04.284360 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:21:04.284479 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:04.284389 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:21:04.284479 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:04.284408 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-46f47\"" Oct 30 16:21:04.329064 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:04.329035 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lr2cd_ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc/dns-node-resolver/0.log" Oct 30 16:21:04.547369 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:04.547289 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fd5ww" event={"ID":"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5","Type":"ContainerStarted","Data":"e9cd127763d1c7c0f84362c8a6c40203016fed3577d4303d69b3e5e13dbdaf44"} Oct 30 16:21:04.547519 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:04.547394 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fd5ww" event={"ID":"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5","Type":"ContainerStarted","Data":"942e7f40e4b118dc513d80f7d2713711abde6daecaad622c0611d2c0cd89bfa5"} Oct 30 16:21:04.547519 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:04.547423 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fd5ww" event={"ID":"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5","Type":"ContainerStarted","Data":"80f44efa402a758d3adac4a4a32036485bd2b2370cac6cfb091fc80ea6a343d8"} Oct 30 16:21:04.548578 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:04.548543 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ckz2r" event={"ID":"cd7dac20-991b-4048-acd5-08dcbdb57892","Type":"ContainerStarted","Data":"e181cbd1daf8abf4db197ac2ae288fa1833186a1c601945425f04c61bac34840"} Oct 30 16:21:04.549999 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:04.549971 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xrskd" event={"ID":"86634960-264a-48a7-969a-5fbe6e58447e","Type":"ContainerStarted","Data":"092b4c87f40344e29f6960c03f2d7fdbc986bc2222db5324fde6d7c135ac2683"} Oct 30 16:21:05.179646 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.179617 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:05.299896 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.299852 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl"] Oct 30 16:21:05.302250 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.302226 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:05.321912 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.321888 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 16:21:05.324051 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.324033 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 16:21:05.334227 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.334190 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7w7js_17726533-46eb-4d51-bc1d-b4e9959d7ba5/node-ca/0.log" Oct 30 16:21:05.335646 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.335619 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:05.335748 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.335652 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:05.335748 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.335674 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c8t5\" (UniqueName: \"kubernetes.io/projected/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-kube-api-access-9c8t5\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:05.335748 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.335728 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:05.336048 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.336036 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qlnbk\"" Oct 30 16:21:05.336145 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.336133 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 16:21:05.336214 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.336202 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 16:21:05.337206 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.337169 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:05.348120 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.348061 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 16:21:05.351095 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.351070 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl"] Oct 30 16:21:05.436088 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.436013 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:05.436088 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.436078 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:05.436285 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.436111 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:05.436285 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.436148 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c8t5\" (UniqueName: \"kubernetes.io/projected/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-kube-api-access-9c8t5\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:05.436839 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.436814 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:05.439241 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.439219 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:05.439366 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.439309 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:05.449820 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.449787 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:05.461541 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.461511 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c8t5\" (UniqueName: \"kubernetes.io/projected/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-kube-api-access-9c8t5\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:05.553125 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.553058 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xrskd" event={"ID":"86634960-264a-48a7-969a-5fbe6e58447e","Type":"ContainerStarted","Data":"1571f36e70693722b2f5fa00aa966b6ca7039074624ea89a7035a0277d37f1ab"} Oct 30 16:21:05.616861 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.616825 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:05.918413 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:05.918383 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl"] Oct 30 16:21:05.920133 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:21:05.920102 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dcde09b_ef11_4bb9_810b_883b0ee4cc18.slice/crio-dcf9b8eaa9ca92fd91eb3158868a0ce4d581a0af752e56faf0b65bbc1d0152fe WatchSource:0}: Error finding container dcf9b8eaa9ca92fd91eb3158868a0ce4d581a0af752e56faf0b65bbc1d0152fe: Status 404 returned error can't find the container with id dcf9b8eaa9ca92fd91eb3158868a0ce4d581a0af752e56faf0b65bbc1d0152fe Oct 30 16:21:06.557175 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:06.557142 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ckz2r" event={"ID":"cd7dac20-991b-4048-acd5-08dcbdb57892","Type":"ContainerStarted","Data":"3c5340331edf6a8e8c1859e31ad785db7472bfbb3bb6dd38c257eaa9f2778187"} Oct 30 16:21:06.559314 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:06.559272 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fd5ww" event={"ID":"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5","Type":"ContainerStarted","Data":"1d8a5a7f3a8b8ba53cccbb81a01c68d8145d20dc1c0e2f528241e05780f5dca0"} Oct 30 16:21:06.561095 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:06.561070 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xrskd" event={"ID":"86634960-264a-48a7-969a-5fbe6e58447e","Type":"ContainerStarted","Data":"594122db0a002f21db9a0352c86bdc3e367dbcacc2654aa236434fd9cd74088b"} Oct 30 16:21:06.561222 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:06.561204 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:06.562214 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:06.562194 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" event={"ID":"0dcde09b-ef11-4bb9-810b-883b0ee4cc18","Type":"ContainerStarted","Data":"dcf9b8eaa9ca92fd91eb3158868a0ce4d581a0af752e56faf0b65bbc1d0152fe"} Oct 30 16:21:06.600120 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:06.600080 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ckz2r" podStartSLOduration=0.914137439 podStartE2EDuration="3.600067618s" totalImagesPullingTime="2.685930179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:06.599447689 +0000 UTC m=+52.881977585" watchObservedRunningTime="2026-10-30 16:21:06.600067618 +0000 UTC m=+52.882597513" Oct 30 16:21:06.644695 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:06.644658 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fd5ww" podStartSLOduration=1.4879159610000001 podStartE2EDuration="3.644645008s" totalImagesPullingTime="2.156729047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:21:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:06.64424657 +0000 UTC m=+52.926776466" watchObservedRunningTime="2026-10-30 16:21:06.644645008 +0000 UTC m=+52.927174901" Oct 30 16:21:06.689468 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:06.689403 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-xrskd" podStartSLOduration=1.9144714889999999 podStartE2EDuration="3.689387284s" totalImagesPullingTime="1.774915795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:06.688705854 +0000 UTC m=+52.971235749" watchObservedRunningTime="2026-10-30 16:21:06.689387284 +0000 UTC m=+52.971917181" Oct 30 16:21:07.567108 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:07.567073 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" event={"ID":"0dcde09b-ef11-4bb9-810b-883b0ee4cc18","Type":"ContainerStarted","Data":"e1fbf815d00738f29b21d26bcbb95723cc72dd6323d8f92e1a6645c4670eaa36"} Oct 30 16:21:07.567495 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:07.567156 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" event={"ID":"0dcde09b-ef11-4bb9-810b-883b0ee4cc18","Type":"ContainerStarted","Data":"589241ee35988881775a48e0cac3cd0898fd32b5c8d1e1120423d46d7b1d11df"} Oct 30 16:21:07.610967 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:07.610923 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" podStartSLOduration=1.128375104 podStartE2EDuration="2.61091213s" totalImagesPullingTime="1.482537026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:07.610569837 +0000 UTC m=+53.893099733" watchObservedRunningTime="2026-10-30 16:21:07.61091213 +0000 UTC m=+53.893442025" Oct 30 16:21:07.727614 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:07.727542 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:07.852625 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:07.852590 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:08.329338 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:08.329307 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:09.117291 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:09.117252 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:09.226737 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:09.226707 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:09.651945 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:09.651917 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:09.851992 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:09.851959 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:09.965713 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:09.965617 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dktqn"] Oct 30 16:21:09.967781 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:09.967753 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:09.981956 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:09.981934 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:10.004716 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.004688 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 16:21:10.004716 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.004688 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-95j9k\"" Oct 30 16:21:10.004716 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.004717 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 16:21:10.035395 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.035373 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 16:21:10.073960 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.073932 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.073960 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.073960 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-accelerators-collector-config\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.074090 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.073983 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-tls\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.074090 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.074003 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9031958e-b627-4334-9e47-b9b62862c805-metrics-client-ca\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.074090 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.074063 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9031958e-b627-4334-9e47-b9b62862c805-root\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.074180 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.074122 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9031958e-b627-4334-9e47-b9b62862c805-sys\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.074238 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.074218 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-wtmp\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.074279 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.074267 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-textfile\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.074324 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.074312 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsbds\" (UniqueName: \"kubernetes.io/projected/9031958e-b627-4334-9e47-b9b62862c805-kube-api-access-fsbds\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.098819 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.098791 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:10.175384 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.175359 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-wtmp\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.175756 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.175404 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-textfile\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.175756 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.175427 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsbds\" (UniqueName: \"kubernetes.io/projected/9031958e-b627-4334-9e47-b9b62862c805-kube-api-access-fsbds\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.175756 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.175449 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.175756 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.175466 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-accelerators-collector-config\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.175756 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.175524 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-wtmp\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.175756 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.175597 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-tls\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.175756 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.175631 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9031958e-b627-4334-9e47-b9b62862c805-metrics-client-ca\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.175756 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.175650 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9031958e-b627-4334-9e47-b9b62862c805-root\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.175756 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.175669 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9031958e-b627-4334-9e47-b9b62862c805-sys\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.175756 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.175759 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9031958e-b627-4334-9e47-b9b62862c805-sys\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.176174 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.175758 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9031958e-b627-4334-9e47-b9b62862c805-root\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.176174 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.175876 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-textfile\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.176174 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.176110 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-accelerators-collector-config\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.176258 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.176189 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9031958e-b627-4334-9e47-b9b62862c805-metrics-client-ca\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.177959 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.177940 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.178072 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.178054 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-tls\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.200933 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.200910 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsbds\" (UniqueName: \"kubernetes.io/projected/9031958e-b627-4334-9e47-b9b62862c805-kube-api-access-fsbds\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.207553 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.207528 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:10.280318 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.280255 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:10.290378 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:21:10.290356 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9031958e_b627_4334_9e47_b9b62862c805.slice/crio-354ce355789adf5f3065a2003cf339bc2d50a64f9306912f0360e6df36fe49c0 WatchSource:0}: Error finding container 354ce355789adf5f3065a2003cf339bc2d50a64f9306912f0360e6df36fe49c0: Status 404 returned error can't find the container with id 354ce355789adf5f3065a2003cf339bc2d50a64f9306912f0360e6df36fe49c0 Oct 30 16:21:10.328641 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.328617 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:10.443417 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.443392 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:10.579806 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:10.579758 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dktqn" event={"ID":"9031958e-b627-4334-9e47-b9b62862c805","Type":"ContainerStarted","Data":"354ce355789adf5f3065a2003cf339bc2d50a64f9306912f0360e6df36fe49c0"} Oct 30 16:21:11.135337 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:11.135309 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:11.290409 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:11.290372 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:11.472894 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:11.472856 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:11.584100 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:11.584065 2566 generic.go:356] "Generic (PLEG): container finished" podID="9031958e-b627-4334-9e47-b9b62862c805" containerID="c43c7df45f897127b74a5113638915b98982611115112ce68d2e27783e1f8785" exitCode=0 Oct 30 16:21:11.584263 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:11.584140 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dktqn" event={"ID":"9031958e-b627-4334-9e47-b9b62862c805","Type":"ContainerDied","Data":"c43c7df45f897127b74a5113638915b98982611115112ce68d2e27783e1f8785"} Oct 30 16:21:11.698526 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:11.698505 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:11.748233 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:11.748167 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:11.873833 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:11.873795 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:12.221319 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.221286 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-59c44d56bd-vwqs7"] Oct 30 16:21:12.224638 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.224618 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.241716 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.241694 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 16:21:12.241848 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.241795 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 16:21:12.241848 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.241801 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 16:21:12.255224 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.255203 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 16:21:12.255325 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.255252 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-v9kbq\"" Oct 30 16:21:12.255488 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.255457 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 16:21:12.255488 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.255456 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dnl1dtfqtp3b6\"" Oct 30 16:21:12.256874 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.256854 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-59c44d56bd-vwqs7"] Oct 30 16:21:12.291050 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.291030 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85bfc58a-5f99-41ad-a651-05ba06bc2be1-metrics-client-ca\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.291346 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.291055 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-grpc-tls\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.291346 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.291095 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.291346 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.291208 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-tls\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.291346 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.291252 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.291346 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.291272 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.291346 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.291314 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zthc\" (UniqueName: \"kubernetes.io/projected/85bfc58a-5f99-41ad-a651-05ba06bc2be1-kube-api-access-6zthc\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.291521 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.291447 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.291553 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.291518 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85bfc58a-5f99-41ad-a651-05ba06bc2be1-tmp\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.392804 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.392741 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.392967 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.392814 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-tls\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.392967 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.392857 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.392967 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.392910 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.392967 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.392937 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zthc\" (UniqueName: \"kubernetes.io/projected/85bfc58a-5f99-41ad-a651-05ba06bc2be1-kube-api-access-6zthc\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.393168 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.393013 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.393168 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.393069 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85bfc58a-5f99-41ad-a651-05ba06bc2be1-tmp\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.393168 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.393126 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85bfc58a-5f99-41ad-a651-05ba06bc2be1-metrics-client-ca\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.393292 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.393150 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-grpc-tls\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.393587 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.393523 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85bfc58a-5f99-41ad-a651-05ba06bc2be1-tmp\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.394410 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.394381 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85bfc58a-5f99-41ad-a651-05ba06bc2be1-metrics-client-ca\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.395720 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.395689 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.395837 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.395742 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.396169 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.396151 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-tls\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.396315 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.396296 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-grpc-tls\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.396373 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.396323 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.396373 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.396331 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.418023 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.417997 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zthc\" (UniqueName: \"kubernetes.io/projected/85bfc58a-5f99-41ad-a651-05ba06bc2be1-kube-api-access-6zthc\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.545672 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.545607 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:12.555864 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.555834 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:12.594572 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.594547 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dktqn" event={"ID":"9031958e-b627-4334-9e47-b9b62862c805","Type":"ContainerStarted","Data":"f4ba878043be51197f0f4358baa68083d26189b82be483a24029dad8226847cb"} Oct 30 16:21:12.594652 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.594628 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dktqn" event={"ID":"9031958e-b627-4334-9e47-b9b62862c805","Type":"ContainerStarted","Data":"0059d193d987d84cd202cefa2983a56588677cda128ec128eddb22905f11c770"} Oct 30 16:21:12.642741 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.642687 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dktqn" podStartSLOduration=2.7926204500000003 podStartE2EDuration="3.642675168s" totalImagesPullingTime="774.417619ms" totalInitContainerRuntime="75.637099ms" isStatefulPod=true podCreationTimestamp="2026-10-30 16:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:12.642173785 +0000 UTC m=+58.924703681" watchObservedRunningTime="2026-10-30 16:21:12.642675168 +0000 UTC m=+58.925205069" Oct 30 16:21:12.855277 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:12.855249 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-59c44d56bd-vwqs7"] Oct 30 16:21:12.855809 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:21:12.855784 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bfc58a_5f99_41ad_a651_05ba06bc2be1.slice/crio-8535ee19f8469395827f9b3b48a6be7b1a95399d68977e205bbc9282932d7e11 WatchSource:0}: Error finding container 8535ee19f8469395827f9b3b48a6be7b1a95399d68977e205bbc9282932d7e11: Status 404 returned error can't find the container with id 8535ee19f8469395827f9b3b48a6be7b1a95399d68977e205bbc9282932d7e11 Oct 30 16:21:13.204430 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:13.204354 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:13.313085 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:13.313038 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:13.513794 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:13.513696 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:13.599510 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:13.599473 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" event={"ID":"85bfc58a-5f99-41ad-a651-05ba06bc2be1","Type":"ContainerStarted","Data":"8535ee19f8469395827f9b3b48a6be7b1a95399d68977e205bbc9282932d7e11"} Oct 30 16:21:14.745728 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:14.745704 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:14.857455 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:14.857429 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:15.049013 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.048930 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:15.288095 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.288064 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6b5445f54d-42vtq"] Oct 30 16:21:15.291329 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.291314 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.308033 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.307982 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6b5445f54d-42vtq"] Oct 30 16:21:15.320530 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.320490 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 16:21:15.322203 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.322188 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 16:21:15.334065 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.334050 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-z7tv8\"" Oct 30 16:21:15.334126 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.334062 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 16:21:15.334227 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.334211 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 16:21:15.348030 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.348013 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 16:21:15.348246 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.348233 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 16:21:15.420283 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.420246 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-federate-client-tls\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.420433 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.420291 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-secret-telemeter-client\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.420433 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.420320 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32474e2a-2206-4b53-a41e-6eaab7de8285-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.420433 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.420396 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-telemeter-client-tls\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.420601 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.420468 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32474e2a-2206-4b53-a41e-6eaab7de8285-serving-certs-ca-bundle\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.420601 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.420571 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32474e2a-2206-4b53-a41e-6eaab7de8285-metrics-client-ca\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.420690 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.420626 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.420690 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.420675 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shzmg\" (UniqueName: \"kubernetes.io/projected/32474e2a-2206-4b53-a41e-6eaab7de8285-kube-api-access-shzmg\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.521607 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.521576 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-secret-telemeter-client\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.521808 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.521616 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32474e2a-2206-4b53-a41e-6eaab7de8285-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.521808 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.521646 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-telemeter-client-tls\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.521808 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.521672 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32474e2a-2206-4b53-a41e-6eaab7de8285-serving-certs-ca-bundle\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.521808 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.521718 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32474e2a-2206-4b53-a41e-6eaab7de8285-metrics-client-ca\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.521808 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.521804 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.522072 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.521855 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shzmg\" (UniqueName: \"kubernetes.io/projected/32474e2a-2206-4b53-a41e-6eaab7de8285-kube-api-access-shzmg\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.522072 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.521899 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-federate-client-tls\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.522577 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.522547 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32474e2a-2206-4b53-a41e-6eaab7de8285-serving-certs-ca-bundle\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.522664 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.522631 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32474e2a-2206-4b53-a41e-6eaab7de8285-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.522728 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.522707 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32474e2a-2206-4b53-a41e-6eaab7de8285-metrics-client-ca\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.525005 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.524983 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-federate-client-tls\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.525100 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.525025 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.525100 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.525030 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-telemeter-client-tls\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.525524 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.525508 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-secret-telemeter-client\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.547867 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.547842 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shzmg\" (UniqueName: \"kubernetes.io/projected/32474e2a-2206-4b53-a41e-6eaab7de8285-kube-api-access-shzmg\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.606157 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.606129 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:15.608339 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.608310 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" event={"ID":"85bfc58a-5f99-41ad-a651-05ba06bc2be1","Type":"ContainerStarted","Data":"fc29d6d6c2c0b72dce18c96115ee81be2d90874166bdef546307ac3f6e227e2c"} Oct 30 16:21:15.608424 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.608380 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" event={"ID":"85bfc58a-5f99-41ad-a651-05ba06bc2be1","Type":"ContainerStarted","Data":"1c3331ce4e074ab63cc06a7d1a926be374e9973c74149c69d23287249f3db1bf"} Oct 30 16:21:15.608424 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.608405 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" event={"ID":"85bfc58a-5f99-41ad-a651-05ba06bc2be1","Type":"ContainerStarted","Data":"e4bcbdcb3693834bed3981629f346647caf6b2c05f2ddfcd39e98de2ff20ce8e"} Oct 30 16:21:15.946654 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:15.946630 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6b5445f54d-42vtq"] Oct 30 16:21:15.947009 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:21:15.946856 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32474e2a_2206_4b53_a41e_6eaab7de8285.slice/crio-61d1b743e19f574488ed9f6112dff6fc4f35560f8d6df778fcb0e22c822b6659 WatchSource:0}: Error finding container 61d1b743e19f574488ed9f6112dff6fc4f35560f8d6df778fcb0e22c822b6659: Status 404 returned error can't find the container with id 61d1b743e19f574488ed9f6112dff6fc4f35560f8d6df778fcb0e22c822b6659 Oct 30 16:21:16.569930 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:16.569893 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:16.610555 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:16.610517 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:16.614988 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:16.614950 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" event={"ID":"85bfc58a-5f99-41ad-a651-05ba06bc2be1","Type":"ContainerStarted","Data":"240b246cc5455683f1a3f03f113344833048ab136f4a60ea89f00ffa7662be2d"} Oct 30 16:21:16.615121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:16.615037 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" event={"ID":"85bfc58a-5f99-41ad-a651-05ba06bc2be1","Type":"ContainerStarted","Data":"e30ff21c0834490aaa0cf6196ddc3cedf910ee1ba4d45d0a40115b9b5089b634"} Oct 30 16:21:16.615121 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:16.615095 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:16.615210 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:16.615135 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" event={"ID":"85bfc58a-5f99-41ad-a651-05ba06bc2be1","Type":"ContainerStarted","Data":"e996f216f08e23c80e8b547fcfab70d01735b0c78895d64c0a91a46e7200c5f6"} Oct 30 16:21:16.616134 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:16.616114 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" event={"ID":"32474e2a-2206-4b53-a41e-6eaab7de8285","Type":"ContainerStarted","Data":"61d1b743e19f574488ed9f6112dff6fc4f35560f8d6df778fcb0e22c822b6659"} Oct 30 16:21:16.681649 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:16.681127 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" podStartSLOduration=2.085587027 podStartE2EDuration="4.681107651s" totalImagesPullingTime="2.595520624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:16.679615006 +0000 UTC m=+62.962144904" watchObservedRunningTime="2026-10-30 16:21:16.681107651 +0000 UTC m=+62.963637549" Oct 30 16:21:16.766709 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:16.766586 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:16.949417 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:16.949387 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 16:21:16.955182 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:16.955149 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:16.976280 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:16.976250 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:16.977461 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:16.977436 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 16:21:16.978564 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:16.978529 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 16:21:16.979392 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:16.979351 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 16:21:16.993497 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:16.993472 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 16:21:16.993753 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:16.993729 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 16:21:17.004209 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.004183 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 16:21:17.007129 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.007106 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 16:21:17.035361 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.035331 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.035513 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.035370 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.035513 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.035397 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.035586 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.035507 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.035586 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.035567 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.035675 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.035597 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.035675 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.035623 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.035675 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.035656 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.035799 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.035687 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.035799 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.035754 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.035876 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.035804 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.035876 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.035836 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.035876 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.035866 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.035986 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.035904 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.035986 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.035932 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.036069 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.036008 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvfl4\" (UniqueName: \"kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-kube-api-access-xvfl4\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.036069 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.036037 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.036136 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.036066 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.036136 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.036095 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-config\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.036136 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.036119 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.049708 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.049674 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 16:21:17.049856 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.049791 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 16:21:17.050030 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.050012 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 16:21:17.050164 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.050136 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 16:21:17.050241 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.050225 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 16:21:17.050704 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.050684 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 16:21:17.050704 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.050694 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 16:21:17.051002 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.050905 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ae6ube48gfedf\"" Oct 30 16:21:17.051002 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.050922 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 16:21:17.051002 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.050907 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mwg5k\"" Oct 30 16:21:17.051002 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.050974 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 16:21:17.136875 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.136847 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.136875 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.136879 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137076 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.136904 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137076 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.136933 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137076 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.136961 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137076 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.136987 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137076 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.137053 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvfl4\" (UniqueName: \"kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-kube-api-access-xvfl4\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137076 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.137076 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137356 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.137100 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137356 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.137125 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-config\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137356 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.137146 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137356 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.137173 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137356 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.137200 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137356 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.137253 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137356 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.137282 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137356 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.137311 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137356 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.137338 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137737 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.137364 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137737 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.137397 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137737 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.137424 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.137737 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.137567 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.138400 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.138370 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.138705 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.138681 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.139330 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.139304 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.140110 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.140086 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.141168 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.141143 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.141409 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.141272 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.141466 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.141451 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.141705 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.141680 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.142660 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.142003 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.142660 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.142120 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.142660 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.142345 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.142660 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.142623 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.142939 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.142696 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.143082 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.143064 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.143404 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.143359 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.144155 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.144133 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.144296 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.144279 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-config\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.144712 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.144693 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.168733 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.168708 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvfl4\" (UniqueName: \"kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-kube-api-access-xvfl4\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.288948 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.288872 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:17.620622 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:17.620594 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 16:21:17.621207 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:21:17.621173 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6698df_8df4_4b71_bba1_d276c753fb2a.slice/crio-286b205a58dc7fae2093663c64c5d8d53e424c94c5a8349577030ea935356632 WatchSource:0}: Error finding container 286b205a58dc7fae2093663c64c5d8d53e424c94c5a8349577030ea935356632: Status 404 returned error can't find the container with id 286b205a58dc7fae2093663c64c5d8d53e424c94c5a8349577030ea935356632 Oct 30 16:21:18.625474 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:18.625429 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" event={"ID":"32474e2a-2206-4b53-a41e-6eaab7de8285","Type":"ContainerStarted","Data":"7814bc1af7828ab3a177d090bc447ff71c929b24b83e60b48fd3ccd1a47299fa"} Oct 30 16:21:18.626879 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:18.626854 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6698df-8df4-4b71-bba1-d276c753fb2a","Type":"ContainerStarted","Data":"286b205a58dc7fae2093663c64c5d8d53e424c94c5a8349577030ea935356632"} Oct 30 16:21:18.675411 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:18.675382 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:18.795572 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:18.795545 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:19.055792 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.055681 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:21:19.055792 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.055716 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs\") pod \"network-metrics-daemon-wtk5g\" (UID: \"35d777f2-4005-45e4-af04-2a61c0567fc0\") " pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:21:19.071013 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.070978 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:21:19.071123 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.071028 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:21:19.078498 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.078474 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs\") pod \"network-metrics-daemon-wtk5g\" (UID: \"35d777f2-4005-45e4-af04-2a61c0567fc0\") " pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:21:19.078624 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.078608 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:21:19.156558 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.156518 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpwt\" (UniqueName: \"kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt\") pod \"network-check-target-xchk6\" (UID: \"73012827-44d9-4cec-9f8d-40b2055c0c69\") " pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:21:19.172218 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.172194 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:21:19.190500 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.190476 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:19.192126 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.192113 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:21:19.200601 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.200583 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mpwt\" (UniqueName: \"kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt\") pod \"network-check-target-xchk6\" (UID: \"73012827-44d9-4cec-9f8d-40b2055c0c69\") " pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:21:19.284191 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.284167 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d8rcz\"" Oct 30 16:21:19.289275 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.289252 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:21:19.297522 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.297505 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-m64dw\"" Oct 30 16:21:19.302713 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.302690 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:21:19.303920 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.303895 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:19.305503 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.305482 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-46f47\"" Oct 30 16:21:19.311123 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.311100 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:21:19.582715 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.582686 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wtk5g"] Oct 30 16:21:19.584284 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:21:19.584259 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d777f2_4005_45e4_af04_2a61c0567fc0.slice/crio-6fa39b352958666fab825a1dba0ea4922861e2c591372fd665e7a058da4b833c WatchSource:0}: Error finding container 6fa39b352958666fab825a1dba0ea4922861e2c591372fd665e7a058da4b833c: Status 404 returned error can't find the container with id 6fa39b352958666fab825a1dba0ea4922861e2c591372fd665e7a058da4b833c Oct 30 16:21:19.612815 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.612797 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k4k2w"] Oct 30 16:21:19.614056 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:21:19.614032 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffdd3593_6091_4874_b55e_4049aa26ab60.slice/crio-0c6b6e84125e285f1df02ee703c894a5a99a2368d7827fa969f881212dc50abf WatchSource:0}: Error finding container 0c6b6e84125e285f1df02ee703c894a5a99a2368d7827fa969f881212dc50abf: Status 404 returned error can't find the container with id 0c6b6e84125e285f1df02ee703c894a5a99a2368d7827fa969f881212dc50abf Oct 30 16:21:19.620164 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.620143 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xchk6"] Oct 30 16:21:19.620815 ip-10-0-129-65 kubenswrapper[2566]: W1030 16:21:19.620752 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73012827_44d9_4cec_9f8d_40b2055c0c69.slice/crio-416d7cedbe52f7134a4fa5d068b118590f0e867b108d5dbaf8efa741c3c6adf4 WatchSource:0}: Error finding container 416d7cedbe52f7134a4fa5d068b118590f0e867b108d5dbaf8efa741c3c6adf4: Status 404 returned error can't find the container with id 416d7cedbe52f7134a4fa5d068b118590f0e867b108d5dbaf8efa741c3c6adf4 Oct 30 16:21:19.631071 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.631049 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k4k2w" event={"ID":"ffdd3593-6091-4874-b55e-4049aa26ab60","Type":"ContainerStarted","Data":"0c6b6e84125e285f1df02ee703c894a5a99a2368d7827fa969f881212dc50abf"} Oct 30 16:21:19.632238 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.632212 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wtk5g" event={"ID":"35d777f2-4005-45e4-af04-2a61c0567fc0","Type":"ContainerStarted","Data":"6fa39b352958666fab825a1dba0ea4922861e2c591372fd665e7a058da4b833c"} Oct 30 16:21:19.633730 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.633709 2566 generic.go:356] "Generic (PLEG): container finished" podID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerID="68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd" exitCode=0 Oct 30 16:21:19.633823 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.633800 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6698df-8df4-4b71-bba1-d276c753fb2a","Type":"ContainerDied","Data":"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd"} Oct 30 16:21:19.636211 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.636144 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" event={"ID":"32474e2a-2206-4b53-a41e-6eaab7de8285","Type":"ContainerStarted","Data":"116992e6a9bc5f2dd59da75584a7a483e8b77120632e1816115cabed8c92ab6a"} Oct 30 16:21:19.636211 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.636199 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" event={"ID":"32474e2a-2206-4b53-a41e-6eaab7de8285","Type":"ContainerStarted","Data":"30a7ca2f67028eb18e26f531216eb86668ec9f229e670d30927d464add2d1215"} Oct 30 16:21:19.637380 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.637362 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xchk6" event={"ID":"73012827-44d9-4cec-9f8d-40b2055c0c69","Type":"ContainerStarted","Data":"416d7cedbe52f7134a4fa5d068b118590f0e867b108d5dbaf8efa741c3c6adf4"} Oct 30 16:21:19.781840 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:19.781796 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" podStartSLOduration=2.032326997 podStartE2EDuration="4.781761694s" totalImagesPullingTime="2.749434697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:19.781030785 +0000 UTC m=+66.063560706" watchObservedRunningTime="2026-10-30 16:21:19.781761694 +0000 UTC m=+66.064291589" Oct 30 16:21:21.654495 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:21.654460 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wtk5g" event={"ID":"35d777f2-4005-45e4-af04-2a61c0567fc0","Type":"ContainerStarted","Data":"9464f3434d64721cec940e7b7273a1ccf313acc0f6b92d8fc8c5746d022d001e"} Oct 30 16:21:21.655020 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:21.654528 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wtk5g" event={"ID":"35d777f2-4005-45e4-af04-2a61c0567fc0","Type":"ContainerStarted","Data":"332aa2929907d83fbb9b3ac96ebd02f6cd23d866c10dac264b931cd358f1f84c"} Oct 30 16:21:22.114590 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:22.114469 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:22.261859 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:22.261821 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:22.379843 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:22.379737 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:22.630743 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:22.630669 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:22.717790 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:22.717695 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wtk5g" podStartSLOduration=66.627125857 podStartE2EDuration="1m7.717673828s" totalImagesPullingTime="1.090547971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:21.699698562 +0000 UTC m=+67.982228460" watchObservedRunningTime="2026-10-30 16:21:22.717673828 +0000 UTC m=+69.000203823" Oct 30 16:21:24.023990 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:24.023933 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:24.137473 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:24.137441 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:24.259509 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:24.259474 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:24.672954 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:24.672916 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xchk6" event={"ID":"73012827-44d9-4cec-9f8d-40b2055c0c69","Type":"ContainerStarted","Data":"eeb3410757ce6d393ddafa47f38e11b2ef38c97a1eadd689d92fddbe62ccfb3c"} Oct 30 16:21:24.673117 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:24.673049 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:21:24.717033 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:24.716974 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xchk6" podStartSLOduration=64.827595696 podStartE2EDuration="1m9.716953873s" totalImagesPullingTime="4.889358177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:24.715822504 +0000 UTC m=+70.998352399" watchObservedRunningTime="2026-10-30 16:21:24.716953873 +0000 UTC m=+70.999483783" Oct 30 16:21:25.184483 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:25.184415 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:25.297459 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:25.297425 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:25.681069 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:25.681030 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6698df-8df4-4b71-bba1-d276c753fb2a","Type":"ContainerStarted","Data":"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300"} Oct 30 16:21:25.681243 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:25.681111 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6698df-8df4-4b71-bba1-d276c753fb2a","Type":"ContainerStarted","Data":"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc"} Oct 30 16:21:25.681243 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:25.681148 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6698df-8df4-4b71-bba1-d276c753fb2a","Type":"ContainerStarted","Data":"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2"} Oct 30 16:21:25.681243 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:25.681191 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6698df-8df4-4b71-bba1-d276c753fb2a","Type":"ContainerStarted","Data":"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0"} Oct 30 16:21:25.681243 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:25.681235 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6698df-8df4-4b71-bba1-d276c753fb2a","Type":"ContainerStarted","Data":"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91"} Oct 30 16:21:25.681377 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:25.681272 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6698df-8df4-4b71-bba1-d276c753fb2a","Type":"ContainerStarted","Data":"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254"} Oct 30 16:21:25.783325 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:25.783291 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:25.893847 ip-10-0-129-65 kubenswrapper[2566]: I1030 16:21:25.893811 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:26.230237 ip-10-0-129-65 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 16:21:26.238318 ip-10-0-129-65 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 16:21:26.238497 ip-10-0-129-65 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 16:21:26.238547 ip-10-0-129-65 systemd[1]: kubelet.service: Consumed 4.115s CPU time. Oct 30 16:21:26.250599 ip-10-0-129-65 systemd[1]: Starting Kubernetes Kubelet... Oct 30 16:21:26.321426 ip-10-0-129-65 kubenswrapper[6202]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:21:26.321426 ip-10-0-129-65 kubenswrapper[6202]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 16:21:26.321426 ip-10-0-129-65 kubenswrapper[6202]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:21:26.321426 ip-10-0-129-65 kubenswrapper[6202]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:21:26.323801 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323689 6202 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:21:26.323801 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323797 6202 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:21:26.323801 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323803 6202 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:21:26.323801 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323808 6202 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323812 6202 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323816 6202 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323820 6202 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323824 6202 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323828 6202 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323832 6202 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323836 6202 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323841 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323844 6202 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323850 6202 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323854 6202 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323858 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323862 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323866 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323870 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323874 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323877 6202 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323881 6202 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:21:26.324069 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323885 6202 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323892 6202 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323899 6202 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323912 6202 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323916 6202 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323920 6202 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323924 6202 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323928 6202 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323932 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323936 6202 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323940 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323944 6202 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323949 6202 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323953 6202 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323957 6202 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323961 6202 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323965 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323970 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323974 6202 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:21:26.324884 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323978 6202 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323982 6202 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323986 6202 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323990 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.323994 6202 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324000 6202 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324005 6202 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324009 6202 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324014 6202 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324018 6202 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324022 6202 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324026 6202 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324030 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324035 6202 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324040 6202 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324044 6202 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324049 6202 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324054 6202 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324058 6202 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324062 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:21:26.325670 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324066 6202 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324071 6202 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324075 6202 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324079 6202 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324083 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324088 6202 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324092 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324097 6202 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324101 6202 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324105 6202 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324110 6202 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324114 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324118 6202 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324122 6202 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324126 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324131 6202 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324136 6202 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324140 6202 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324145 6202 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324150 6202 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:21:26.326558 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324154 6202 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324159 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324165 6202 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324170 6202 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324174 6202 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324179 6202 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324183 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324188 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324192 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324196 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324201 6202 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324205 6202 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324210 6202 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324214 6202 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324218 6202 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324223 6202 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324227 6202 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324231 6202 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324235 6202 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:21:26.327425 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324239 6202 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324243 6202 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324248 6202 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324252 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324256 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.324264 6202 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325518 6202 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325530 6202 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325536 6202 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325541 6202 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325546 6202 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325551 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325556 6202 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325561 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325565 6202 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325569 6202 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325574 6202 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325579 6202 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325584 6202 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325588 6202 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:21:26.328336 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325593 6202 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325597 6202 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325601 6202 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325606 6202 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325610 6202 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325614 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325618 6202 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325622 6202 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325626 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325631 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325635 6202 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325639 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325643 6202 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325647 6202 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325651 6202 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325655 6202 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325659 6202 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325663 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325667 6202 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:21:26.329227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325671 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325676 6202 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325682 6202 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325687 6202 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325691 6202 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325697 6202 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325705 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325710 6202 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325714 6202 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325720 6202 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325724 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325729 6202 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325734 6202 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325738 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325742 6202 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325746 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325750 6202 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325755 6202 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325758 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:21:26.330063 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325785 6202 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325789 6202 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325795 6202 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325800 6202 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325804 6202 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325808 6202 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325812 6202 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325816 6202 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325820 6202 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325824 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325829 6202 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325833 6202 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325837 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325841 6202 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325846 6202 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325850 6202 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325855 6202 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325859 6202 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325863 6202 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:21:26.330582 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325867 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325873 6202 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325878 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325882 6202 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325886 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325890 6202 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325895 6202 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325900 6202 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325904 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325908 6202 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325912 6202 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325916 6202 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325920 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325924 6202 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325929 6202 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325933 6202 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325937 6202 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325941 6202 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325945 6202 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:21:26.331072 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325949 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325953 6202 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325957 6202 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325962 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325966 6202 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325970 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325974 6202 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325978 6202 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325982 6202 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325988 6202 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325992 6202 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.325996 6202 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326000 6202 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326004 6202 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326008 6202 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326013 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326890 6202 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326901 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326906 6202 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326911 6202 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:21:26.331563 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326915 6202 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326919 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326924 6202 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326928 6202 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326933 6202 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326940 6202 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326946 6202 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326951 6202 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326955 6202 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326959 6202 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326964 6202 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326968 6202 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326973 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326977 6202 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326981 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326986 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326990 6202 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326995 6202 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.326999 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:21:26.332123 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327003 6202 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327007 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327011 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327015 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327019 6202 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327024 6202 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327028 6202 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327032 6202 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327044 6202 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327049 6202 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327053 6202 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327057 6202 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327062 6202 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327066 6202 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327070 6202 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327074 6202 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327078 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327082 6202 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327087 6202 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327091 6202 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:21:26.332568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327095 6202 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327099 6202 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327103 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327108 6202 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327112 6202 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327116 6202 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327121 6202 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327125 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327129 6202 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327133 6202 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327138 6202 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327141 6202 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327146 6202 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327150 6202 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327154 6202 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327159 6202 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327163 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327167 6202 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327171 6202 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327175 6202 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:21:26.333051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327178 6202 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327184 6202 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327193 6202 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327202 6202 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327207 6202 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327213 6202 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327218 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327223 6202 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327227 6202 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327231 6202 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327235 6202 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327239 6202 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327243 6202 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327248 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327252 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327256 6202 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327260 6202 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327264 6202 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327268 6202 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:21:26.333584 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327273 6202 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327277 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327281 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327285 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327289 6202 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327294 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327298 6202 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327302 6202 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327306 6202 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327311 6202 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327315 6202 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327320 6202 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327324 6202 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327328 6202 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327334 6202 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327338 6202 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327345 6202 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327348 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327353 6202 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:21:26.334327 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327357 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327361 6202 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327365 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327369 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.327373 6202 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327509 6202 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327520 6202 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327530 6202 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327537 6202 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327544 6202 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327549 6202 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327556 6202 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327562 6202 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327567 6202 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327573 6202 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327578 6202 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327584 6202 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327589 6202 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327593 6202 flags.go:64] FLAG: --cgroup-root="" Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327598 6202 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327603 6202 flags.go:64] FLAG: --client-ca-file="" Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327608 6202 flags.go:64] FLAG: --cloud-provider="external" Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327612 6202 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 16:21:26.334863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327618 6202 flags.go:64] FLAG: --cluster-domain="" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327622 6202 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327632 6202 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327637 6202 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327642 6202 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327649 6202 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327655 6202 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327660 6202 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327665 6202 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327670 6202 flags.go:64] FLAG: --contention-profiling="false" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327675 6202 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327679 6202 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327685 6202 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327689 6202 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327696 6202 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327701 6202 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327706 6202 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327710 6202 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327714 6202 flags.go:64] FLAG: --enable-server="true" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327719 6202 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327727 6202 flags.go:64] FLAG: --event-burst="100" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327732 6202 flags.go:64] FLAG: --event-qps="50" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327736 6202 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327741 6202 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 16:21:26.335387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327745 6202 flags.go:64] FLAG: --eviction-hard="" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327752 6202 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327756 6202 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327780 6202 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327785 6202 flags.go:64] FLAG: --eviction-soft="" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327790 6202 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327795 6202 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327800 6202 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327805 6202 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327809 6202 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327818 6202 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327823 6202 flags.go:64] FLAG: --feature-gates="" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327829 6202 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327834 6202 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327840 6202 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327847 6202 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327852 6202 flags.go:64] FLAG: --healthz-port="10248" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327857 6202 flags.go:64] FLAG: --help="false" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327861 6202 flags.go:64] FLAG: --hostname-override="ip-10-0-129-65.ec2.internal" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327866 6202 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327871 6202 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327876 6202 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327881 6202 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 16:21:26.336007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327887 6202 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327892 6202 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327896 6202 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327900 6202 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327912 6202 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327916 6202 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327922 6202 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327926 6202 flags.go:64] FLAG: --kube-reserved="" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327931 6202 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327935 6202 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327940 6202 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327944 6202 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327949 6202 flags.go:64] FLAG: --lock-file="" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327953 6202 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327958 6202 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327964 6202 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327974 6202 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327979 6202 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327983 6202 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327990 6202 flags.go:64] FLAG: --logging-format="text" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.327994 6202 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328000 6202 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328005 6202 flags.go:64] FLAG: --manifest-url="" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328010 6202 flags.go:64] FLAG: --manifest-url-header="" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328017 6202 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 16:21:26.336545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328023 6202 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328029 6202 flags.go:64] FLAG: --max-pods="110" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328034 6202 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328039 6202 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328044 6202 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328048 6202 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328053 6202 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328058 6202 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328063 6202 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328074 6202 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328079 6202 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328083 6202 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328089 6202 flags.go:64] FLAG: --pod-cidr="" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328093 6202 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328097 6202 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328103 6202 flags.go:64] FLAG: --pods-per-core="0" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328108 6202 flags.go:64] FLAG: --port="10250" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328113 6202 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328117 6202 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05b5b6dea8bd51d43" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328123 6202 flags.go:64] FLAG: --qos-reserved="" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328127 6202 flags.go:64] FLAG: --read-only-port="10255" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328132 6202 flags.go:64] FLAG: --register-node="true" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328137 6202 flags.go:64] FLAG: --register-with-taints="" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328144 6202 flags.go:64] FLAG: --registry-burst="10" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328149 6202 flags.go:64] FLAG: --registry-qps="5" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328154 6202 flags.go:64] FLAG: --reserved-cpus="" Oct 30 16:21:26.337177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328160 6202 flags.go:64] FLAG: --reserved-memory="" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328167 6202 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328172 6202 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328177 6202 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328181 6202 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328186 6202 flags.go:64] FLAG: --runonce="false" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328190 6202 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328196 6202 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328201 6202 flags.go:64] FLAG: --seccomp-default="false" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328206 6202 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328210 6202 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328215 6202 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328220 6202 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328225 6202 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328229 6202 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328234 6202 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328238 6202 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328243 6202 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328248 6202 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328253 6202 flags.go:64] FLAG: --system-cgroups="" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328258 6202 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328268 6202 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328273 6202 flags.go:64] FLAG: --tls-cert-file="" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328278 6202 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328284 6202 flags.go:64] FLAG: --tls-min-version="" Oct 30 16:21:26.337803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328289 6202 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328293 6202 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328297 6202 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328302 6202 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328307 6202 flags.go:64] FLAG: --v="2" Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328313 6202 flags.go:64] FLAG: --version="false" Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328319 6202 flags.go:64] FLAG: --vmodule="" Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328327 6202 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.328333 6202 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328541 6202 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328549 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328555 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328561 6202 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328568 6202 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328573 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328578 6202 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328583 6202 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328587 6202 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328592 6202 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328596 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328600 6202 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:21:26.338382 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328604 6202 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328608 6202 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328613 6202 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328617 6202 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328621 6202 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328625 6202 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328629 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328635 6202 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328642 6202 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328647 6202 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328652 6202 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328656 6202 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328661 6202 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328665 6202 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328669 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328673 6202 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328677 6202 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328682 6202 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328688 6202 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:21:26.338888 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328692 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328696 6202 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328701 6202 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328705 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328709 6202 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328713 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328718 6202 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328722 6202 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328726 6202 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328731 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328735 6202 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328739 6202 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328743 6202 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328747 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328751 6202 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328756 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328779 6202 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328784 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328787 6202 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328792 6202 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:21:26.339344 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328797 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328802 6202 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328806 6202 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328810 6202 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328815 6202 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328819 6202 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328823 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328828 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328832 6202 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328836 6202 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328840 6202 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328846 6202 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328850 6202 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328854 6202 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328859 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328863 6202 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328868 6202 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328872 6202 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328876 6202 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328880 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:21:26.339853 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328884 6202 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328890 6202 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328893 6202 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328897 6202 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328902 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328906 6202 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328910 6202 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328914 6202 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328918 6202 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328922 6202 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328926 6202 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328930 6202 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328936 6202 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328940 6202 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328945 6202 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328949 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328953 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328957 6202 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328961 6202 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328965 6202 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:21:26.340429 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328969 6202 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:21:26.341021 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328973 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:21:26.341021 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328978 6202 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:21:26.341021 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328984 6202 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:21:26.341021 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328988 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:21:26.341021 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328993 6202 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:21:26.341021 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.328997 6202 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:21:26.341021 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329001 6202 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:21:26.341021 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329006 6202 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:21:26.341021 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329010 6202 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:21:26.341021 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329014 6202 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:21:26.341021 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329019 6202 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:21:26.341021 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329023 6202 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:21:26.341021 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329027 6202 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:21:26.341021 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329033 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:21:26.341021 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.329076 6202 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:21:26.341021 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329132 6202 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329139 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329144 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329148 6202 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329152 6202 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329156 6202 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329161 6202 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329165 6202 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329170 6202 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329174 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329179 6202 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329183 6202 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329187 6202 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329191 6202 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329195 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329199 6202 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329204 6202 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329208 6202 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329212 6202 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:21:26.341406 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329216 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329222 6202 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329226 6202 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329230 6202 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329234 6202 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329238 6202 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329242 6202 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329247 6202 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329251 6202 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329255 6202 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329260 6202 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329264 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329269 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329273 6202 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329277 6202 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329282 6202 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329286 6202 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329290 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329294 6202 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329298 6202 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:21:26.341898 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329302 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329308 6202 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329313 6202 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329317 6202 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329322 6202 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329326 6202 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329330 6202 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329335 6202 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329339 6202 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329343 6202 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329348 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329352 6202 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329356 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329362 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329370 6202 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329375 6202 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329380 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329385 6202 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329389 6202 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329394 6202 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:21:26.342502 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329398 6202 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329402 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329406 6202 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329410 6202 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329415 6202 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329420 6202 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329424 6202 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329428 6202 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329432 6202 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329436 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329440 6202 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329447 6202 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329452 6202 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329458 6202 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329462 6202 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329466 6202 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329471 6202 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329475 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329480 6202 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:21:26.343152 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329484 6202 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329488 6202 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329492 6202 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329496 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329500 6202 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329504 6202 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329510 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329515 6202 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329519 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329523 6202 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329527 6202 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329531 6202 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329536 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329540 6202 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329544 6202 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329548 6202 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329553 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329557 6202 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329561 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:21:26.343598 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329567 6202 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329571 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329575 6202 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329579 6202 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329583 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329587 6202 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329591 6202 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329598 6202 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.329604 6202 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.334809 6202 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.334830 6202 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.334849 6202 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.334854 6202 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334918 6202 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334923 6202 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334926 6202 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334929 6202 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334932 6202 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334935 6202 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334938 6202 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:21:26.344079 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334941 6202 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334943 6202 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334946 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334948 6202 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334951 6202 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334954 6202 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334956 6202 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334959 6202 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334962 6202 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334964 6202 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334967 6202 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334969 6202 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334972 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334974 6202 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334977 6202 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334980 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334983 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334985 6202 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334988 6202 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334990 6202 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:21:26.344557 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334993 6202 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334996 6202 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.334999 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335002 6202 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335004 6202 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335007 6202 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335009 6202 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335012 6202 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335014 6202 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335016 6202 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335019 6202 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335022 6202 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335024 6202 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335026 6202 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335029 6202 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335032 6202 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335034 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335037 6202 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335040 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335043 6202 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:21:26.345169 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335046 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335048 6202 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335051 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335054 6202 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335056 6202 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335059 6202 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335061 6202 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335064 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335067 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335070 6202 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335073 6202 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335075 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335078 6202 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335080 6202 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335083 6202 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335085 6202 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335088 6202 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335090 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335093 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:21:26.345632 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335096 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:21:26.346100 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335099 6202 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:21:26.346100 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335101 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:21:26.346100 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335103 6202 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:21:26.346100 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335106 6202 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:21:26.346100 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335108 6202 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:21:26.346100 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335111 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:21:26.346100 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335113 6202 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:21:26.346100 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335115 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:21:26.346100 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335118 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:21:26.346100 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335121 6202 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:21:26.346100 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335123 6202 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:21:26.346100 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335125 6202 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:21:26.346100 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335129 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:21:26.346100 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335133 6202 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:21:26.346100 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335137 6202 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:21:26.346100 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335140 6202 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:21:26.346100 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335142 6202 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:21:26.346100 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335145 6202 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335147 6202 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335151 6202 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335154 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335156 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335159 6202 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335161 6202 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335164 6202 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335167 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335169 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335172 6202 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335174 6202 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335177 6202 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335180 6202 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335182 6202 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335185 6202 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335187 6202 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335190 6202 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335193 6202 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335196 6202 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:21:26.346519 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335198 6202 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:21:26.347008 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335202 6202 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:21:26.347008 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.335232 6202 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:21:26.347008 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335393 6202 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:21:26.347008 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335399 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:21:26.347008 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335402 6202 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:21:26.347008 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335406 6202 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:21:26.347008 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335408 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:21:26.347008 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335411 6202 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:21:26.347008 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335414 6202 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:21:26.347008 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335417 6202 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:21:26.347008 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335419 6202 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:21:26.347008 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335422 6202 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:21:26.347008 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335425 6202 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:21:26.347008 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335427 6202 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:21:26.347008 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335431 6202 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:21:26.347008 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335433 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335436 6202 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335438 6202 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335441 6202 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335443 6202 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335446 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335448 6202 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335451 6202 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335453 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335456 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335458 6202 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335461 6202 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335463 6202 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335466 6202 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335468 6202 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335471 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335473 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335476 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335478 6202 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:21:26.347386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335481 6202 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335483 6202 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335485 6202 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335488 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335490 6202 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335492 6202 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335495 6202 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335498 6202 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335500 6202 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335503 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335506 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335508 6202 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335510 6202 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335513 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335515 6202 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335518 6202 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335520 6202 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335522 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335525 6202 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335527 6202 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:21:26.347838 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335529 6202 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335532 6202 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335534 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335537 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335539 6202 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335542 6202 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335544 6202 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335547 6202 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335549 6202 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335551 6202 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335554 6202 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335556 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335558 6202 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335561 6202 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335563 6202 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335566 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335568 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335571 6202 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335573 6202 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:21:26.348342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335577 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335580 6202 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335582 6202 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335584 6202 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335587 6202 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335590 6202 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335592 6202 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335595 6202 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335597 6202 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335600 6202 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335602 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335605 6202 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335607 6202 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335609 6202 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335612 6202 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335615 6202 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335617 6202 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335620 6202 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335622 6202 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:21:26.348810 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335625 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:21:26.349253 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335629 6202 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:21:26.349253 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335632 6202 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:21:26.349253 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335635 6202 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:21:26.349253 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335638 6202 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:21:26.349253 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335641 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:21:26.349253 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335643 6202 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:21:26.349253 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335647 6202 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:21:26.349253 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335650 6202 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:21:26.349253 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335653 6202 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:21:26.349253 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335656 6202 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:21:26.349253 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335659 6202 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:21:26.349253 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335662 6202 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:21:26.349253 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335664 6202 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:21:26.349253 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335667 6202 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:21:26.349253 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335669 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:21:26.349867 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.335692 6202 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:21:26.349867 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335723 6202 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:21:26.349867 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335727 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:21:26.349867 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335730 6202 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:21:26.349867 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335733 6202 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:21:26.349867 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335735 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:21:26.349867 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335738 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:21:26.349867 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335740 6202 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:21:26.349867 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335743 6202 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:21:26.349867 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335745 6202 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:21:26.349867 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335748 6202 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:21:26.349867 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335750 6202 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:21:26.349867 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335753 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:21:26.349867 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335755 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:21:26.349867 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335757 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:21:26.349867 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335760 6202 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335783 6202 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335786 6202 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335789 6202 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335791 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335794 6202 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335796 6202 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335799 6202 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335801 6202 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335804 6202 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335806 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335808 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335811 6202 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335813 6202 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335815 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335818 6202 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335821 6202 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335823 6202 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335826 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335829 6202 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:21:26.350496 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335831 6202 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335834 6202 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335836 6202 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335839 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335842 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335844 6202 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335847 6202 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335849 6202 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335852 6202 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335854 6202 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335856 6202 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335859 6202 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335861 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335863 6202 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335866 6202 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335869 6202 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335871 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335874 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335877 6202 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:21:26.351040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335879 6202 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335882 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335884 6202 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335886 6202 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335889 6202 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335891 6202 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335893 6202 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335895 6202 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335898 6202 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335900 6202 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335911 6202 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335914 6202 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335917 6202 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335920 6202 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335922 6202 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335925 6202 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335927 6202 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335930 6202 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335932 6202 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335934 6202 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:21:26.351485 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335937 6202 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335941 6202 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335945 6202 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335947 6202 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335951 6202 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335953 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335956 6202 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335960 6202 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335963 6202 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335965 6202 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335968 6202 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335970 6202 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335973 6202 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335975 6202 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335978 6202 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335980 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335982 6202 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335985 6202 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335987 6202 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:21:26.352000 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335990 6202 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:21:26.352470 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335992 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:21:26.352470 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335994 6202 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:21:26.352470 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.335997 6202 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:21:26.352470 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.336000 6202 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:21:26.352470 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.336005 6202 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:21:26.352470 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.336008 6202 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:21:26.352470 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.336011 6202 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:21:26.352470 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.336014 6202 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:21:26.352470 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.336017 6202 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:21:26.352470 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.336020 6202 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:21:26.352470 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.336022 6202 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:21:26.352470 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.336025 6202 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:21:26.352470 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:26.336027 6202 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:21:26.352470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.336087 6202 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 16:21:26.352470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.337057 6202 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 16:21:26.352470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.337111 6202 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 16:21:26.352470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.337291 6202 server.go:1014] "Starting client certificate rotation" Oct 30 16:21:26.352470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.337433 6202 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:21:26.352919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.337475 6202 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:15:14 +0000 UTC" deadline="2028-04-05 11:21:50.939844808 +0000 UTC" Oct 30 16:21:26.352919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.337509 6202 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12547h0m24.602337973s" Oct 30 16:21:26.352919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.337790 6202 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:21:26.352919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.338891 6202 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:21:26.352919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.340245 6202 log.go:25] "Validated CRI v1 runtime API" Oct 30 16:21:26.352919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.342012 6202 log.go:25] "Validated CRI v1 image API" Oct 30 16:21:26.352919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.342683 6202 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 16:21:26.352919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.344681 6202 fs.go:135] Filesystem UUIDs: map[4ce666f9-2d08-4baf-9d89-b699101e4f8e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d486a53b-5a12-4a19-b01e-8659668cb2ef:/dev/nvme0n1p3] Oct 30 16:21:26.353143 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.344702 6202 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/092b4c87f40344e29f6960c03f2d7fdbc986bc2222db5324fde6d7c135ac2683/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/092b4c87f40344e29f6960c03f2d7fdbc986bc2222db5324fde6d7c135ac2683/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c6b6e84125e285f1df02ee703c894a5a99a2368d7827fa969f881212dc50abf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c6b6e84125e285f1df02ee703c894a5a99a2368d7827fa969f881212dc50abf/userdata/shm major:0 minor:375 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f31f459ebbe6e8f2965c2fd4d8ff1b385c2cfd2342cc2417bf8e1ca7cb21da7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f31f459ebbe6e8f2965c2fd4d8ff1b385c2cfd2342cc2417bf8e1ca7cb21da7/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25905daee0d773c71b579005e239a140a15a514e6aa7a146fa1ba5eabbc9cb79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25905daee0d773c71b579005e239a140a15a514e6aa7a146fa1ba5eabbc9cb79/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/286b205a58dc7fae2093663c64c5d8d53e424c94c5a8349577030ea935356632/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/286b205a58dc7fae2093663c64c5d8d53e424c94c5a8349577030ea935356632/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d0aa1ec165a544356719160f8a7c8b019d8db79dcc1309892994e71a1bdcede/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d0aa1ec165a544356719160f8a7c8b019d8db79dcc1309892994e71a1bdcede/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ef5e9a5a1ba2ce970fe79131d48f63f2821bfef48f192fc8ec0fb5cc54415e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ef5e9a5a1ba2ce970fe79131d48f63f2821bfef48f192fc8ec0fb5cc54415e3/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/354ce355789adf5f3065a2003cf339bc2d50a64f9306912f0360e6df36fe49c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/354ce355789adf5f3065a2003cf339bc2d50a64f9306912f0360e6df36fe49c0/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/416d7cedbe52f7134a4fa5d068b118590f0e867b108d5dbaf8efa741c3c6adf4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/416d7cedbe52f7134a4fa5d068b118590f0e867b108d5dbaf8efa741c3c6adf4/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61d1b743e19f574488ed9f6112dff6fc4f35560f8d6df778fcb0e22c822b6659/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61d1b743e19f574488ed9f6112dff6fc4f35560f8d6df778fcb0e22c822b6659/userdata/shm major:0 minor:317 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fa39b352958666fab825a1dba0ea4922861e2c591372fd665e7a058da4b833c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fa39b352958666fab825a1dba0ea4922861e2c591372fd665e7a058da4b833c/userdata/shm major:0 minor:371 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ae24ea35dd910421df6922ed257e7e1ebbae16c456a86eade15e12bf9bb4cf1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ae24ea35dd910421df6922ed257e7e1ebbae16c456a86eade15e12bf9bb4cf1/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7af113a1c1ae3ca9d88081baddb267079c3c8eee8a2e653bf0bc7c654c50da47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7af113a1c1ae3ca9d88081baddb267079c3c8eee8a2e653bf0bc7c654c50da47/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80f44efa402a758d3adac4a4a32036485bd2b2370cac6cfb091fc80ea6a343d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80f44efa402a758d3adac4a4a32036485bd2b2370cac6cfb091fc80ea6a343d8/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8535ee19f8469395827f9b3b48a6be7b1a95399d68977e205bbc9282932d7e11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8535ee19f8469395827f9b3b48a6be7b1a95399d68977e205bbc9282932d7e11/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a74a378aac3d83f9a93fc7212e59dc36ec5df65b30c05569341922dd6f0b669/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a74a378aac3d83f9a93fc7212e59dc36ec5df65b30c05569341922dd6f0b669/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d3d29cc7e093b1be18697db482c2e1f86cca240144f5b21745dc05d7271928a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d3d29cc7e093b1be18697db482c2e1f86cca240144f5b21745dc05d7271928a/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc3bd5082efb84b9694faee499d28f189117d32c69e400ca7377b315d0b1058b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc3bd5082efb84b9694faee499d28f189117d32c69e400ca7377b315d0b1058b/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dcf9b8eaa9ca92fd91eb3158868a0ce4d581a0af752e56faf0b65bbc1d0152fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dcf9b8eaa9ca92fd91eb3158868a0ce4d581a0af752e56faf0b65bbc1d0152fe/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e181cbd1daf8abf4db197ac2ae288fa1833186a1c601945425f04c61bac34840/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e181cbd1daf8abf4db197ac2ae288fa1833186a1c601945425f04c61bac34840/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eaa7ac44907717dfd5e76801c725ac94f8101ae4512b52404e04893350cbf434/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eaa7ac44907717dfd5e76801c725ac94f8101ae4512b52404e04893350cbf434/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dcde09b-ef11-4bb9-810b-883b0ee4cc18/volumes/kubernetes.io~projected/kube-api-access-9c8t5:{mountpoint:/var/lib/kubelet/pods/0dcde09b-ef11-4bb9-810b-883b0ee4cc18/volumes/kubernetes.io~projected/kube-api-access-9c8t5 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dcde09b-ef11-4bb9-810b-883b0ee4cc18/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0dcde09b-ef11-4bb9-810b-883b0ee4cc18/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dcde09b-ef11-4bb9-810b-883b0ee4cc18/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/0dcde09b-ef11-4bb9-810b-883b0ee4cc18/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17726533-46eb-4d51-bc1d-b4e9959d7ba5/volumes/kubernetes.io~projected/kube-api-access-znx2b:{mountpoint:/var/lib/kubelet/pods/17726533-46eb-4d51-bc1d-b4e9959d7ba5/volumes/kubernetes.io~projected/kube-api-access-znx2b major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25906571-b2ad-4134-9674-e67842de154d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/25906571-b2ad-4134-9674-e67842de154d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25906571-b2ad-4134-9674-e67842de154d/volumes/kubernetes.io~projected/kube-api-access-tmxkq:{mountpoint:/var/lib/kubelet/pods/25906571-b2ad-4134-9674-e67842de154d/volumes/kubernetes.io~projected/kube-api-access-tmxkq major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25906571-b2ad-4134-9674-e67842de154d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/25906571-b2ad-4134-9674-e67842de154d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32474e2a-2206-4b53-a41e-6eaab7de8285/volumes/kubernetes.io~projected/kube-api-access-shzmg:{mountpoint:/var/lib/kubelet/pods/32474e2a-2206-4b53-a41e-6eaab7de8285/volumes/kubernetes.io~projected/kube-api-access-shzmg major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32474e2a-2206-4b53-a41e-6eaab7de8285/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/32474e2a-2206-4b53-a41e-6eaab7de8285/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32474e2a-2206-4b53-a41e-6eaab7de8285/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/32474e2a-2206-4b53-a41e-6eaab7de8285/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32474e2a-2206-4b53-a41e-6eaab7de8285/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/32474e2a-2206-4b53-a41e-6eaab7de8285/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32474e2a-2206-4b53-a41e-6eaab7de8285/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/32474e2a-2206-4b53-a41e-6eaab7de8285/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32528057-07fa-4eb9-882f-a43c9cd36b31/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/32528057-07fa-4eb9-882f-a43c9cd36b31/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35d777f2-4005-45e4-af04-2a61c0567fc0/volumes/kubernetes.io~projected/kube-api-access-wslqp:{mountpoint:/var/lib/kubelet/pods/35d777f2-4005-45e4-af04-2a61c0567fc0/volumes/kubernetes.io~projected/kube-api-access-wslqp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35d777f2-4005-45e4-af04-2a61c0567fc0/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/35d777f2-4005-45e4-af04-2a61c0567fc0/volumes/kubernetes.io~secret/metrics-certs major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42aa85ec-676a-4f13-90f4-0fb13e94b22f/volumes/kubernetes.io~projected/kube-api-access-dsrqq:{mountpoint:/var/lib/kubelet/pods/42aa85ec-676a-4f13-90f4-0fb13e94b22f/volumes/kubernetes.io~projected/kube-api-access-dsrqq major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4512b7bb-6729-4045-aac9-4a1ece176016/volumes/kubernetes.io~projected/kube-api-access-ghbq5:{mountpoint:/var/lib/kubelet/pods/4512b7bb-6729-4045-aac9-4a1ece176016/volumes/kubernetes.io~projected/kube-api-access-ghbq5 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73012827-44d9-4cec-9f8d-40b2055c0c69/volumes/kubernetes.io~projected/kube-api-access-5mpwt:{mountpoint:/var/lib/kubelet/pods/73012827-44d9-4cec-9f8d-40b2055c0c69/volumes/kubernetes.io~projected/kube-api-access-5mpwt major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~projected/kube-api-access-6zthc:{mountpoint:/var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~projected/kube-api-access-6zthc major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86634960-264a-48a7-969a-5fbe6e58447e/volumes/kubernetes.io~projected/kube-api-access-5f28s:{mountpoint:/var/lib/kubelet/pods/86634960-264a-48a7-969a-5fbe6e58447e/volumes/kubernetes.io~projected/kube-api-access-5f28s major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86634960-264a-48a7-969a-5fbe6e58447e/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/86634960-264a-48a7-969a-5fbe6e58447e/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volume-subpaths/web-config/config-reloader/2 major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volume-subpaths/web-config/prometheus/15 major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~empty-dir/config-out major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~projected/kube-api-access-xvfl4:{mountpoint:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~projected/kube-api-access-xvfl4 major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~projected/tls-assets major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/config major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/web-config major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9031958e-b627-4334-9e47-b9b62862c805/volumes/kubernetes.io~projected/kube-api-access-fsbds:{mountpoint:/var/lib/kubelet/pods/9031958e-b627-4334-9e47-b9b62862c805/volumes/kubernetes.io~projected/kube-api-access-fsbds major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9031958e-b627-4334-9e47-b9b62862c805/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9031958e-b627-4334-9e47-b9b62862c805/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9031958e-b627-4334-9e47-b9b62862c805/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/9031958e-b627-4334-9e47-b9b62862c805/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b17ec352-725f-4bea-aaee-a3ca414fde6e/volumes/kubernetes.io~projected/kube-api-access-765lm:{mountpoint:/var/lib/kubelet/pods/b17ec352-725f-4bea-aaee-a3ca414fde6e/volumes/kubernetes.io~projected/kube-api-access-765lm major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1bbf041-5696-4a2c-94a8-69ab0e995096/volumes/kubernetes.io~projected/kube-api-access-c2p9c:{mountpoint:/var/lib/kubelet/pods/c1bbf041-5696-4a2c-94a8-69ab0e995096/volumes/kubernetes.io~projected/kube-api-access-c2p9c major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc/volumes/kubernetes.io~projected/kube-api-access-5k2rm:{mountpoint:/var/lib/kubelet/pods/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc/volumes/kubernetes.io~projected/kube-api-access-5k2rm major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd7dac20-991b-4048-acd5-08dcbdb57892/volumes/kubernetes.io~projected/kube-api-access-vd4m6:{mountpoint:/var/lib/kubelet/pods/cd7dac20-991b-4048-acd5-08dcbdb57892/volumes/kubernetes.io~projected/kube-api-access-vd4m6 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd7dac20-991b-4048-acd5-08dcbdb57892/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/cd7dac20-991b-4048-acd5-08dcbdb57892/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7b597f3-5dc9-45e6-840f-b492ba06c241/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e7b597f3-5dc9-45e6-840f-b492ba06c241/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7b597f3-5dc9-45e6-840f-b492ba06c241/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e7b597f3-5dc9-45e6-840f-b492ba06c241/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7b597f3-5dc9-45e6-840f-b492ba06c241/volumes/kubernetes.io~projected/kube-api-access-z6r8b:{mountpoint:/var/lib/kubelet/pods/e7b597f3-5dc9-45e6-840f-b492ba06c241/volumes/kubernetes.io~projected/kube-api-access-z6r8b major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5/volumes/kubernetes.io~projected/kube-api-access-5pjpq:{mountpoint:/var/lib/kubelet/pods/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5/volumes/kubernetes.io~projected/kube-api-access-5pjpq major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffdd3593-6091-4874-b55e-4049aa26ab60/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ffdd3593-6091-4874-b55e-4049aa26ab60/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:369 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a2324da90cfa369fb2f8b5b81c916d8843ed35d1f67e411188eb6f584eab6bf7/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/332a7f8844e1d77177a955ff9f21d3e6350acee26a1498fbab4881bc6c255a69/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b7e24c8619fc9f2691a8996b040d6f86be73957c69f73296886f7d260d0bf568/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/885755f8448872bd4ff4ab60482b333aaed23fc0ed34abeb7a41ec9c87a3cc06/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-112:{mountpoint:/var/lib/containers/storage/overlay/1befb70f84662195473069157c8a44f61e241cd5e677401fee699f5e77d96dac/merged major:0 minor:112 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/799a01e916bb49b25168ed62d957dee80f2d3d57992070d2d29480f50a1652f5/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f3c65c70fede5f3dc95985a7dd39520914be811274664bbfa01dcc0c621bdb5d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/02dea7f9dddc0339e6fc75d83a2a97dd6c1f03833e59a71cd83cdd7314fedc4e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/1b25fd49e44bf406f3a46304f068205ac1d058686caea9dc4a2a5cf24a63e495/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/63dbf50e8c00fb862bbf552de225b9b5a22bdbba8235f3c9e2695008eaaf238e/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/138c644889260eba08c25d2666ece115d585e5e13eed8154d862b83bd6bd2e36/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/a3c137f214485f2507e1b12036127d5c771accca84453d810e912a56c6c384cb/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/046f45d2e4ceaec0ab6f76f7ccf6a0edaaeb1598610ac58d952220ffa6a39a84/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/948d48b42b12abf6035f5f0f9b3bb5ce2ef91460ea6fb6244328629b44c00ffd/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/54749a9918f9c75cd9f5e28ac37295af2d051e6841047b5e1603c57ca2f8e09a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/2e1472ed56ab35fbd0e5fae1f4754024af2ae2fd1a40d8b659b8d73048e41f20/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a395693bc332fc429bd11016a9c50ea5e904a4284497814c47c517e5fbd7518f/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/09fddd182ec1a9878ca66c57fbe57dcb8450b16602d435bed0e0e176da8f8193/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d28efd2b9c872a03f60e064cb732b54879b2290c368da3b5bc2a17ede31f922a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/3404f0bc6eb27ecbea5dc663226e9ec32f9185af035915562bab659c392ce3d7/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/1bacde228681f5b4ccb00dd2510733fb36043379a43a1250a83591a206479f6e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/67658d707238455782feccd589550d52e0e9f482c6e11b29c28ca0d73599ac2f/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/c259d5739c1d368125727bacdb77d497b3562aef2d63385d2c17ec49befc3603/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/1515ba1fa955ca4a2696aa6d583295badc7168d93a5299a335308036aacf4b66/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/ef23ae8d7d8fb74804aaaa9aacacb8c984a643386c923df47d2da0df8b69f7f5/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/62e6989426f5a09c0e330cdf2358c3c2faac405ab67613198da32985b402837b/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/daa8dda9d27af970a8493ee87fd799aaa207b615ea968a2f61e5686960128ac9/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/28e6c2edec14e164f2da45fd40a95c395fddd9fcf69598d788bd6df9b519e7f8/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/698181c9da5290bc3a17c5352449273875d89ef2e00bbea5fe32b2483328900f/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/76b2c1494f4ae9fab835bd7fce56bf4f00dbc86ac9b1d04ce4af7aafa8617e15/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/5c7518318087a8a52cf34c94c16cd41352ff572cba6191effeb6ea39fb5548d8/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/60dc01938f42148c8d53aec10af6cf31e7ad3459609519105185ab1159bda3e3/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/310ff726bfcb002094ce4c09c8d9932561f2ab7b70e97c99745cdf1ac0d0f889/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/d1105939eb52ad768913d5884b3f4a1cab95981f3846de8526db33255f62fca8/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/e0ec8ea3a0ee0db67fcbe06877ea336003b594605ac8bdd6fe0b9a7b9e889eef/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/8b02ce7bd9f5bbccf7ce1eabe1c46b867a7f917c3a367c9d12a71019453f0b57/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/ef16bc529df05579b77566c088871633564506232de442ab4fee085ec168b046/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/de7432d31919a0905c2db274588fb8436d547d7e2883fb888134a46bb9fc7063/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/5fdbea0495f0c55ea431342a2bea79b0fb1347d6a8d6d28d556a72d1c5f8b4b2/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/e860a482015b83a3dea8f73b755221491fe6cd3a3c7bca8a127d26eee33d3e65/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/1e31cf0635df6763d20de85a453ac7ea99f97f3896869231883844ac32e858ee/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/376d0debf7771e5d9504c370658208fa61fedd0a9a911b8d2b5c44648dbe0cc8/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/b6b33474afa30dd5df4d669af3a4f1d1fccaab27223c06660690e97938ef7f01/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/8702375ff8b3d67b4f20f5fa7ced4ed678e49fc80186ae4be8fd9b56aeb6bbc9/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/6450645f58e1243415baa5b06ae202b52d473974e2e709d28164f84890e33e12/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/78adf9784565937cd4608590ca09e6e1a051440e4e17dc44d473527a6b032327/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/da6e53eccb23ab58a341331830ea03014ea30fa52ed4119d62322864dd39a2e2/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/7ca9b3f2eb2a1dd57a42bd7fd92a48ef524f0a68e70fcd70497ce959fab5db4b/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/361ded9d6404403c9d7177283119776354d677cd16ce9fb2abc05c704ed256d0/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/0ad83a546f795d0fc999e892492d5c64d7cdd2c9a78293ea5011bf4401c1c13a/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/20d54a209e8e69391a95427c03e754bcf80828169755ab0329874046cdd22346/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/e1a4fbb1855aea47ec3172a18b26583fecd5023fe27b7d520f1c9ca0f4fee3e8/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/b00e8abe5122f8900bbd825d58bacd8f521c717b4c844f9f477576cf4b58a802/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/1572c334a2e71ed5ee18727071d1d888cb4312a4af0259fa69fa311a01506724/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/556749a25708fe69b4ab9bdf1de479af20e205c42e8898a05c60ca7f66b941c9/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/c6c015f612a789e7a61f4de32177ff49d7ac496062127576b6542ba2adcf0726/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/64b374eff2a91659e68ecbf4faa9e9976996e54cba36514384c810e83361ebfe/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/5c1af00c77fce300256ebd39e149cbed29047b79aa8fcf519d4d5e6664983ec8/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/5d16e46c9b7d406f9bc3c418c7f5d5d0e5a0d9dbfac835929617046e3d22109a/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/bc9d4cf4b2754c23947b6451210baf2b952be52596faa1652064ce981bbe4a1b/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/1d7ab526f15b487988ce8dab6ced434701bae3866efb2fe84bf07d32bb8a55fc/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/90da9f6b4ebbc5ff64fe0d69697ba15fe5810b0e7189dbc66232f6ace4336d82/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/0aab40ba20086b4dcea3339ff27906cb57ff44fe509d60055b98fd4c6e3bd230/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/95abeeb957d6efb29eaf21d7913aebc4ec7e651af71b0c55a86857493571f9d8/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/d9c5dddb734d3ad80955be179b4042ca9979c69edee10060008aaa1f4d5b8fc4/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/f055543a30c82e82855d99240f25495f1019da6471c409fe1ed92c6e96beaf7d/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/16ff0eee8c1af8d563b02fc9e8060215cd7f7c16c5b9223afaa998f0e8b16fb8/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/4e3e5195a00b7fd81af82c56129f90ba604dbb54ee282f79a5df58d6b1bb2438/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/5f6b174cf2f5a50c34e2f3ac1d8d2b7d99f99833c83e2bbbf91931890faa5b4b/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/6dbebaccea75e5e17ecf3857d6b23fdd7d5b026eeeee6659f028e7740e7cf18c/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/12133564b45445783c9d7fbb3fea3e0e26bcaa488ee21bb1047abcd62a636ce8/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 16:21:26.353646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.353271 6202 manager.go:217] Machine: {Timestamp:2026-10-30 16:21:26.352813544 +0000 UTC m=+0.063268278 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109217 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec274d11ebf734d657f9c790077b8c7c SystemUUID:ec274d11-ebf7-34d6-57f9-c790077b8c7c BootID:76abe132-bc16-42d7-85d9-36ba7b516edd Filesystems:[{Device:/var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8535ee19f8469395827f9b3b48a6be7b1a95399d68977e205bbc9282932d7e11/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:341 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e7b597f3-5dc9-45e6-840f-b492ba06c241/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7b597f3-5dc9-45e6-840f-b492ba06c241/volumes/kubernetes.io~projected/kube-api-access-z6r8b DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4512b7bb-6729-4045-aac9-4a1ece176016/volumes/kubernetes.io~projected/kube-api-access-ghbq5 DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25905daee0d773c71b579005e239a140a15a514e6aa7a146fa1ba5eabbc9cb79/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9031958e-b627-4334-9e47-b9b62862c805/volumes/kubernetes.io~projected/kube-api-access-fsbds DeviceMajor:0 DeviceMinor:270 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:290 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/25906571-b2ad-4134-9674-e67842de154d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:345 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/35d777f2-4005-45e4-af04-2a61c0567fc0/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:365 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/354ce355789adf5f3065a2003cf339bc2d50a64f9306912f0360e6df36fe49c0/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:338 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fa39b352958666fab825a1dba0ea4922861e2c591372fd665e7a058da4b833c/userdata/shm DeviceMajor:0 DeviceMinor:371 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/17726533-46eb-4d51-bc1d-b4e9959d7ba5/volumes/kubernetes.io~projected/kube-api-access-znx2b DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d0aa1ec165a544356719160f8a7c8b019d8db79dcc1309892994e71a1bdcede/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f31f459ebbe6e8f2965c2fd4d8ff1b385c2cfd2342cc2417bf8e1ca7cb21da7/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25906571-b2ad-4134-9674-e67842de154d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25906571-b2ad-4134-9674-e67842de154d/volumes/kubernetes.io~projected/kube-api-access-tmxkq DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9031958e-b627-4334-9e47-b9b62862c805/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:269 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7b597f3-5dc9-45e6-840f-b492ba06c241/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7af113a1c1ae3ca9d88081baddb267079c3c8eee8a2e653bf0bc7c654c50da47/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9031958e-b627-4334-9e47-b9b62862c805/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:265 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/416d7cedbe52f7134a4fa5d068b118590f0e867b108d5dbaf8efa741c3c6adf4/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d3d29cc7e093b1be18697db482c2e1f86cca240144f5b21745dc05d7271928a/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0dcde09b-ef11-4bb9-810b-883b0ee4cc18/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/32474e2a-2206-4b53-a41e-6eaab7de8285/volumes/kubernetes.io~projected/kube-api-access-shzmg DeviceMajor:0 DeviceMinor:316 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:344 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~projected/kube-api-access-xvfl4 DeviceMajor:0 DeviceMinor:347 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ae24ea35dd910421df6922ed257e7e1ebbae16c456a86eade15e12bf9bb4cf1/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c1bbf041-5696-4a2c-94a8-69ab0e995096/volumes/kubernetes.io~projected/kube-api-access-c2p9c DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e181cbd1daf8abf4db197ac2ae288fa1833186a1c601945425f04c61bac34840/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80f44efa402a758d3adac4a4a32036485bd2b2370cac6cfb091fc80ea6a343d8/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:337 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:340 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc3bd5082efb84b9694faee499d28f189117d32c69e400ca7377b315d0b1058b/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61d1b743e19f574488ed9f6112dff6fc4f35560f8d6df778fcb0e22c822b6659/userdata/shm DeviceMajor:0 DeviceMinor:317 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35d777f2-4005-45e4-af04-2a61c0567fc0/volumes/kubernetes.io~projected/kube-api-access-wslqp DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0dcde09b-ef11-4bb9-810b-883b0ee4cc18/volumes/kubernetes.io~projected/kube-api-access-9c8t5 DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~projected/kube-api-access-6zthc DeviceMajor:0 DeviceMinor:291 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/32474e2a-2206-4b53-a41e-6eaab7de8285/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:314 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-112 DeviceMajor:0 DeviceMinor:112 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:336 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5/volumes/kubernetes.io~projected/kube-api-access-5pjpq DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:289 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/32474e2a-2206-4b53-a41e-6eaab7de8285/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:309 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:343 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dcf9b8eaa9ca92fd91eb3158868a0ce4d581a0af752e56faf0b65bbc1d0152fe/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/092b4c87f40344e29f6960c03f2d7fdbc986bc2222db5324fde6d7c135ac2683/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b17ec352-725f-4bea-aaee-a3ca414fde6e/volumes/kubernetes.io~projected/kube-api-access-765lm DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/42aa85ec-676a-4f13-90f4-0fb13e94b22f/volumes/kubernetes.io~projected/kube-api-access-dsrqq DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eaa7ac44907717dfd5e76801c725ac94f8101ae4512b52404e04893350cbf434/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cd7dac20-991b-4048-acd5-08dcbdb57892/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:342 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/32528057-07fa-4eb9-882f-a43c9cd36b31/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc/volumes/kubernetes.io~projected/kube-api-access-5k2rm DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd7dac20-991b-4048-acd5-08dcbdb57892/volumes/kubernetes.io~projected/kube-api-access-vd4m6 DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0dcde09b-ef11-4bb9-810b-883b0ee4cc18/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/32474e2a-2206-4b53-a41e-6eaab7de8285/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:313 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/73012827-44d9-4cec-9f8d-40b2055c0c69/volumes/kubernetes.io~projected/kube-api-access-5mpwt DeviceMajor:0 DeviceMinor:370 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c6b6e84125e285f1df02ee703c894a5a99a2368d7827fa969f881212dc50abf/userdata/shm DeviceMajor:0 DeviceMinor:375 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ef5e9a5a1ba2ce970fe79131d48f63f2821bfef48f192fc8ec0fb5cc54415e3/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:288 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/32474e2a-2206-4b53-a41e-6eaab7de8285/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:315 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86634960-264a-48a7-969a-5fbe6e58447e/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/85bfc58a-5f99-41ad-a651-05ba06bc2be1/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:346 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86634960-264a-48a7-969a-5fbe6e58447e/volumes/kubernetes.io~projected/kube-api-access-5f28s DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/286b205a58dc7fae2093663c64c5d8d53e424c94c5a8349577030ea935356632/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a74a378aac3d83f9a93fc7212e59dc36ec5df65b30c05569341922dd6f0b669/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffdd3593-6091-4874-b55e-4049aa26ab60/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:369 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:092b4c87f40344e MacAddress:8e:e0:3b:14:f9:76 Speed:10000 Mtu:8901} {Name:0c6b6e84125e285 MacAddress:06:73:34:9a:49:f8 Speed:10000 Mtu:8901} {Name:286b205a58dc7fa MacAddress:26:22:86:13:1f:22 Speed:10000 Mtu:8901} {Name:416d7cedbe52f71 MacAddress:76:6e:d2:2d:85:81 Speed:10000 Mtu:8901} {Name:61d1b743e19f574 MacAddress:02:f5:62:7b:74:26 Speed:10000 Mtu:8901} {Name:6fa39b352958666 MacAddress:6a:4b:c1:10:0e:54 Speed:10000 Mtu:8901} {Name:80f44efa402a758 MacAddress:d2:26:8a:a5:e9:23 Speed:10000 Mtu:8901} {Name:8535ee19f846939 MacAddress:ae:ad:15:0e:46:a9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d1:56:5b:39 Speed:0 Mtu:9001} {Name:br-int MacAddress:c6:52:33:d7:51:02 Speed:0 Mtu:8901} {Name:dcf9b8eaa9ca92f MacAddress:06:f3:fe:a0:3b:05 Speed:10000 Mtu:8901} {Name:e181cbd1daf8abf MacAddress:5a:70:44:a9:91:f7 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d1:56:5b:39 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:aa:7a:75:01:ec:ba Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:8a:65:af:29:43:87 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 16:21:26.353989 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.353650 6202 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 16:21:26.353989 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.353695 6202 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 16:21:26.354054 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.353996 6202 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 16:21:26.354149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.354022 6202 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-65.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 16:21:26.354193 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.354151 6202 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 16:21:26.354193 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.354158 6202 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 16:21:26.354193 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.354168 6202 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:21:26.354193 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.354187 6202 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:21:26.354291 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.354262 6202 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 16:21:26.354429 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.354420 6202 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 16:21:26.354540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.354532 6202 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 16:21:26.354576 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.354569 6202 kubelet.go:493] "Attempting to sync node with API server" Oct 30 16:21:26.354604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.354583 6202 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 16:21:26.354604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.354593 6202 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 16:21:26.354604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.354602 6202 kubelet.go:405] "Adding apiserver pod source" Oct 30 16:21:26.354688 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.354610 6202 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 16:21:26.355466 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.355450 6202 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 16:21:26.355523 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.355468 6202 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:21:26.355979 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.355962 6202 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 16:21:26.358620 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.358604 6202 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 16:21:26.358620 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.358618 6202 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:21:26.358730 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.358720 6202 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 16:21:26.358946 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.358934 6202 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 16:21:26.358977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.358964 6202 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 16:21:26.359068 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.359059 6202 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 16:21:26.359097 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.359074 6202 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 16:21:26.359097 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.359080 6202 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 16:21:26.359097 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.359086 6202 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 16:21:26.359097 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.359091 6202 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 16:21:26.359097 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.359096 6202 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 16:21:26.359255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.359102 6202 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 16:21:26.359255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.359107 6202 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 16:21:26.359255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.359114 6202 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 16:21:26.359255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.359120 6202 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 16:21:26.359255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.359137 6202 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 16:21:26.359255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.359160 6202 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 16:21:26.359255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.359201 6202 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 16:21:26.359255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.359207 6202 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 16:21:26.362621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.362608 6202 server.go:1263] "Started kubelet" Oct 30 16:21:26.362685 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.362656 6202 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 16:21:26.362940 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.362712 6202 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 16:21:26.363453 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.363035 6202 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 16:21:26.363453 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.363377 6202 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 16:21:26.363298 ip-10-0-129-65 systemd[1]: Started Kubernetes Kubelet. Oct 30 16:21:26.367169 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.367152 6202 server.go:317] "Adding debug handlers to kubelet server" Oct 30 16:21:26.368792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.367316 6202 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 16:21:26.368792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.367349 6202 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 16:21:26.368792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.367466 6202 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 16:21:26.368792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.367483 6202 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 16:21:26.368792 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:21:26.367474 6202 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-65.ec2.internal\" not found" Oct 30 16:21:26.368792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.367725 6202 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:15:15 +0000 UTC" deadline="2028-07-24 02:42:46.389591626 +0000 UTC" Oct 30 16:21:26.368792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.367760 6202 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15178h21m20.021836498s" Oct 30 16:21:26.368792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.367871 6202 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373074 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25906571-b2ad-4134-9674-e67842de154d" volumeName="kubernetes.io/secret/25906571-b2ad-4134-9674-e67842de154d-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373109 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bfc58a-5f99-41ad-a651-05ba06bc2be1" volumeName="kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373124 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-kube-api-access-xvfl4" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373141 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-grpc-tls" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373153 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17726533-46eb-4d51-bc1d-b4e9959d7ba5" volumeName="kubernetes.io/projected/17726533-46eb-4d51-bc1d-b4e9959d7ba5-kube-api-access-znx2b" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373164 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bfc58a-5f99-41ad-a651-05ba06bc2be1" volumeName="kubernetes.io/projected/85bfc58a-5f99-41ad-a651-05ba06bc2be1-kube-api-access-6zthc" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373179 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bfc58a-5f99-41ad-a651-05ba06bc2be1" volumeName="kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373190 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86634960-264a-48a7-969a-5fbe6e58447e" volumeName="kubernetes.io/configmap/86634960-264a-48a7-969a-5fbe6e58447e-config-volume" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373203 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373220 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b17ec352-725f-4bea-aaee-a3ca414fde6e" volumeName="kubernetes.io/configmap/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-daemon-config" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373231 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dcde09b-ef11-4bb9-810b-883b0ee4cc18" volumeName="kubernetes.io/projected/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-kube-api-access-9c8t5" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373244 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4512b7bb-6729-4045-aac9-4a1ece176016" volumeName="kubernetes.io/configmap/4512b7bb-6729-4045-aac9-4a1ece176016-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373258 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bfc58a-5f99-41ad-a651-05ba06bc2be1" volumeName="kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-grpc-tls" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373270 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-config" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373288 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9031958e-b627-4334-9e47-b9b62862c805" volumeName="kubernetes.io/configmap/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373302 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9031958e-b627-4334-9e47-b9b62862c805" volumeName="kubernetes.io/secret/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-tls" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373314 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc" volumeName="kubernetes.io/projected/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc-kube-api-access-5k2rm" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373325 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17726533-46eb-4d51-bc1d-b4e9959d7ba5" volumeName="kubernetes.io/configmap/17726533-46eb-4d51-bc1d-b4e9959d7ba5-serviceca" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373339 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32528057-07fa-4eb9-882f-a43c9cd36b31" volumeName="kubernetes.io/secret/32528057-07fa-4eb9-882f-a43c9cd36b31-agent-certs" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373351 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bfc58a-5f99-41ad-a651-05ba06bc2be1" volumeName="kubernetes.io/empty-dir/85bfc58a-5f99-41ad-a651-05ba06bc2be1-tmp" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373362 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373377 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-tls-assets" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373388 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9031958e-b627-4334-9e47-b9b62862c805" volumeName="kubernetes.io/projected/9031958e-b627-4334-9e47-b9b62862c805-kube-api-access-fsbds" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373399 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b17ec352-725f-4bea-aaee-a3ca414fde6e" volumeName="kubernetes.io/projected/b17ec352-725f-4bea-aaee-a3ca414fde6e-kube-api-access-765lm" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373413 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd7dac20-991b-4048-acd5-08dcbdb57892" volumeName="kubernetes.io/secret/cd7dac20-991b-4048-acd5-08dcbdb57892-cert" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373424 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25906571-b2ad-4134-9674-e67842de154d" volumeName="kubernetes.io/configmap/25906571-b2ad-4134-9674-e67842de154d-ovnkube-config" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373436 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32474e2a-2206-4b53-a41e-6eaab7de8285" volumeName="kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-federate-client-tls" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373457 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32528057-07fa-4eb9-882f-a43c9cd36b31" volumeName="kubernetes.io/configmap/32528057-07fa-4eb9-882f-a43c9cd36b31-konnectivity-ca" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373473 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4512b7bb-6729-4045-aac9-4a1ece176016" volumeName="kubernetes.io/configmap/4512b7bb-6729-4045-aac9-4a1ece176016-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373490 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86634960-264a-48a7-969a-5fbe6e58447e" volumeName="kubernetes.io/secret/86634960-264a-48a7-969a-5fbe6e58447e-metrics-tls" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373507 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-config-out" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373518 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5" volumeName="kubernetes.io/secret/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373530 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25906571-b2ad-4134-9674-e67842de154d" volumeName="kubernetes.io/projected/25906571-b2ad-4134-9674-e67842de154d-kube-api-access-tmxkq" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373545 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32474e2a-2206-4b53-a41e-6eaab7de8285" volumeName="kubernetes.io/configmap/32474e2a-2206-4b53-a41e-6eaab7de8285-metrics-client-ca" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373556 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bfc58a-5f99-41ad-a651-05ba06bc2be1" volumeName="kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373567 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373581 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7b597f3-5dc9-45e6-840f-b492ba06c241" volumeName="kubernetes.io/empty-dir/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-tuned" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373592 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5" volumeName="kubernetes.io/configmap/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373608 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32474e2a-2206-4b53-a41e-6eaab7de8285" volumeName="kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-secret-telemeter-client" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373622 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373634 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373645 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7b597f3-5dc9-45e6-840f-b492ba06c241" volumeName="kubernetes.io/empty-dir/e7b597f3-5dc9-45e6-840f-b492ba06c241-tmp" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373660 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32474e2a-2206-4b53-a41e-6eaab7de8285" volumeName="kubernetes.io/configmap/32474e2a-2206-4b53-a41e-6eaab7de8285-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373671 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373685 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373700 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-web-config" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373714 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc" volumeName="kubernetes.io/empty-dir/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc-tmp-dir" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373725 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffdd3593-6091-4874-b55e-4049aa26ab60" volumeName="kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373739 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32474e2a-2206-4b53-a41e-6eaab7de8285" volumeName="kubernetes.io/configmap/32474e2a-2206-4b53-a41e-6eaab7de8285-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373752 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42aa85ec-676a-4f13-90f4-0fb13e94b22f" volumeName="kubernetes.io/projected/42aa85ec-676a-4f13-90f4-0fb13e94b22f-kube-api-access-dsrqq" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373783 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73012827-44d9-4cec-9f8d-40b2055c0c69" volumeName="kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373800 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373811 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9031958e-b627-4334-9e47-b9b62862c805" volumeName="kubernetes.io/secret/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373823 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd7dac20-991b-4048-acd5-08dcbdb57892" volumeName="kubernetes.io/projected/cd7dac20-991b-4048-acd5-08dcbdb57892-kube-api-access-vd4m6" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373846 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5" volumeName="kubernetes.io/projected/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-kube-api-access-5pjpq" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373858 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32474e2a-2206-4b53-a41e-6eaab7de8285" volumeName="kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373877 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35d777f2-4005-45e4-af04-2a61c0567fc0" volumeName="kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373892 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4512b7bb-6729-4045-aac9-4a1ece176016" volumeName="kubernetes.io/projected/4512b7bb-6729-4045-aac9-4a1ece176016-kube-api-access-ghbq5" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373903 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bfc58a-5f99-41ad-a651-05ba06bc2be1" volumeName="kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373916 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86634960-264a-48a7-969a-5fbe6e58447e" volumeName="kubernetes.io/projected/86634960-264a-48a7-969a-5fbe6e58447e-kube-api-access-5f28s" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373932 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373944 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9031958e-b627-4334-9e47-b9b62862c805" volumeName="kubernetes.io/empty-dir/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-textfile" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373960 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7b597f3-5dc9-45e6-840f-b492ba06c241" volumeName="kubernetes.io/projected/e7b597f3-5dc9-45e6-840f-b492ba06c241-kube-api-access-z6r8b" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373976 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dcde09b-ef11-4bb9-810b-883b0ee4cc18" volumeName="kubernetes.io/secret/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-prometheus-operator-tls" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.373992 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dcde09b-ef11-4bb9-810b-883b0ee4cc18" volumeName="kubernetes.io/secret/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374005 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25906571-b2ad-4134-9674-e67842de154d" volumeName="kubernetes.io/configmap/25906571-b2ad-4134-9674-e67842de154d-env-overrides" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374019 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374030 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dcde09b-ef11-4bb9-810b-883b0ee4cc18" volumeName="kubernetes.io/configmap/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-metrics-client-ca" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374041 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4512b7bb-6729-4045-aac9-4a1ece176016" volumeName="kubernetes.io/configmap/4512b7bb-6729-4045-aac9-4a1ece176016-cni-binary-copy" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374056 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-db" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374070 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9031958e-b627-4334-9e47-b9b62862c805" volumeName="kubernetes.io/configmap/9031958e-b627-4334-9e47-b9b62862c805-metrics-client-ca" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374104 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32474e2a-2206-4b53-a41e-6eaab7de8285" volumeName="kubernetes.io/projected/32474e2a-2206-4b53-a41e-6eaab7de8285-kube-api-access-shzmg" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374123 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32474e2a-2206-4b53-a41e-6eaab7de8285" volumeName="kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-telemeter-client-tls" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374138 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86634960-264a-48a7-969a-5fbe6e58447e" volumeName="kubernetes.io/empty-dir/86634960-264a-48a7-969a-5fbe6e58447e-tmp-dir" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374151 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1bbf041-5696-4a2c-94a8-69ab0e995096" volumeName="kubernetes.io/projected/c1bbf041-5696-4a2c-94a8-69ab0e995096-kube-api-access-c2p9c" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374167 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374183 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374202 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6698df-8df4-4b71-bba1-d276c753fb2a" volumeName="kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374214 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25906571-b2ad-4134-9674-e67842de154d" volumeName="kubernetes.io/configmap/25906571-b2ad-4134-9674-e67842de154d-ovnkube-script-lib" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374226 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35d777f2-4005-45e4-af04-2a61c0567fc0" volumeName="kubernetes.io/projected/35d777f2-4005-45e4-af04-2a61c0567fc0-kube-api-access-wslqp" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374238 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bfc58a-5f99-41ad-a651-05ba06bc2be1" volumeName="kubernetes.io/configmap/85bfc58a-5f99-41ad-a651-05ba06bc2be1-metrics-client-ca" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374251 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b17ec352-725f-4bea-aaee-a3ca414fde6e" volumeName="kubernetes.io/configmap/b17ec352-725f-4bea-aaee-a3ca414fde6e-cni-binary-copy" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374263 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1bbf041-5696-4a2c-94a8-69ab0e995096" volumeName="kubernetes.io/configmap/c1bbf041-5696-4a2c-94a8-69ab0e995096-iptables-alerter-script" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374275 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5" volumeName="kubernetes.io/empty-dir/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-data-volume" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374287 6202 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bfc58a-5f99-41ad-a651-05ba06bc2be1" volumeName="kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374299 6202 reconstruct.go:97] "Volume reconstruction finished" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374307 6202 reconciler.go:29] "Reconciler: start to sync state" Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374893 6202 factory.go:55] Registering systemd factory Oct 30 16:21:26.375007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.374910 6202 factory.go:223] Registration of the systemd container factory successfully Oct 30 16:21:26.381100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.375137 6202 factory.go:153] Registering CRI-O factory Oct 30 16:21:26.381100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.375153 6202 factory.go:223] Registration of the crio container factory successfully Oct 30 16:21:26.381100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.375204 6202 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 16:21:26.381100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.375222 6202 factory.go:103] Registering Raw factory Oct 30 16:21:26.381100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.375236 6202 manager.go:1199] Started watching for new ooms in manager Oct 30 16:21:26.381100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.375613 6202 manager.go:319] Starting recovery of all containers Oct 30 16:21:26.400384 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.400358 6202 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 16:21:26.402120 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.402097 6202 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 16:21:26.402120 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.402119 6202 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 16:21:26.402291 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.402137 6202 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 16:21:26.402291 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:21:26.402188 6202 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 16:21:26.413526 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.413485 6202 generic.go:356] "Generic (PLEG): container finished" podID="4512b7bb-6729-4045-aac9-4a1ece176016" containerID="958ab40ebef67e47dce9b9671eb69a276b14b2b4b8429d80ab012ed43e75e815" exitCode=0 Oct 30 16:21:26.413526 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.413523 6202 generic.go:356] "Generic (PLEG): container finished" podID="4512b7bb-6729-4045-aac9-4a1ece176016" containerID="e07268ef7214a1ba6541b92b70500c8c440dc1bbbd58fb1e51f1bf32e8de0693" exitCode=0 Oct 30 16:21:26.413695 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.413533 6202 generic.go:356] "Generic (PLEG): container finished" podID="4512b7bb-6729-4045-aac9-4a1ece176016" containerID="c86b03b18b60610ee9b894a0cf19f0c743462ace4b95c4544bcb27a617a5e282" exitCode=0 Oct 30 16:21:26.413695 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.413539 6202 generic.go:356] "Generic (PLEG): container finished" podID="4512b7bb-6729-4045-aac9-4a1ece176016" containerID="aefdfe161a3994246f1b18c4d79933e24c86207aaddb0d078cf06396c40a149b" exitCode=0 Oct 30 16:21:26.413695 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.413543 6202 generic.go:356] "Generic (PLEG): container finished" podID="4512b7bb-6729-4045-aac9-4a1ece176016" containerID="e5ed0c0a4778077480bf47e0fa2d282b68cc9d9e616ec1738574b035160bf988" exitCode=0 Oct 30 16:21:26.413695 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.413548 6202 generic.go:356] "Generic (PLEG): container finished" podID="4512b7bb-6729-4045-aac9-4a1ece176016" containerID="6888dc52e4bc627bcc56a608c369543d2dc3b8a6926020a8932ee87b47b8ea46" exitCode=0 Oct 30 16:21:26.426571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.426377 6202 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:21:26.426571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.426450 6202 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 16:21:26.427518 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.427496 6202 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:21:26.429279 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.429256 6202 generic.go:356] "Generic (PLEG): container finished" podID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerID="68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd" exitCode=0 Oct 30 16:21:26.444435 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.444410 6202 generic.go:356] "Generic (PLEG): container finished" podID="d7ecc0c6cd8524d6054106d923cd48af" containerID="a157ffac181ba6670e958e074630632719958af6aa11121031653652ddcaa994" exitCode=0 Oct 30 16:21:26.449923 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.449897 6202 generic.go:356] "Generic (PLEG): container finished" podID="9031958e-b627-4334-9e47-b9b62862c805" containerID="c43c7df45f897127b74a5113638915b98982611115112ce68d2e27783e1f8785" exitCode=0 Oct 30 16:21:26.453250 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.453227 6202 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:21:26.453412 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.453388 6202 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:21:26.478935 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.478916 6202 manager.go:324] Recovery completed Oct 30 16:21:26.501742 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.501685 6202 cpu_manager.go:226] "Starting" policy="none" Oct 30 16:21:26.501742 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.501699 6202 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 16:21:26.501742 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.501715 6202 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 16:21:26.501914 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.501837 6202 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 16:21:26.501914 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.501848 6202 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 16:21:26.501914 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.501863 6202 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 16:21:26.501914 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.501868 6202 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 16:21:26.501914 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.501873 6202 policy_none.go:50] "Start" Oct 30 16:21:26.501914 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.501881 6202 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 16:21:26.501914 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.501890 6202 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 16:21:26.502704 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:21:26.502682 6202 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 16:21:26.503157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.503146 6202 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 16:21:26.503195 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.503157 6202 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 16:21:26.503195 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.503163 6202 policy_none.go:44] "Start" Oct 30 16:21:26.507553 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.507529 6202 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 16:21:26.510932 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.510920 6202 manager.go:341] "Starting Device Plugin manager" Oct 30 16:21:26.511611 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:21:26.510941 6202 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 16:21:26.511667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.511616 6202 server.go:87] "Starting device plugin registration server" Oct 30 16:21:26.511954 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.511942 6202 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 16:21:26.511988 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.511958 6202 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 16:21:26.512077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.512053 6202 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 16:21:26.512199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.512158 6202 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 16:21:26.512296 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.512203 6202 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 16:21:26.512296 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.512211 6202 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 16:21:26.512392 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.512259 6202 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:21:26.512192336Z","UUID":"e711e4c3-cbe4-4124-b1d8-6ddf8b626d2f","Handler":null,"Name":"","Endpoint":""} Oct 30 16:21:26.513694 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.513677 6202 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 16:21:26.513798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.513702 6202 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 16:21:26.513798 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:21:26.513707 6202 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 16:21:26.612898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.612859 6202 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:21:26.614743 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.614723 6202 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:21:26.614848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.614752 6202 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:21:26.614848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.614784 6202 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:21:26.614848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.614830 6202 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-65.ec2.internal" Oct 30 16:21:26.651595 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.651570 6202 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-65.ec2.internal" Oct 30 16:21:26.651677 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.651631 6202 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-65.ec2.internal" Oct 30 16:21:26.703005 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.702964 6202 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal"] Oct 30 16:21:26.703632 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.703614 6202 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:21:26.703704 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.703644 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" event={"ID":"829ec3bbd4f4a9d896bac875cd5fa87a","Type":"ContainerStarted","Data":"a6103ccba7696cfa758e65f3829b41c4a1a814617135af748c6b8f2222494f7b"} Oct 30 16:21:26.703737 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.703711 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" event={"ID":"829ec3bbd4f4a9d896bac875cd5fa87a","Type":"ContainerStarted","Data":"7ae24ea35dd910421df6922ed257e7e1ebbae16c456a86eade15e12bf9bb4cf1"} Oct 30 16:21:26.703816 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.703736 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" event={"ID":"d7ecc0c6cd8524d6054106d923cd48af","Type":"ContainerStarted","Data":"c9afdce4e6fbfa1edf648258f1ad56686eb1d959ecae672bb763c1baf7d889c6"} Oct 30 16:21:26.703816 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.703743 6202 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Oct 30 16:21:26.703816 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.703755 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" event={"ID":"d7ecc0c6cd8524d6054106d923cd48af","Type":"ContainerDied","Data":"a157ffac181ba6670e958e074630632719958af6aa11121031653652ddcaa994"} Oct 30 16:21:26.703816 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.703800 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" event={"ID":"d7ecc0c6cd8524d6054106d923cd48af","Type":"ContainerStarted","Data":"2ef5e9a5a1ba2ce970fe79131d48f63f2821bfef48f192fc8ec0fb5cc54415e3"} Oct 30 16:21:26.729900 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.729884 6202 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:21:26.729946 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:21:26.729927 6202 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-65.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Oct 30 16:21:26.730536 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.730522 6202 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:21:26.730567 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:21:26.730558 6202 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:21:26.828907 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.828888 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7ecc0c6cd8524d6054106d923cd48af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" (UID: \"d7ecc0c6cd8524d6054106d923cd48af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:21:26.828984 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.828915 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ecc0c6cd8524d6054106d923cd48af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" (UID: \"d7ecc0c6cd8524d6054106d923cd48af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:21:26.828984 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.828957 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/829ec3bbd4f4a9d896bac875cd5fa87a-config\") pod \"kube-apiserver-proxy-ip-10-0-129-65.ec2.internal\" (UID: \"829ec3bbd4f4a9d896bac875cd5fa87a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Oct 30 16:21:26.929492 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.929465 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7ecc0c6cd8524d6054106d923cd48af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" (UID: \"d7ecc0c6cd8524d6054106d923cd48af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:21:26.929492 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.929419 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7ecc0c6cd8524d6054106d923cd48af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" (UID: \"d7ecc0c6cd8524d6054106d923cd48af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:21:26.929626 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.929536 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ecc0c6cd8524d6054106d923cd48af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" (UID: \"d7ecc0c6cd8524d6054106d923cd48af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:21:26.929626 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.929556 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/829ec3bbd4f4a9d896bac875cd5fa87a-config\") pod \"kube-apiserver-proxy-ip-10-0-129-65.ec2.internal\" (UID: \"829ec3bbd4f4a9d896bac875cd5fa87a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Oct 30 16:21:26.929626 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.929582 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/829ec3bbd4f4a9d896bac875cd5fa87a-config\") pod \"kube-apiserver-proxy-ip-10-0-129-65.ec2.internal\" (UID: \"829ec3bbd4f4a9d896bac875cd5fa87a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Oct 30 16:21:26.929626 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:26.929621 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ecc0c6cd8524d6054106d923cd48af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" (UID: \"d7ecc0c6cd8524d6054106d923cd48af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:21:27.355488 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.355459 6202 apiserver.go:52] "Watching apiserver" Oct 30 16:21:27.412959 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.412933 6202 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 16:21:27.413632 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.413608 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-99l8n","openshift-cluster-node-tuning-operator/tuned-868tl","openshift-ingress-canary/ingress-canary-ckz2r","openshift-network-diagnostics/network-check-target-xchk6","openshift-network-operator/iptables-alerter-p694c","openshift-monitoring/prometheus-k8s-0","kube-system/global-pull-secret-syncer-k4k2w","openshift-monitoring/node-exporter-dktqn","openshift-monitoring/thanos-querier-59c44d56bd-vwqs7","openshift-multus/network-metrics-daemon-wtk5g","openshift-multus/multus-additional-cni-plugins-m72dq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2","openshift-image-registry/node-ca-7w7js","openshift-insights/insights-runtime-extractor-fd5ww","openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl","kube-system/konnectivity-agent-4n84w","kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal","openshift-dns/node-resolver-lr2cd","openshift-monitoring/telemeter-client-6b5445f54d-42vtq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-6sprd","openshift-dns/dns-default-xrskd"] Oct 30 16:21:27.430596 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.430566 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 16:21:27.430596 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.430569 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n7jsz\"" Oct 30 16:21:27.430852 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.430784 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 16:21:27.430973 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.430933 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 16:21:27.430973 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.430952 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 16:21:27.431087 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.430963 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hxhjs\"" Oct 30 16:21:27.431087 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.430998 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 16:21:27.431087 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.431010 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lngbh\"" Oct 30 16:21:27.431087 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.431045 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 16:21:27.431250 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.431120 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 16:21:27.431250 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.431129 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 16:21:27.431250 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.431210 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 16:21:27.431250 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.431214 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 16:21:27.431442 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.431249 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-m64dw\"" Oct 30 16:21:27.431442 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.431286 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:21:27.431442 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.431344 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:21:27.431442 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.431366 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 16:21:27.431584 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.431463 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:21:27.431584 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.431448 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sg8tz\"" Oct 30 16:21:27.431647 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.431588 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 16:21:27.431786 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.431757 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 16:21:27.434506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.434492 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 16:21:27.434578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.434566 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 16:21:27.434609 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.434590 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fpzw7\"" Oct 30 16:21:27.442807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.442795 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 16:21:27.443003 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.442990 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 16:21:27.443087 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.443075 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 16:21:27.443150 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.443132 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 16:21:27.443222 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.443207 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 16:21:27.443285 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.443274 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 16:21:27.443535 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.443525 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 16:21:27.443650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.443637 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 16:21:27.443690 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.443679 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 16:21:27.444223 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.444212 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 16:21:27.444263 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.444249 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 16:21:27.444263 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.444256 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 16:21:27.445412 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.445395 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 16:21:27.445512 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.445493 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-46f47\"" Oct 30 16:21:27.445570 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.445534 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 16:21:27.446055 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.446039 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 16:21:27.446158 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.446046 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 16:21:27.446158 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.446140 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 16:21:27.446267 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.446160 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 16:21:27.446267 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.446165 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 16:21:27.446267 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.446221 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jrplp\"" Oct 30 16:21:27.446267 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.446245 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 16:21:27.446889 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.446874 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 16:21:27.446946 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.446920 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 16:21:27.447000 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.446985 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 16:21:27.447000 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.446988 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 16:21:27.447141 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.446986 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 16:21:27.447889 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.447874 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 16:21:27.447978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.447939 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 16:21:27.448064 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.448042 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mwg5k\"" Oct 30 16:21:27.448064 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.448046 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 16:21:27.448327 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.448064 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 16:21:27.448327 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.448076 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 16:21:27.448327 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.448085 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 16:21:27.448327 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.448091 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 16:21:27.448327 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.448138 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ae6ube48gfedf\"" Oct 30 16:21:27.456680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.456632 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 16:21:27.456729 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.456685 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 16:21:27.456729 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.456693 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:21:27.456841 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.456739 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d8rcz\"" Oct 30 16:21:27.456841 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.456759 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l28xz\"" Oct 30 16:21:27.456841 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.456806 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 16:21:27.456841 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.456808 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5kvnn\"" Oct 30 16:21:27.456841 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.456817 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 16:21:27.456841 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.456828 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 16:21:27.457966 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.457945 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 16:21:27.457966 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.457945 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-m9dmv\"" Oct 30 16:21:27.458084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.457977 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6q4zp\"" Oct 30 16:21:27.458084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.457981 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qlnbk\"" Oct 30 16:21:27.458084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.457979 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 16:21:27.458084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.458022 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 16:21:27.458084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.458023 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 16:21:27.458084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.458071 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dnl1dtfqtp3b6\"" Oct 30 16:21:27.458353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.458099 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 16:21:27.458353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.458107 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-95j9k\"" Oct 30 16:21:27.458353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.458118 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 16:21:27.458353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.458127 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-v9kbq\"" Oct 30 16:21:27.458353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.458160 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 16:21:27.458353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.458167 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 16:21:27.458353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.458161 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-z7tv8\"" Oct 30 16:21:27.458353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.458181 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 16:21:27.458699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.458364 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 16:21:27.458699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.458369 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gfrzf\"" Oct 30 16:21:27.458699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.458398 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 16:21:27.458699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.458423 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 16:21:27.458699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.458596 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:21:27.459170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.459152 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 16:21:27.459296 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.459260 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:21:27.460549 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.460528 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s8kck\"" Oct 30 16:21:27.460972 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.460953 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 16:21:27.465937 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.465907 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 16:21:27.467587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.467563 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 16:21:27.468339 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.468324 6202 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Oct 30 16:21:27.468493 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.468479 6202 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:21:27.469741 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.469727 6202 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 16:21:27.490917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.490902 6202 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:21:27.490992 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:21:27.490943 6202 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-65.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Oct 30 16:21:27.491601 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.491589 6202 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:21:27.491644 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:21:27.491622 6202 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Oct 30 16:21:27.532286 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532257 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-run-netns\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.532286 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532289 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffdd3593-6091-4874-b55e-4049aa26ab60-kubelet-config\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:21:27.532445 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532315 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-accelerators-collector-config\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.532445 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532362 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-var-lib-kubelet\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.532445 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532395 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-run-netns\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.532527 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532480 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:27.532527 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532515 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85bfc58a-5f99-41ad-a651-05ba06bc2be1-tmp\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:27.532589 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532534 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86634960-264a-48a7-969a-5fbe6e58447e-tmp-dir\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:27.532589 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532551 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.532589 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532549 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-accelerators-collector-config\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.532589 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532570 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32528057-07fa-4eb9-882f-a43c9cd36b31-konnectivity-ca\") pod \"konnectivity-agent-4n84w\" (UID: \"32528057-07fa-4eb9-882f-a43c9cd36b31\") " pod="kube-system/konnectivity-agent-4n84w" Oct 30 16:21:27.532759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532599 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4512b7bb-6729-4045-aac9-4a1ece176016-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:21:27.532759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532626 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-registration-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.532759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532632 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85bfc58a-5f99-41ad-a651-05ba06bc2be1-tmp\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:27.532759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532649 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-run-systemd\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.532759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532632 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86634960-264a-48a7-969a-5fbe6e58447e-tmp-dir\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:27.532759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532684 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-run-ovn\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.532759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532726 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:27.532759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532730 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpwt\" (UniqueName: \"kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt\") pod \"network-check-target-xchk6\" (UID: \"73012827-44d9-4cec-9f8d-40b2055c0c69\") " pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:21:27.533163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532758 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86634960-264a-48a7-969a-5fbe6e58447e-config-volume\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:27.533163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532865 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4512b7bb-6729-4045-aac9-4a1ece176016-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:21:27.533163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532870 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.533163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532882 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f28s\" (UniqueName: \"kubernetes.io/projected/86634960-264a-48a7-969a-5fbe6e58447e-kube-api-access-5f28s\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:27.533163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532926 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32528057-07fa-4eb9-882f-a43c9cd36b31-konnectivity-ca\") pod \"konnectivity-agent-4n84w\" (UID: \"32528057-07fa-4eb9-882f-a43c9cd36b31\") " pod="kube-system/konnectivity-agent-4n84w" Oct 30 16:21:27.533163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532930 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9031958e-b627-4334-9e47-b9b62862c805-metrics-client-ca\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.533163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.532971 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86634960-264a-48a7-969a-5fbe6e58447e-config-volume\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:27.533163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533060 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-kubernetes\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.533163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533100 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-socket-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.533163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533115 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9031958e-b627-4334-9e47-b9b62862c805-metrics-client-ca\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.533163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533130 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-cni-netd\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.533477 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533179 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.533477 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533196 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-cnibin\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.533477 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533225 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-hostroot\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.533477 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533291 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c8t5\" (UniqueName: \"kubernetes.io/projected/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-kube-api-access-9c8t5\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:27.533477 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533318 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-slash\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.533477 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533341 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.533477 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533364 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.533477 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533381 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.533477 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533401 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-systemd-units\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.533822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533503 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmxkq\" (UniqueName: \"kubernetes.io/projected/25906571-b2ad-4134-9674-e67842de154d-kube-api-access-tmxkq\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.533822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533533 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.533822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533562 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-tls\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:27.533822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533594 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:27.533822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533601 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.533822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533534 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.533822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533621 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-sysconfig\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.533822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533647 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-sysctl-conf\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.533822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533795 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-etc-selinux\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.533822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533816 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17726533-46eb-4d51-bc1d-b4e9959d7ba5-host\") pod \"node-ca-7w7js\" (UID: \"17726533-46eb-4d51-bc1d-b4e9959d7ba5\") " pod="openshift-image-registry/node-ca-7w7js" Oct 30 16:21:27.533822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533820 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-tls\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:27.534241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533834 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/25906571-b2ad-4134-9674-e67842de154d-env-overrides\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.534241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533846 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:27.534241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533920 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc-tmp-dir\") pod \"node-resolver-lr2cd\" (UID: \"ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc\") " pod="openshift-dns/node-resolver-lr2cd" Oct 30 16:21:27.534241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533936 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/25906571-b2ad-4134-9674-e67842de154d-env-overrides\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.534241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.533976 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc-tmp-dir\") pod \"node-resolver-lr2cd\" (UID: \"ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc\") " pod="openshift-dns/node-resolver-lr2cd" Oct 30 16:21:27.534241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534002 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.534241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534033 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghbq5\" (UniqueName: \"kubernetes.io/projected/4512b7bb-6729-4045-aac9-4a1ece176016-kube-api-access-ghbq5\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:21:27.534241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534048 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.534241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534075 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:27.534241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534091 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-device-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.534241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534128 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/25906571-b2ad-4134-9674-e67842de154d-ovnkube-script-lib\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.534241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534158 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4512b7bb-6729-4045-aac9-4a1ece176016-cnibin\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:21:27.534241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534183 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9031958e-b627-4334-9e47-b9b62862c805-root\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.534241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534210 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffdd3593-6091-4874-b55e-4049aa26ab60-dbus\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:21:27.534241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534226 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:27.534819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534331 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:21:27.534819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534379 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-etc-openvswitch\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.534819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534386 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/25906571-b2ad-4134-9674-e67842de154d-ovnkube-script-lib\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.534819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534433 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.534819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534453 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32528057-07fa-4eb9-882f-a43c9cd36b31-agent-certs\") pod \"konnectivity-agent-4n84w\" (UID: \"32528057-07fa-4eb9-882f-a43c9cd36b31\") " pod="kube-system/konnectivity-agent-4n84w" Oct 30 16:21:27.534819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534522 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsbds\" (UniqueName: \"kubernetes.io/projected/9031958e-b627-4334-9e47-b9b62862c805-kube-api-access-fsbds\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.534819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534602 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.534819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534624 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffdd3593-6091-4874-b55e-4049aa26ab60-original-pull-secret\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:21:27.534819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534634 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-socket-dir-parent\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.534819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534660 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.534819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534679 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32528057-07fa-4eb9-882f-a43c9cd36b31-agent-certs\") pod \"konnectivity-agent-4n84w\" (UID: \"32528057-07fa-4eb9-882f-a43c9cd36b31\") " pod="kube-system/konnectivity-agent-4n84w" Oct 30 16:21:27.534819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534678 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.534819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534712 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-tuned\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.534819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534805 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-var-lib-cni-bin\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.534819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534816 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-tuned\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534853 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znx2b\" (UniqueName: \"kubernetes.io/projected/17726533-46eb-4d51-bc1d-b4e9959d7ba5-kube-api-access-znx2b\") pod \"node-ca-7w7js\" (UID: \"17726533-46eb-4d51-bc1d-b4e9959d7ba5\") " pod="openshift-image-registry/node-ca-7w7js" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534898 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534932 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k2rm\" (UniqueName: \"kubernetes.io/projected/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc-kube-api-access-5k2rm\") pod \"node-resolver-lr2cd\" (UID: \"ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc\") " pod="openshift-dns/node-resolver-lr2cd" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.534948 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535057 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4512b7bb-6729-4045-aac9-4a1ece176016-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535088 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b17ec352-725f-4bea-aaee-a3ca414fde6e-cni-binary-copy\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535114 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535115 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-run-k8s-cni-cncf-io\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535156 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535178 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd7dac20-991b-4048-acd5-08dcbdb57892-cert\") pod \"ingress-canary-ckz2r\" (UID: \"cd7dac20-991b-4048-acd5-08dcbdb57892\") " pod="openshift-ingress-canary/ingress-canary-ckz2r" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535194 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32474e2a-2206-4b53-a41e-6eaab7de8285-metrics-client-ca\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535305 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b17ec352-725f-4bea-aaee-a3ca414fde6e-cni-binary-copy\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535329 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535362 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-run\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535370 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535393 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535398 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32474e2a-2206-4b53-a41e-6eaab7de8285-metrics-client-ca\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535407 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd7dac20-991b-4048-acd5-08dcbdb57892-cert\") pod \"ingress-canary-ckz2r\" (UID: \"cd7dac20-991b-4048-acd5-08dcbdb57892\") " pod="openshift-ingress-canary/ingress-canary-ckz2r" Oct 30 16:21:27.535433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535426 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-sys-fs\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.536108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535488 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsrqq\" (UniqueName: \"kubernetes.io/projected/42aa85ec-676a-4f13-90f4-0fb13e94b22f-kube-api-access-dsrqq\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.536108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535528 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:27.536108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535603 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd4m6\" (UniqueName: \"kubernetes.io/projected/cd7dac20-991b-4048-acd5-08dcbdb57892-kube-api-access-vd4m6\") pod \"ingress-canary-ckz2r\" (UID: \"cd7dac20-991b-4048-acd5-08dcbdb57892\") " pod="openshift-ingress-canary/ingress-canary-ckz2r" Oct 30 16:21:27.536108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535618 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.536108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535711 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-secret-telemeter-client\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:27.536108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535820 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvfl4\" (UniqueName: \"kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-kube-api-access-xvfl4\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.536108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535937 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1bbf041-5696-4a2c-94a8-69ab0e995096-host-slash\") pod \"iptables-alerter-p694c\" (UID: \"c1bbf041-5696-4a2c-94a8-69ab0e995096\") " pod="openshift-network-operator/iptables-alerter-p694c" Oct 30 16:21:27.536108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535968 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-daemon-config\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.536108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535991 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.536108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535993 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2p9c\" (UniqueName: \"kubernetes.io/projected/c1bbf041-5696-4a2c-94a8-69ab0e995096-kube-api-access-c2p9c\") pod \"iptables-alerter-p694c\" (UID: \"c1bbf041-5696-4a2c-94a8-69ab0e995096\") " pod="openshift-network-operator/iptables-alerter-p694c" Oct 30 16:21:27.536108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.535996 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-secret-telemeter-client\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:27.536587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536121 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4512b7bb-6729-4045-aac9-4a1ece176016-cni-binary-copy\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:21:27.536587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536150 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-sys\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.536587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536154 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-daemon-config\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.536587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536197 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-var-lib-kubelet\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.536587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536219 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:27.536587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536236 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc-hosts-file\") pod \"node-resolver-lr2cd\" (UID: \"ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc\") " pod="openshift-dns/node-resolver-lr2cd" Oct 30 16:21:27.536587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536273 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-textfile\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.536587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536304 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4512b7bb-6729-4045-aac9-4a1ece176016-cni-binary-copy\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:21:27.536587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536311 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pjpq\" (UniqueName: \"kubernetes.io/projected/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-kube-api-access-5pjpq\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:27.536587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536340 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1bbf041-5696-4a2c-94a8-69ab0e995096-iptables-alerter-script\") pod \"iptables-alerter-p694c\" (UID: \"c1bbf041-5696-4a2c-94a8-69ab0e995096\") " pod="openshift-network-operator/iptables-alerter-p694c" Oct 30 16:21:27.536587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536356 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6r8b\" (UniqueName: \"kubernetes.io/projected/e7b597f3-5dc9-45e6-840f-b492ba06c241-kube-api-access-z6r8b\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.536587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536373 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-log-socket\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.536587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536374 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-textfile\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.536587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536389 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.536587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536492 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-cni-bin\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.536587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536519 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:27.536587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536567 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-config\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.537213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536618 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-var-lib-cni-multus\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.537213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536630 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1bbf041-5696-4a2c-94a8-69ab0e995096-iptables-alerter-script\") pod \"iptables-alerter-p694c\" (UID: \"c1bbf041-5696-4a2c-94a8-69ab0e995096\") " pod="openshift-network-operator/iptables-alerter-p694c" Oct 30 16:21:27.537213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536636 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-run-openvswitch\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.537213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536694 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.537213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536715 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wslqp\" (UniqueName: \"kubernetes.io/projected/35d777f2-4005-45e4-af04-2a61c0567fc0-kube-api-access-wslqp\") pod \"network-metrics-daemon-wtk5g\" (UID: \"35d777f2-4005-45e4-af04-2a61c0567fc0\") " pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:21:27.537213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536745 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9031958e-b627-4334-9e47-b9b62862c805-sys\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.537213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536803 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-conf-dir\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.537213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536816 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-config\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.537213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536827 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-run-multus-certs\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.537213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536843 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-crio-socket\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:27.537213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536877 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-lib-modules\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.537213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536918 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-cni-dir\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.537213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536954 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86634960-264a-48a7-969a-5fbe6e58447e-metrics-tls\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:27.537213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.536974 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32474e2a-2206-4b53-a41e-6eaab7de8285-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:27.537213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537013 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.537213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537119 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-etc-kubernetes\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.537213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537175 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86634960-264a-48a7-969a-5fbe6e58447e-metrics-tls\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:27.537213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537219 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:27.537853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537261 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:27.537853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537281 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.537853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537312 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32474e2a-2206-4b53-a41e-6eaab7de8285-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:27.537853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537366 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-telemeter-client-tls\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:27.537853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537398 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:27.537853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537462 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:27.537853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537489 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:27.537853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537541 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.537853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537598 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-telemeter-client-tls\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:27.537853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537636 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.537853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537667 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:27.537853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537707 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-host\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.537853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537738 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.537853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537803 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-kubelet\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.537853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537831 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32474e2a-2206-4b53-a41e-6eaab7de8285-serving-certs-ca-bundle\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:27.537853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537849 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.537853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537858 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.538506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537887 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-modprobe-d\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.538506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537912 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-765lm\" (UniqueName: \"kubernetes.io/projected/b17ec352-725f-4bea-aaee-a3ca414fde6e-kube-api-access-765lm\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.538506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537932 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-grpc-tls\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:27.538506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537955 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17726533-46eb-4d51-bc1d-b4e9959d7ba5-serviceca\") pod \"node-ca-7w7js\" (UID: \"17726533-46eb-4d51-bc1d-b4e9959d7ba5\") " pod="openshift-image-registry/node-ca-7w7js" Oct 30 16:21:27.538506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.537982 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32474e2a-2206-4b53-a41e-6eaab7de8285-serving-certs-ca-bundle\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:27.538506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538041 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/25906571-b2ad-4134-9674-e67842de154d-ovn-node-metrics-cert\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.538506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538132 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.538506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538231 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-grpc-tls\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:27.538506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538236 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-tls\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.538506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538326 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85bfc58a-5f99-41ad-a651-05ba06bc2be1-metrics-client-ca\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:27.538506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538401 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.538506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538414 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/25906571-b2ad-4134-9674-e67842de154d-ovn-node-metrics-cert\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.538506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538416 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.538506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538499 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-tls\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.538915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538515 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs\") pod \"network-metrics-daemon-wtk5g\" (UID: \"35d777f2-4005-45e4-af04-2a61c0567fc0\") " pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:21:27.538915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538546 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-wtmp\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.538915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538573 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-data-volume\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:27.538915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538577 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85bfc58a-5f99-41ad-a651-05ba06bc2be1-metrics-client-ca\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:27.538915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538601 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:27.538915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538616 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17726533-46eb-4d51-bc1d-b4e9959d7ba5-serviceca\") pod \"node-ca-7w7js\" (UID: \"17726533-46eb-4d51-bc1d-b4e9959d7ba5\") " pod="openshift-image-registry/node-ca-7w7js" Oct 30 16:21:27.538915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538628 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.538915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538638 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zthc\" (UniqueName: \"kubernetes.io/projected/85bfc58a-5f99-41ad-a651-05ba06bc2be1-kube-api-access-6zthc\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:27.538915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538632 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.538915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538665 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/25906571-b2ad-4134-9674-e67842de154d-ovnkube-config\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.538915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538684 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-data-volume\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:27.538915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538692 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d777f2-4005-45e4-af04-2a61c0567fc0-metrics-certs\") pod \"network-metrics-daemon-wtk5g\" (UID: \"35d777f2-4005-45e4-af04-2a61c0567fc0\") " pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:21:27.538915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538793 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.538915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538823 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-var-lib-openvswitch\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.538915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538842 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/85bfc58a-5f99-41ad-a651-05ba06bc2be1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:27.538915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538886 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-node-log\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.538915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538909 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-federate-client-tls\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:27.539483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538954 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.539483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.538954 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/25906571-b2ad-4134-9674-e67842de154d-ovnkube-config\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.539483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.539037 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shzmg\" (UniqueName: \"kubernetes.io/projected/32474e2a-2206-4b53-a41e-6eaab7de8285-kube-api-access-shzmg\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:27.539483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.539059 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4512b7bb-6729-4045-aac9-4a1ece176016-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:21:27.539483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.539081 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-sysctl-d\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.539483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.539097 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-systemd\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.539483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.539111 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7b597f3-5dc9-45e6-840f-b492ba06c241-tmp\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.539483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.539130 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-system-cni-dir\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.539483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.539140 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/32474e2a-2206-4b53-a41e-6eaab7de8285-federate-client-tls\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:27.539483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.539175 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.539483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.539195 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7b597f3-5dc9-45e6-840f-b492ba06c241-tmp\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.539483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.539206 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.539483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.539299 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4512b7bb-6729-4045-aac9-4a1ece176016-system-cni-dir\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:21:27.539483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.539309 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.539483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.539380 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4512b7bb-6729-4045-aac9-4a1ece176016-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:21:27.539483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.539454 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.555680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.555655 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mpwt\" (UniqueName: \"kubernetes.io/projected/73012827-44d9-4cec-9f8d-40b2055c0c69-kube-api-access-5mpwt\") pod \"network-check-target-xchk6\" (UID: \"73012827-44d9-4cec-9f8d-40b2055c0c69\") " pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:21:27.555814 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.555795 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f28s\" (UniqueName: \"kubernetes.io/projected/86634960-264a-48a7-969a-5fbe6e58447e-kube-api-access-5f28s\") pod \"dns-default-xrskd\" (UID: \"86634960-264a-48a7-969a-5fbe6e58447e\") " pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:27.555860 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.555798 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c8t5\" (UniqueName: \"kubernetes.io/projected/0dcde09b-ef11-4bb9-810b-883b0ee4cc18-kube-api-access-9c8t5\") pod \"prometheus-operator-9cd89b74c-rf8hl\" (UID: \"0dcde09b-ef11-4bb9-810b-883b0ee4cc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf8hl" Oct 30 16:21:27.556177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.556146 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghbq5\" (UniqueName: \"kubernetes.io/projected/4512b7bb-6729-4045-aac9-4a1ece176016-kube-api-access-ghbq5\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:21:27.556177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.556173 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmxkq\" (UniqueName: \"kubernetes.io/projected/25906571-b2ad-4134-9674-e67842de154d-kube-api-access-tmxkq\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.556276 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.556261 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsbds\" (UniqueName: \"kubernetes.io/projected/9031958e-b627-4334-9e47-b9b62862c805-kube-api-access-fsbds\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.556914 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.556899 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znx2b\" (UniqueName: \"kubernetes.io/projected/17726533-46eb-4d51-bc1d-b4e9959d7ba5-kube-api-access-znx2b\") pod \"node-ca-7w7js\" (UID: \"17726533-46eb-4d51-bc1d-b4e9959d7ba5\") " pod="openshift-image-registry/node-ca-7w7js" Oct 30 16:21:27.556971 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.556957 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k2rm\" (UniqueName: \"kubernetes.io/projected/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc-kube-api-access-5k2rm\") pod \"node-resolver-lr2cd\" (UID: \"ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc\") " pod="openshift-dns/node-resolver-lr2cd" Oct 30 16:21:27.557575 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.557559 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsrqq\" (UniqueName: \"kubernetes.io/projected/42aa85ec-676a-4f13-90f4-0fb13e94b22f-kube-api-access-dsrqq\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.558054 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.558034 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2p9c\" (UniqueName: \"kubernetes.io/projected/c1bbf041-5696-4a2c-94a8-69ab0e995096-kube-api-access-c2p9c\") pod \"iptables-alerter-p694c\" (UID: \"c1bbf041-5696-4a2c-94a8-69ab0e995096\") " pod="openshift-network-operator/iptables-alerter-p694c" Oct 30 16:21:27.558372 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.558353 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd4m6\" (UniqueName: \"kubernetes.io/projected/cd7dac20-991b-4048-acd5-08dcbdb57892-kube-api-access-vd4m6\") pod \"ingress-canary-ckz2r\" (UID: \"cd7dac20-991b-4048-acd5-08dcbdb57892\") " pod="openshift-ingress-canary/ingress-canary-ckz2r" Oct 30 16:21:27.558451 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.558424 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6r8b\" (UniqueName: \"kubernetes.io/projected/e7b597f3-5dc9-45e6-840f-b492ba06c241-kube-api-access-z6r8b\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.558497 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.558425 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvfl4\" (UniqueName: \"kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-kube-api-access-xvfl4\") pod \"prometheus-k8s-0\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:27.558497 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.558452 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pjpq\" (UniqueName: \"kubernetes.io/projected/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-kube-api-access-5pjpq\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:27.558959 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.558944 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wslqp\" (UniqueName: \"kubernetes.io/projected/35d777f2-4005-45e4-af04-2a61c0567fc0-kube-api-access-wslqp\") pod \"network-metrics-daemon-wtk5g\" (UID: \"35d777f2-4005-45e4-af04-2a61c0567fc0\") " pod="openshift-multus/network-metrics-daemon-wtk5g" Oct 30 16:21:27.561790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.561752 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-765lm\" (UniqueName: \"kubernetes.io/projected/b17ec352-725f-4bea-aaee-a3ca414fde6e-kube-api-access-765lm\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.561987 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.561971 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zthc\" (UniqueName: \"kubernetes.io/projected/85bfc58a-5f99-41ad-a651-05ba06bc2be1-kube-api-access-6zthc\") pod \"thanos-querier-59c44d56bd-vwqs7\" (UID: \"85bfc58a-5f99-41ad-a651-05ba06bc2be1\") " pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:27.566712 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.566691 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shzmg\" (UniqueName: \"kubernetes.io/projected/32474e2a-2206-4b53-a41e-6eaab7de8285-kube-api-access-shzmg\") pod \"telemeter-client-6b5445f54d-42vtq\" (UID: \"32474e2a-2206-4b53-a41e-6eaab7de8285\") " pod="openshift-monitoring/telemeter-client-6b5445f54d-42vtq" Oct 30 16:21:27.640641 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640567 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-run\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.640641 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640602 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-sys-fs\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.640641 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640619 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1bbf041-5696-4a2c-94a8-69ab0e995096-host-slash\") pod \"iptables-alerter-p694c\" (UID: \"c1bbf041-5696-4a2c-94a8-69ab0e995096\") " pod="openshift-network-operator/iptables-alerter-p694c" Oct 30 16:21:27.640641 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640634 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-sys\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.640901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640649 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-var-lib-kubelet\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.640901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640672 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc-hosts-file\") pod \"node-resolver-lr2cd\" (UID: \"ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc\") " pod="openshift-dns/node-resolver-lr2cd" Oct 30 16:21:27.640901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640691 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-run\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.640901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640706 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-sys-fs\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.640901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640695 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1bbf041-5696-4a2c-94a8-69ab0e995096-host-slash\") pod \"iptables-alerter-p694c\" (UID: \"c1bbf041-5696-4a2c-94a8-69ab0e995096\") " pod="openshift-network-operator/iptables-alerter-p694c" Oct 30 16:21:27.640901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640711 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-sys\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.640901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640732 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-log-socket\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.640901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640716 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-var-lib-kubelet\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.640901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640742 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc-hosts-file\") pod \"node-resolver-lr2cd\" (UID: \"ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc\") " pod="openshift-dns/node-resolver-lr2cd" Oct 30 16:21:27.640901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640698 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-log-socket\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.640901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640806 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.640901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640823 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-cni-bin\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.640901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640848 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-var-lib-cni-multus\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.640901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640864 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.640901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640873 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-run-openvswitch\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.640901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640901 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.640901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640905 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-cni-bin\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640926 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9031958e-b627-4334-9e47-b9b62862c805-sys\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640932 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-var-lib-cni-multus\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640949 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-run-openvswitch\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640972 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9031958e-b627-4334-9e47-b9b62862c805-sys\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640950 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-conf-dir\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640948 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.640991 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-conf-dir\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641009 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-run-multus-certs\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641028 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-crio-socket\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641046 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-run-multus-certs\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641044 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-lib-modules\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641070 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-cni-dir\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641081 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5-crio-socket\") pod \"insights-runtime-extractor-fd5ww\" (UID: \"ecfe68e7-95cd-46c9-ae4a-e2207e61b9c5\") " pod="openshift-insights/insights-runtime-extractor-fd5ww" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641085 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-etc-kubernetes\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641102 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-host\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641117 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-kubelet\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641134 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-modprobe-d\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641150 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-etc-kubernetes\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641164 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-wtmp\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641169 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-lib-modules\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641189 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-cni-dir\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641194 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-host\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641192 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-kubelet\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641192 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-var-lib-openvswitch\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641226 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-var-lib-openvswitch\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641227 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9031958e-b627-4334-9e47-b9b62862c805-node-exporter-wtmp\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641240 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-node-log\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641250 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-modprobe-d\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641259 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-sysctl-d\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641273 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-systemd\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641286 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-node-log\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641292 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-system-cni-dir\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641317 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4512b7bb-6729-4045-aac9-4a1ece176016-system-cni-dir\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641330 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-systemd\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641337 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-run-netns\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641358 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffdd3593-6091-4874-b55e-4049aa26ab60-kubelet-config\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641364 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4512b7bb-6729-4045-aac9-4a1ece176016-system-cni-dir\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641384 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-var-lib-kubelet\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641386 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-sysctl-d\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641363 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-system-cni-dir\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641397 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-run-netns\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641410 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-run-netns\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641431 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-var-lib-kubelet\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641436 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-registration-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641451 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-run-systemd\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641463 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-run-ovn\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.641458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641432 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffdd3593-6091-4874-b55e-4049aa26ab60-kubelet-config\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641489 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-kubernetes\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641512 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-registration-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641520 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-run-systemd\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641526 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-run-netns\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641521 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-run-ovn\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641514 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-socket-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641554 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-socket-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641555 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-cni-netd\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641569 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-kubernetes\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641574 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-cni-netd\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641577 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-cnibin\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641600 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-hostroot\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641608 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-cnibin\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641616 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-slash\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641638 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641654 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-systemd-units\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641666 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-hostroot\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641677 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-sysconfig\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641685 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-host-slash\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641694 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-sysctl-conf\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641702 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641707 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-sysconfig\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641707 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-etc-selinux\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641737 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-etc-selinux\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641749 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17726533-46eb-4d51-bc1d-b4e9959d7ba5-host\") pod \"node-ca-7w7js\" (UID: \"17726533-46eb-4d51-bc1d-b4e9959d7ba5\") " pod="openshift-image-registry/node-ca-7w7js" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641709 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-systemd-units\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641810 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e7b597f3-5dc9-45e6-840f-b492ba06c241-etc-sysctl-conf\") pod \"tuned-868tl\" (UID: \"e7b597f3-5dc9-45e6-840f-b492ba06c241\") " pod="openshift-cluster-node-tuning-operator/tuned-868tl" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641819 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17726533-46eb-4d51-bc1d-b4e9959d7ba5-host\") pod \"node-ca-7w7js\" (UID: \"17726533-46eb-4d51-bc1d-b4e9959d7ba5\") " pod="openshift-image-registry/node-ca-7w7js" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641820 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-device-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641843 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4512b7bb-6729-4045-aac9-4a1ece176016-cnibin\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641859 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9031958e-b627-4334-9e47-b9b62862c805-root\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641861 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42aa85ec-676a-4f13-90f4-0fb13e94b22f-device-dir\") pod \"aws-ebs-csi-driver-node-7stj2\" (UID: \"42aa85ec-676a-4f13-90f4-0fb13e94b22f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7stj2" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641884 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffdd3593-6091-4874-b55e-4049aa26ab60-dbus\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641915 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4512b7bb-6729-4045-aac9-4a1ece176016-cnibin\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641916 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-etc-openvswitch\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641933 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9031958e-b627-4334-9e47-b9b62862c805-root\") pod \"node-exporter-dktqn\" (UID: \"9031958e-b627-4334-9e47-b9b62862c805\") " pod="openshift-monitoring/node-exporter-dktqn" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641944 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25906571-b2ad-4134-9674-e67842de154d-etc-openvswitch\") pod \"ovnkube-node-6sprd\" (UID: \"25906571-b2ad-4134-9674-e67842de154d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641930 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffdd3593-6091-4874-b55e-4049aa26ab60-dbus\") pod \"global-pull-secret-syncer-k4k2w\" (UID: \"ffdd3593-6091-4874-b55e-4049aa26ab60\") " pod="kube-system/global-pull-secret-syncer-k4k2w" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641955 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-socket-dir-parent\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641977 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-var-lib-cni-bin\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641988 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-multus-socket-dir-parent\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.641993 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4512b7bb-6729-4045-aac9-4a1ece176016-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.642026 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-run-k8s-cni-cncf-io\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.642030 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4512b7bb-6729-4045-aac9-4a1ece176016-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m72dq\" (UID: \"4512b7bb-6729-4045-aac9-4a1ece176016\") " pod="openshift-multus/multus-additional-cni-plugins-m72dq" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.642026 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-var-lib-cni-bin\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:27.642897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:27.642061 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b17ec352-725f-4bea-aaee-a3ca414fde6e-host-run-k8s-cni-cncf-io\") pod \"multus-99l8n\" (UID: \"b17ec352-725f-4bea-aaee-a3ca414fde6e\") " pod="openshift-multus/multus-99l8n" Oct 30 16:21:28.075346 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:28.075317 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4n84w" Oct 30 16:21:28.321641 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:28.321603 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:28.441355 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:28.441260 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:28.652511 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:28.652480 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xrskd" Oct 30 16:21:28.975470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:28.975426 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:29.095296 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:29.095250 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:31.235738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:31.235709 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-59c44d56bd-vwqs7" Oct 30 16:21:31.862655 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:31.862628 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:32.001614 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:32.001582 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:32.020473 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:32.020449 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:32.942200 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:32.942168 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:32.942578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:32.942360 6202 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:21:33.656476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:33.656440 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:33.769077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:33.769045 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:34.713660 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:34.713626 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:34.862673 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:34.862636 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:35.774232 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:35.774153 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:21:40.435274 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:40.435247 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:40.746056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:40.745980 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:40.852292 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:40.852258 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:41.573038 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:41.573004 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:41.685444 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:41.685409 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:42.279688 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:42.279653 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:42.389843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:42.389819 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:43.025334 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:43.025298 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sprd" Oct 30 16:21:44.065779 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:21:44.065727 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:21:48.654145 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:48.654110 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:48.760790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:48.760743 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:50.450502 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.450474 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 16:21:50.450966 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.450868 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="prometheus" containerID="cri-o://bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254" gracePeriod=600 Oct 30 16:21:50.450966 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.450921 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="kube-rbac-proxy" containerID="cri-o://a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc" gracePeriod=600 Oct 30 16:21:50.451118 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.450969 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="thanos-sidecar" containerID="cri-o://bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0" gracePeriod=600 Oct 30 16:21:50.451118 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.451023 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="kube-rbac-proxy-web" containerID="cri-o://46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2" gracePeriod=600 Oct 30 16:21:50.451118 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.451065 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300" gracePeriod=600 Oct 30 16:21:50.451118 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.451087 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="config-reloader" containerID="cri-o://4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91" gracePeriod=600 Oct 30 16:21:50.822519 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.822486 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:50.919252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.919218 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-2\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.919419 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.919262 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-kube-api-access-xvfl4\" (UniqueName: \"kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-kube-api-access-xvfl4\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.919419 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.919290 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-tls\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.919419 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.919318 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-trusted-ca-bundle\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.919581 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.919496 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.919581 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.919562 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-1\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.919686 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.919594 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-metrics-client-ca\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.919686 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.919635 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-kubelet-serving-ca-bundle\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.919686 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.919633 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-2" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:50.919686 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.919668 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-0\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.919684 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-trusted-ca-bundle" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.919708 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-metrics-client-certs\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.920061 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-1" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.920245 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-kubelet-serving-ca-bundle" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.920263 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-web-config\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-web-config\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.920319 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-serving-certs-ca-bundle\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.920355 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-kube-rbac-proxy\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.920219 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-metrics-client-ca" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.920383 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-grpc-tls\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.920431 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-tls-assets\" (UniqueName: \"kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-tls-assets\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.920461 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-thanos-prometheus-http-client-file\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.920502 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.920533 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-db\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.920570 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-config\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-config\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.920599 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-config-out\") pod \"8e6698df-8df4-4b71-bba1-d276c753fb2a\" (UID: \"8e6698df-8df4-4b71-bba1-d276c753fb2a\") " Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.920839 6202 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.920861 6202 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.920878 6202 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.920892 6202 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-metrics-client-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.920910 6202 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.921388 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-0" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:50.922109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.921398 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-serving-certs-ca-bundle" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:50.923449 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.922750 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-db" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:21:50.923707 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.923583 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-tls" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:50.924074 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.923948 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-thanos-sidecar-tls" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:50.924074 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.924045 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-kube-api-access-xvfl4" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "kube-api-access-xvfl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:21:50.924690 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.924654 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-metrics-client-certs" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:50.924801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.924741 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-config-out" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:21:50.924869 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.924836 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-kube-rbac-proxy" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:50.924943 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.924920 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-grpc-tls" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:50.925031 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.925009 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-config" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:50.925189 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.925168 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-kube-rbac-proxy-web" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:50.925503 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.925479 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-tls-assets" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:21:50.925606 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.925584 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-thanos-prometheus-http-client-file" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:50.934065 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:50.934041 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-web-config" pod "8e6698df-8df4-4b71-bba1-d276c753fb2a" (UID: "8e6698df-8df4-4b71-bba1-d276c753fb2a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:51.022048 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.021988 6202 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:51.022048 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.022010 6202 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:51.022048 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.022020 6202 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-metrics-client-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:51.022048 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.022028 6202 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-web-config\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:51.022048 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.022038 6202 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6698df-8df4-4b71-bba1-d276c753fb2a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:51.022048 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.022047 6202 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-kube-rbac-proxy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:51.022310 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.022056 6202 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-grpc-tls\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:51.022310 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.022065 6202 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-tls-assets\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:51.022310 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.022072 6202 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:51.022310 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.022080 6202 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:51.022310 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.022089 6202 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-prometheus-k8s-db\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:51.022310 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.022097 6202 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-config\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:51.022310 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.022105 6202 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e6698df-8df4-4b71-bba1-d276c753fb2a-config-out\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:51.022310 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.022114 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvfl4\" (UniqueName: \"kubernetes.io/projected/8e6698df-8df4-4b71-bba1-d276c753fb2a-kube-api-access-xvfl4\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:51.022310 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.022122 6202 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e6698df-8df4-4b71-bba1-d276c753fb2a-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:21:51.270036 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.270004 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:51.385142 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.385105 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:51.497609 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.497583 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:51.555827 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.555797 6202 generic.go:356] "Generic (PLEG): container finished" podID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerID="81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300" exitCode=0 Oct 30 16:21:51.555827 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.555819 6202 generic.go:356] "Generic (PLEG): container finished" podID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerID="a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc" exitCode=0 Oct 30 16:21:51.555827 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.555825 6202 generic.go:356] "Generic (PLEG): container finished" podID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerID="46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2" exitCode=0 Oct 30 16:21:51.555827 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.555832 6202 generic.go:356] "Generic (PLEG): container finished" podID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerID="bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0" exitCode=0 Oct 30 16:21:51.556107 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.555840 6202 generic.go:356] "Generic (PLEG): container finished" podID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerID="4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91" exitCode=0 Oct 30 16:21:51.556107 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.555846 6202 generic.go:356] "Generic (PLEG): container finished" podID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerID="bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254" exitCode=0 Oct 30 16:21:51.556107 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.555874 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6698df-8df4-4b71-bba1-d276c753fb2a","Type":"ContainerDied","Data":"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300"} Oct 30 16:21:51.556107 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.555942 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6698df-8df4-4b71-bba1-d276c753fb2a","Type":"ContainerDied","Data":"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc"} Oct 30 16:21:51.556107 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.555885 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.556107 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.555981 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6698df-8df4-4b71-bba1-d276c753fb2a","Type":"ContainerDied","Data":"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2"} Oct 30 16:21:51.556107 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.556007 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6698df-8df4-4b71-bba1-d276c753fb2a","Type":"ContainerDied","Data":"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0"} Oct 30 16:21:51.556107 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.556035 6202 scope.go:122] "RemoveContainer" containerID="81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300" Oct 30 16:21:51.556107 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.556064 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6698df-8df4-4b71-bba1-d276c753fb2a","Type":"ContainerDied","Data":"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91"} Oct 30 16:21:51.556107 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.556094 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6698df-8df4-4b71-bba1-d276c753fb2a","Type":"ContainerDied","Data":"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254"} Oct 30 16:21:51.556567 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.556122 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6698df-8df4-4b71-bba1-d276c753fb2a","Type":"ContainerDied","Data":"286b205a58dc7fae2093663c64c5d8d53e424c94c5a8349577030ea935356632"} Oct 30 16:21:51.563895 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.563880 6202 scope.go:122] "RemoveContainer" containerID="a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc" Oct 30 16:21:51.570491 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.570472 6202 scope.go:122] "RemoveContainer" containerID="46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2" Oct 30 16:21:51.576895 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.576879 6202 scope.go:122] "RemoveContainer" containerID="bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0" Oct 30 16:21:51.583287 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.583268 6202 scope.go:122] "RemoveContainer" containerID="4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91" Oct 30 16:21:51.589522 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.589506 6202 scope.go:122] "RemoveContainer" containerID="bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254" Oct 30 16:21:51.597337 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.597299 6202 scope.go:122] "RemoveContainer" containerID="68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd" Oct 30 16:21:51.603993 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.603976 6202 scope.go:122] "RemoveContainer" containerID="81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300" Oct 30 16:21:51.604246 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:21:51.604228 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300\": container with ID starting with 81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300 not found: ID does not exist" containerID="81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300" Oct 30 16:21:51.604316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.604252 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300"} err="failed to get container status \"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300\": rpc error: code = NotFound desc = could not find container \"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300\": container with ID starting with 81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300 not found: ID does not exist" Oct 30 16:21:51.604316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.604284 6202 scope.go:122] "RemoveContainer" containerID="a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc" Oct 30 16:21:51.604496 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:21:51.604480 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc\": container with ID starting with a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc not found: ID does not exist" containerID="a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc" Oct 30 16:21:51.604555 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.604498 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc"} err="failed to get container status \"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc\": rpc error: code = NotFound desc = could not find container \"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc\": container with ID starting with a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc not found: ID does not exist" Oct 30 16:21:51.604555 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.604510 6202 scope.go:122] "RemoveContainer" containerID="46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2" Oct 30 16:21:51.604792 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:21:51.604758 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2\": container with ID starting with 46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2 not found: ID does not exist" containerID="46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2" Oct 30 16:21:51.604831 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.604797 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2"} err="failed to get container status \"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2\": rpc error: code = NotFound desc = could not find container \"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2\": container with ID starting with 46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2 not found: ID does not exist" Oct 30 16:21:51.604831 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.604815 6202 scope.go:122] "RemoveContainer" containerID="bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0" Oct 30 16:21:51.605052 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:21:51.605035 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0\": container with ID starting with bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0 not found: ID does not exist" containerID="bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0" Oct 30 16:21:51.605125 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.605060 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0"} err="failed to get container status \"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0\": rpc error: code = NotFound desc = could not find container \"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0\": container with ID starting with bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0 not found: ID does not exist" Oct 30 16:21:51.605125 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.605079 6202 scope.go:122] "RemoveContainer" containerID="4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91" Oct 30 16:21:51.605316 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:21:51.605301 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91\": container with ID starting with 4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91 not found: ID does not exist" containerID="4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91" Oct 30 16:21:51.605355 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.605321 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91"} err="failed to get container status \"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91\": rpc error: code = NotFound desc = could not find container \"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91\": container with ID starting with 4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91 not found: ID does not exist" Oct 30 16:21:51.605355 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.605335 6202 scope.go:122] "RemoveContainer" containerID="bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254" Oct 30 16:21:51.605543 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:21:51.605527 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254\": container with ID starting with bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254 not found: ID does not exist" containerID="bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254" Oct 30 16:21:51.605637 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.605550 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254"} err="failed to get container status \"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254\": rpc error: code = NotFound desc = could not find container \"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254\": container with ID starting with bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254 not found: ID does not exist" Oct 30 16:21:51.605637 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.605568 6202 scope.go:122] "RemoveContainer" containerID="68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd" Oct 30 16:21:51.605810 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:21:51.605797 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd\": container with ID starting with 68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd not found: ID does not exist" containerID="68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd" Oct 30 16:21:51.605854 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.605813 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd"} err="failed to get container status \"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd\": rpc error: code = NotFound desc = could not find container \"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd\": container with ID starting with 68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd not found: ID does not exist" Oct 30 16:21:51.605854 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.605824 6202 scope.go:122] "RemoveContainer" containerID="81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300" Oct 30 16:21:51.606026 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.606004 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300"} err="failed to get container status \"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300\": rpc error: code = NotFound desc = could not find container \"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300\": container with ID starting with 81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300 not found: ID does not exist" Oct 30 16:21:51.606077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.606027 6202 scope.go:122] "RemoveContainer" containerID="a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc" Oct 30 16:21:51.606272 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.606249 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc"} err="failed to get container status \"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc\": rpc error: code = NotFound desc = could not find container \"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc\": container with ID starting with a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc not found: ID does not exist" Oct 30 16:21:51.606315 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.606272 6202 scope.go:122] "RemoveContainer" containerID="46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2" Oct 30 16:21:51.606482 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.606464 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2"} err="failed to get container status \"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2\": rpc error: code = NotFound desc = could not find container \"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2\": container with ID starting with 46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2 not found: ID does not exist" Oct 30 16:21:51.606553 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.606483 6202 scope.go:122] "RemoveContainer" containerID="bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0" Oct 30 16:21:51.606701 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.606687 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0"} err="failed to get container status \"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0\": rpc error: code = NotFound desc = could not find container \"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0\": container with ID starting with bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0 not found: ID does not exist" Oct 30 16:21:51.606744 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.606701 6202 scope.go:122] "RemoveContainer" containerID="4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91" Oct 30 16:21:51.606952 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.606934 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91"} err="failed to get container status \"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91\": rpc error: code = NotFound desc = could not find container \"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91\": container with ID starting with 4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91 not found: ID does not exist" Oct 30 16:21:51.607010 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.606952 6202 scope.go:122] "RemoveContainer" containerID="bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254" Oct 30 16:21:51.607201 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.607184 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254"} err="failed to get container status \"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254\": rpc error: code = NotFound desc = could not find container \"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254\": container with ID starting with bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254 not found: ID does not exist" Oct 30 16:21:51.607242 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.607202 6202 scope.go:122] "RemoveContainer" containerID="68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd" Oct 30 16:21:51.607430 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.607410 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd"} err="failed to get container status \"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd\": rpc error: code = NotFound desc = could not find container \"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd\": container with ID starting with 68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd not found: ID does not exist" Oct 30 16:21:51.607430 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.607428 6202 scope.go:122] "RemoveContainer" containerID="81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300" Oct 30 16:21:51.607649 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.607633 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300"} err="failed to get container status \"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300\": rpc error: code = NotFound desc = could not find container \"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300\": container with ID starting with 81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300 not found: ID does not exist" Oct 30 16:21:51.607649 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.607649 6202 scope.go:122] "RemoveContainer" containerID="a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc" Oct 30 16:21:51.607876 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.607860 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc"} err="failed to get container status \"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc\": rpc error: code = NotFound desc = could not find container \"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc\": container with ID starting with a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc not found: ID does not exist" Oct 30 16:21:51.607876 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.607875 6202 scope.go:122] "RemoveContainer" containerID="46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2" Oct 30 16:21:51.608050 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.608035 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2"} err="failed to get container status \"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2\": rpc error: code = NotFound desc = could not find container \"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2\": container with ID starting with 46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2 not found: ID does not exist" Oct 30 16:21:51.608088 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.608049 6202 scope.go:122] "RemoveContainer" containerID="bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0" Oct 30 16:21:51.608283 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.608266 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0"} err="failed to get container status \"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0\": rpc error: code = NotFound desc = could not find container \"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0\": container with ID starting with bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0 not found: ID does not exist" Oct 30 16:21:51.608283 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.608284 6202 scope.go:122] "RemoveContainer" containerID="4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91" Oct 30 16:21:51.608495 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.608476 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91"} err="failed to get container status \"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91\": rpc error: code = NotFound desc = could not find container \"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91\": container with ID starting with 4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91 not found: ID does not exist" Oct 30 16:21:51.608495 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.608492 6202 scope.go:122] "RemoveContainer" containerID="bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254" Oct 30 16:21:51.608687 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.608674 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254"} err="failed to get container status \"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254\": rpc error: code = NotFound desc = could not find container \"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254\": container with ID starting with bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254 not found: ID does not exist" Oct 30 16:21:51.608687 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.608687 6202 scope.go:122] "RemoveContainer" containerID="68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd" Oct 30 16:21:51.608903 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.608888 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd"} err="failed to get container status \"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd\": rpc error: code = NotFound desc = could not find container \"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd\": container with ID starting with 68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd not found: ID does not exist" Oct 30 16:21:51.608903 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.608902 6202 scope.go:122] "RemoveContainer" containerID="81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300" Oct 30 16:21:51.609093 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.609073 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300"} err="failed to get container status \"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300\": rpc error: code = NotFound desc = could not find container \"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300\": container with ID starting with 81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300 not found: ID does not exist" Oct 30 16:21:51.609162 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.609094 6202 scope.go:122] "RemoveContainer" containerID="a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc" Oct 30 16:21:51.609315 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.609300 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc"} err="failed to get container status \"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc\": rpc error: code = NotFound desc = could not find container \"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc\": container with ID starting with a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc not found: ID does not exist" Oct 30 16:21:51.609315 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.609314 6202 scope.go:122] "RemoveContainer" containerID="46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2" Oct 30 16:21:51.609491 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.609476 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2"} err="failed to get container status \"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2\": rpc error: code = NotFound desc = could not find container \"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2\": container with ID starting with 46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2 not found: ID does not exist" Oct 30 16:21:51.609538 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.609491 6202 scope.go:122] "RemoveContainer" containerID="bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0" Oct 30 16:21:51.609692 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.609680 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0"} err="failed to get container status \"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0\": rpc error: code = NotFound desc = could not find container \"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0\": container with ID starting with bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0 not found: ID does not exist" Oct 30 16:21:51.609692 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.609692 6202 scope.go:122] "RemoveContainer" containerID="4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91" Oct 30 16:21:51.609922 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.609903 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91"} err="failed to get container status \"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91\": rpc error: code = NotFound desc = could not find container \"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91\": container with ID starting with 4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91 not found: ID does not exist" Oct 30 16:21:51.609966 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.609924 6202 scope.go:122] "RemoveContainer" containerID="bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254" Oct 30 16:21:51.610178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.610159 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254"} err="failed to get container status \"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254\": rpc error: code = NotFound desc = could not find container \"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254\": container with ID starting with bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254 not found: ID does not exist" Oct 30 16:21:51.610287 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.610179 6202 scope.go:122] "RemoveContainer" containerID="68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd" Oct 30 16:21:51.610409 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.610392 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd"} err="failed to get container status \"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd\": rpc error: code = NotFound desc = could not find container \"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd\": container with ID starting with 68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd not found: ID does not exist" Oct 30 16:21:51.610452 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.610409 6202 scope.go:122] "RemoveContainer" containerID="81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300" Oct 30 16:21:51.610627 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.610608 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300"} err="failed to get container status \"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300\": rpc error: code = NotFound desc = could not find container \"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300\": container with ID starting with 81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300 not found: ID does not exist" Oct 30 16:21:51.610687 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.610629 6202 scope.go:122] "RemoveContainer" containerID="a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc" Oct 30 16:21:51.610894 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.610874 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc"} err="failed to get container status \"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc\": rpc error: code = NotFound desc = could not find container \"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc\": container with ID starting with a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc not found: ID does not exist" Oct 30 16:21:51.610894 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.610894 6202 scope.go:122] "RemoveContainer" containerID="46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2" Oct 30 16:21:51.611151 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.611136 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2"} err="failed to get container status \"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2\": rpc error: code = NotFound desc = could not find container \"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2\": container with ID starting with 46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2 not found: ID does not exist" Oct 30 16:21:51.611151 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.611151 6202 scope.go:122] "RemoveContainer" containerID="bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0" Oct 30 16:21:51.611366 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.611347 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0"} err="failed to get container status \"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0\": rpc error: code = NotFound desc = could not find container \"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0\": container with ID starting with bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0 not found: ID does not exist" Oct 30 16:21:51.611407 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.611367 6202 scope.go:122] "RemoveContainer" containerID="4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91" Oct 30 16:21:51.611593 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.611575 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91"} err="failed to get container status \"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91\": rpc error: code = NotFound desc = could not find container \"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91\": container with ID starting with 4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91 not found: ID does not exist" Oct 30 16:21:51.611659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.611593 6202 scope.go:122] "RemoveContainer" containerID="bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254" Oct 30 16:21:51.611827 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.611812 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254"} err="failed to get container status \"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254\": rpc error: code = NotFound desc = could not find container \"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254\": container with ID starting with bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254 not found: ID does not exist" Oct 30 16:21:51.611827 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.611826 6202 scope.go:122] "RemoveContainer" containerID="68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd" Oct 30 16:21:51.612066 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.612050 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd"} err="failed to get container status \"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd\": rpc error: code = NotFound desc = could not find container \"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd\": container with ID starting with 68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd not found: ID does not exist" Oct 30 16:21:51.612066 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.612066 6202 scope.go:122] "RemoveContainer" containerID="81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300" Oct 30 16:21:51.612302 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.612286 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300"} err="failed to get container status \"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300\": rpc error: code = NotFound desc = could not find container \"81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300\": container with ID starting with 81d7412f95eb2b1d7306c7e3309e0e1837c72b104fe22258f2e5ec0324107300 not found: ID does not exist" Oct 30 16:21:51.612340 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.612303 6202 scope.go:122] "RemoveContainer" containerID="a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc" Oct 30 16:21:51.612487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.612471 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc"} err="failed to get container status \"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc\": rpc error: code = NotFound desc = could not find container \"a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc\": container with ID starting with a7742d5eaeb19e27ae5f765d1f7c5ca8da6f0fcd3756d0084252be98e1b5ddcc not found: ID does not exist" Oct 30 16:21:51.612532 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.612488 6202 scope.go:122] "RemoveContainer" containerID="46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2" Oct 30 16:21:51.612682 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.612663 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2"} err="failed to get container status \"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2\": rpc error: code = NotFound desc = could not find container \"46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2\": container with ID starting with 46d7c63e56c7be53019d5436618fcafcaa219c08b9a4fae0f8c9838b0eb312d2 not found: ID does not exist" Oct 30 16:21:51.612742 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.612684 6202 scope.go:122] "RemoveContainer" containerID="bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0" Oct 30 16:21:51.612929 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.612912 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0"} err="failed to get container status \"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0\": rpc error: code = NotFound desc = could not find container \"bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0\": container with ID starting with bd643a49eaba3dc3c63083b7844d7beca37c7b5a7d98999e8d923e79849160d0 not found: ID does not exist" Oct 30 16:21:51.612978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.612931 6202 scope.go:122] "RemoveContainer" containerID="4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91" Oct 30 16:21:51.613136 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.613119 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91"} err="failed to get container status \"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91\": rpc error: code = NotFound desc = could not find container \"4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91\": container with ID starting with 4a4bbffc00e6e0632ff89b8620740e0a8cd2b47a904cd87724d1d15846f59e91 not found: ID does not exist" Oct 30 16:21:51.613174 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.613137 6202 scope.go:122] "RemoveContainer" containerID="bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254" Oct 30 16:21:51.613354 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.613339 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254"} err="failed to get container status \"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254\": rpc error: code = NotFound desc = could not find container \"bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254\": container with ID starting with bebeab9161c628939871310d7548b314c4d88cd26352dea884dc526fee41a254 not found: ID does not exist" Oct 30 16:21:51.613412 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.613353 6202 scope.go:122] "RemoveContainer" containerID="68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd" Oct 30 16:21:51.613551 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.613534 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd"} err="failed to get container status \"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd\": rpc error: code = NotFound desc = could not find container \"68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd\": container with ID starting with 68cd871fe26afba10e0304473d83e3faee3c02948d68dae13907acb827da64cd not found: ID does not exist" Oct 30 16:21:51.626783 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.626748 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 16:21:51.640147 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.640096 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 16:21:51.662826 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.662810 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 16:21:51.663060 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663042 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="kube-rbac-proxy-web" Oct 30 16:21:51.663060 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663054 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="kube-rbac-proxy-web" Oct 30 16:21:51.663199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663073 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="kube-rbac-proxy" Oct 30 16:21:51.663199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663080 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="kube-rbac-proxy" Oct 30 16:21:51.663199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663090 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="prometheus" Oct 30 16:21:51.663199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663098 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="prometheus" Oct 30 16:21:51.663199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663122 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="config-reloader" Oct 30 16:21:51.663199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663129 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="config-reloader" Oct 30 16:21:51.663199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663141 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="kube-rbac-proxy-thanos" Oct 30 16:21:51.663199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663148 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="kube-rbac-proxy-thanos" Oct 30 16:21:51.663199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663158 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="init-config-reloader" Oct 30 16:21:51.663199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663165 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="init-config-reloader" Oct 30 16:21:51.663199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663179 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="thanos-sidecar" Oct 30 16:21:51.663199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663183 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="thanos-sidecar" Oct 30 16:21:51.663676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663240 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="kube-rbac-proxy-thanos" Oct 30 16:21:51.663676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663251 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="init-config-reloader" Oct 30 16:21:51.663676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663262 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="thanos-sidecar" Oct 30 16:21:51.663676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663273 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="kube-rbac-proxy-web" Oct 30 16:21:51.663676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663282 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="kube-rbac-proxy" Oct 30 16:21:51.663676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663287 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="prometheus" Oct 30 16:21:51.663676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.663294 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" containerName="config-reloader" Oct 30 16:21:51.693561 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.693542 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.697300 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.697278 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 16:21:51.715203 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.715182 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 16:21:51.727461 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.727439 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.727537 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.727471 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.727537 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.727494 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-web-config\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.727640 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.727550 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.727640 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.727616 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.727712 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.727676 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.727712 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.727694 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.727789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.727711 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn7zs\" (UniqueName: \"kubernetes.io/projected/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-kube-api-access-jn7zs\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.727789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.727727 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.727854 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.727827 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.727886 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.727852 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.727886 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.727873 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.727987 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.727904 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.727987 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.727932 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.727987 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.727948 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.727987 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.727979 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.728188 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.728035 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.728188 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.728057 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-config-out\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.728188 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.728080 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.728188 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.728113 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-config\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.728382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.728191 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 16:21:51.728382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.728216 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 16:21:51.728382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.728251 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 16:21:51.728382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.728257 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 16:21:51.728382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.728272 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 16:21:51.728382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.728284 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 16:21:51.728382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.728292 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 16:21:51.728382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.728319 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 16:21:51.728382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.728323 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 16:21:51.728382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.728376 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mwg5k\"" Oct 30 16:21:51.728676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.728447 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ae6ube48gfedf\"" Oct 30 16:21:51.788125 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.788098 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:51.816331 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.816306 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 16:21:51.816407 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.816399 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 16:21:51.816475 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.816454 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 16:21:51.816854 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.816831 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 16:21:51.817909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.817895 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 16:21:51.828397 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828377 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-web-config\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.828454 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828403 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.828454 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828418 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.828454 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828435 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.828551 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828458 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.828595 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828573 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn7zs\" (UniqueName: \"kubernetes.io/projected/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-kube-api-access-jn7zs\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.828643 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828605 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.828689 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828639 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.828689 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828663 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.828808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828688 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.828808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828753 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.828808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828798 6202 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 16:21:51.828984 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828804 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.828984 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828835 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.828984 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828861 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.828984 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828897 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.828984 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828923 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-config-out\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.828984 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828947 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.829318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.828987 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-config\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.829318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.829037 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.829318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.829071 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.829318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.829228 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.829671 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.829473 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.829671 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.829478 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.829671 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.829618 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.829857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.829832 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.830474 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.830446 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.830990 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.830964 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.831752 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.831723 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.831930 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.831908 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.832180 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.832160 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-web-config\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.832180 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.832174 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.833736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.833555 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.833736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.833635 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-config\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.833736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.833690 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-config-out\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.834303 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.834273 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.834379 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.834305 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.834709 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.834691 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.834821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.834797 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.835127 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.835109 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.864300 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.864283 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn7zs\" (UniqueName: \"kubernetes.io/projected/45db0ed9-af63-4d3f-a393-aacbf2e6ad58-kube-api-access-jn7zs\") pod \"prometheus-k8s-0\" (UID: \"45db0ed9-af63-4d3f-a393-aacbf2e6ad58\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:51.937950 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:51.937890 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:52.103196 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:52.103164 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:52.407588 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:52.407558 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e6698df-8df4-4b71-bba1-d276c753fb2a" path="/var/lib/kubelet/pods/8e6698df-8df4-4b71-bba1-d276c753fb2a/volumes" Oct 30 16:21:52.457274 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:52.457248 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 16:21:52.458718 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:21:52.458681 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45db0ed9_af63_4d3f_a393_aacbf2e6ad58.slice/crio-04bdfe8cdc9abf3b4928345b53454f6926050cdd68e4d787beb51341cd80ad5d WatchSource:0}: Error finding container 04bdfe8cdc9abf3b4928345b53454f6926050cdd68e4d787beb51341cd80ad5d: Status 404 returned error can't find the container with id 04bdfe8cdc9abf3b4928345b53454f6926050cdd68e4d787beb51341cd80ad5d Oct 30 16:21:52.564860 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:52.564830 6202 generic.go:356] "Generic (PLEG): container finished" podID="45db0ed9-af63-4d3f-a393-aacbf2e6ad58" containerID="043c940b4d8c58ecc6badf2dbd9536340c5c69d8f96f5745bcec7463265fbb8c" exitCode=0 Oct 30 16:21:52.565256 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:52.564879 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45db0ed9-af63-4d3f-a393-aacbf2e6ad58","Type":"ContainerDied","Data":"043c940b4d8c58ecc6badf2dbd9536340c5c69d8f96f5745bcec7463265fbb8c"} Oct 30 16:21:52.565256 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:52.564948 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45db0ed9-af63-4d3f-a393-aacbf2e6ad58","Type":"ContainerStarted","Data":"04bdfe8cdc9abf3b4928345b53454f6926050cdd68e4d787beb51341cd80ad5d"} Oct 30 16:21:53.572139 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:53.572100 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45db0ed9-af63-4d3f-a393-aacbf2e6ad58","Type":"ContainerStarted","Data":"fccb9b7b8193604c131edac79c934566106b340a88731d9a7b1c6e1b738b6326"} Oct 30 16:21:53.572621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:53.572173 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45db0ed9-af63-4d3f-a393-aacbf2e6ad58","Type":"ContainerStarted","Data":"1617b69db16077a50fc50a2576f953aaf2a4abf8e897b0ebcf9be2dd5fedccd8"} Oct 30 16:21:53.572621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:53.572198 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45db0ed9-af63-4d3f-a393-aacbf2e6ad58","Type":"ContainerStarted","Data":"cdb31337e287b58f6fb285d9e6a6e34ff1a6cf166cc2377a6dcf18036285a496"} Oct 30 16:21:53.572621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:53.572223 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45db0ed9-af63-4d3f-a393-aacbf2e6ad58","Type":"ContainerStarted","Data":"0426f935d4dfd0e223694710600c69c6a9ffba2c03ac24432e59be34d3160a0d"} Oct 30 16:21:53.572621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:53.572244 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45db0ed9-af63-4d3f-a393-aacbf2e6ad58","Type":"ContainerStarted","Data":"90df337e72bb5f1c19262e3b54a25dc7889a24efb0a1a31f7ff558c57275f281"} Oct 30 16:21:53.572621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:53.572282 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45db0ed9-af63-4d3f-a393-aacbf2e6ad58","Type":"ContainerStarted","Data":"fe7e635c4a0686768da30b522dae4c6d25d86277f5c16e49b2f0b58403150e4d"} Oct 30 16:21:53.670942 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:53.670892 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.655970023 podStartE2EDuration="2.670879097s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.909074ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:53.657535508 +0000 UTC m=+27.367990252" watchObservedRunningTime="2026-10-30 16:21:53.670879097 +0000 UTC m=+27.381333841" Oct 30 16:21:54.132574 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:54.132537 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:54.148606 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:54.148575 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:54.576205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:54.576177 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:54.591587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:54.591566 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 16:21:55.513309 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:55.513275 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:55.628314 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:55.628284 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:55.777341 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:55.777274 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xchk6" Oct 30 16:21:59.031820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:59.031761 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:21:59.151845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:21:59.151788 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:03.656445 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:22:03.656398 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:04.405002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:22:04.404974 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:14.066328 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:22:14.066294 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:22:16.548879 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:22:16.548847 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:17.034744 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:22:17.034711 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:18.656252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:22:18.656200 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:18.772981 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:22:18.772951 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:23.609526 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:22:23.609489 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:23.731366 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:22:23.731332 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:30.757847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:22:30.757810 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:30.870332 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:22:30.870298 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:34.846622 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:22:34.846586 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:44.066211 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:22:44.066169 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:22:49.498528 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:22:49.498479 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:49.637382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:22:49.637355 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:14.066364 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:23:14.066326 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:23:21.084969 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:23:21.084925 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:34.958239 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:23:34.958168 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:44.066064 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:23:44.066026 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:24:12.795493 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:12.795462 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:12.981936 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:12.981900 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:14.065321 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:24:14.065285 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:24:19.134276 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:19.134237 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:26.591019 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:26.590973 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:34.715757 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:34.715724 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:35.085351 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:35.085318 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:35.575506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:35.575477 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:38.126829 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:38.126794 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:38.254714 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:38.254679 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:40.877812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:40.877749 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:41.001492 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:41.001464 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:41.406148 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:41.406114 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:41.579020 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:41.578990 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:43.293148 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:43.293111 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:43.538779 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:43.538723 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:44.066990 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:24:44.066957 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:24:44.111681 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:44.111649 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:44.238867 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:44.238835 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:47.629319 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:47.629282 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-s8jwx"] Oct 30 16:24:47.631720 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:47.631702 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-s8jwx" Oct 30 16:24:47.648168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:47.648141 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-xp4ww\"" Oct 30 16:24:47.661409 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:47.661388 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 16:24:47.661518 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:47.661433 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 16:24:47.661925 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:47.661909 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-s8jwx"] Oct 30 16:24:47.718840 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:47.718818 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4fb4318-9331-4344-a4dd-cd307e19f376-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-s8jwx\" (UID: \"e4fb4318-9331-4344-a4dd-cd307e19f376\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-s8jwx" Oct 30 16:24:47.718944 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:47.718849 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7lhk\" (UniqueName: \"kubernetes.io/projected/e4fb4318-9331-4344-a4dd-cd307e19f376-kube-api-access-f7lhk\") pod \"cert-manager-cainjector-5c57bbd4fc-s8jwx\" (UID: \"e4fb4318-9331-4344-a4dd-cd307e19f376\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-s8jwx" Oct 30 16:24:47.820186 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:47.820160 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4fb4318-9331-4344-a4dd-cd307e19f376-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-s8jwx\" (UID: \"e4fb4318-9331-4344-a4dd-cd307e19f376\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-s8jwx" Oct 30 16:24:47.820317 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:47.820192 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7lhk\" (UniqueName: \"kubernetes.io/projected/e4fb4318-9331-4344-a4dd-cd307e19f376-kube-api-access-f7lhk\") pod \"cert-manager-cainjector-5c57bbd4fc-s8jwx\" (UID: \"e4fb4318-9331-4344-a4dd-cd307e19f376\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-s8jwx" Oct 30 16:24:47.845431 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:47.845407 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4fb4318-9331-4344-a4dd-cd307e19f376-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-s8jwx\" (UID: \"e4fb4318-9331-4344-a4dd-cd307e19f376\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-s8jwx" Oct 30 16:24:47.845535 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:47.845467 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7lhk\" (UniqueName: \"kubernetes.io/projected/e4fb4318-9331-4344-a4dd-cd307e19f376-kube-api-access-f7lhk\") pod \"cert-manager-cainjector-5c57bbd4fc-s8jwx\" (UID: \"e4fb4318-9331-4344-a4dd-cd307e19f376\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-s8jwx" Oct 30 16:24:47.954733 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:47.954673 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-s8jwx" Oct 30 16:24:48.241956 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:48.241932 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-s8jwx"] Oct 30 16:24:48.243477 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:24:48.243451 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4fb4318_9331_4344_a4dd_cd307e19f376.slice/crio-ac202d8eaa52583f3b2422c66cbcce2c72b69cd7edce972fb1eeb69f09db6910 WatchSource:0}: Error finding container ac202d8eaa52583f3b2422c66cbcce2c72b69cd7edce972fb1eeb69f09db6910: Status 404 returned error can't find the container with id ac202d8eaa52583f3b2422c66cbcce2c72b69cd7edce972fb1eeb69f09db6910 Oct 30 16:24:48.245457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:48.245438 6202 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:24:49.154226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:49.154182 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-s8jwx" event={"ID":"e4fb4318-9331-4344-a4dd-cd307e19f376","Type":"ContainerStarted","Data":"ac202d8eaa52583f3b2422c66cbcce2c72b69cd7edce972fb1eeb69f09db6910"} Oct 30 16:24:50.524196 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:50.524156 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:52.167089 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:52.167013 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-s8jwx" event={"ID":"e4fb4318-9331-4344-a4dd-cd307e19f376","Type":"ContainerStarted","Data":"aecbfe8e388fd064d14a9e56619c378966d095035919cf972bdedd24ede37ded"} Oct 30 16:24:52.193049 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:52.193018 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:52.209957 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:52.209919 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-s8jwx" podStartSLOduration=2.207832964 podStartE2EDuration="5.209907527s" totalImagesPullingTime="3.002074563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:24:52.209873391 +0000 UTC m=+205.920328135" watchObservedRunningTime="2026-10-30 16:24:52.209907527 +0000 UTC m=+205.920362270" Oct 30 16:24:52.308510 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:52.308475 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:52.628426 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:52.628394 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:52.747665 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:52.747630 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:58.946638 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:58.946589 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:59.061578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:24:59.061541 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:00.519719 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:00.519681 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:08.130030 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:08.129991 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:08.858906 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:08.858868 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:14.066459 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:25:14.066422 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:25:14.934818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:14.934784 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:15.047997 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.047962 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk"] Oct 30 16:25:15.050015 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.049998 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" Oct 30 16:25:15.067088 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.067066 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:15.067456 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.067440 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fmsm8\"" Oct 30 16:25:15.080636 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.080608 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 16:25:15.080881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.080864 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:25:15.080964 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.080920 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 16:25:15.087097 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.087078 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk"] Oct 30 16:25:15.131698 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.131671 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4acfbd91-1540-4ea5-8d50-0f3ca16f0e58-cert\") pod \"lws-controller-manager-6584f6464-9p2gk\" (UID: \"4acfbd91-1540-4ea5-8d50-0f3ca16f0e58\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" Oct 30 16:25:15.131842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.131716 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdv6r\" (UniqueName: \"kubernetes.io/projected/4acfbd91-1540-4ea5-8d50-0f3ca16f0e58-kube-api-access-bdv6r\") pod \"lws-controller-manager-6584f6464-9p2gk\" (UID: \"4acfbd91-1540-4ea5-8d50-0f3ca16f0e58\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" Oct 30 16:25:15.131918 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.131850 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4acfbd91-1540-4ea5-8d50-0f3ca16f0e58-manager-config\") pod \"lws-controller-manager-6584f6464-9p2gk\" (UID: \"4acfbd91-1540-4ea5-8d50-0f3ca16f0e58\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" Oct 30 16:25:15.131918 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.131906 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4acfbd91-1540-4ea5-8d50-0f3ca16f0e58-metrics-cert\") pod \"lws-controller-manager-6584f6464-9p2gk\" (UID: \"4acfbd91-1540-4ea5-8d50-0f3ca16f0e58\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" Oct 30 16:25:15.167316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.167294 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 16:25:15.171404 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.171391 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 16:25:15.203179 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.203116 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:15.233301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.233274 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdv6r\" (UniqueName: \"kubernetes.io/projected/4acfbd91-1540-4ea5-8d50-0f3ca16f0e58-kube-api-access-bdv6r\") pod \"lws-controller-manager-6584f6464-9p2gk\" (UID: \"4acfbd91-1540-4ea5-8d50-0f3ca16f0e58\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" Oct 30 16:25:15.233379 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.233340 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4acfbd91-1540-4ea5-8d50-0f3ca16f0e58-manager-config\") pod \"lws-controller-manager-6584f6464-9p2gk\" (UID: \"4acfbd91-1540-4ea5-8d50-0f3ca16f0e58\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" Oct 30 16:25:15.233440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.233379 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4acfbd91-1540-4ea5-8d50-0f3ca16f0e58-metrics-cert\") pod \"lws-controller-manager-6584f6464-9p2gk\" (UID: \"4acfbd91-1540-4ea5-8d50-0f3ca16f0e58\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" Oct 30 16:25:15.233440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.233431 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4acfbd91-1540-4ea5-8d50-0f3ca16f0e58-cert\") pod \"lws-controller-manager-6584f6464-9p2gk\" (UID: \"4acfbd91-1540-4ea5-8d50-0f3ca16f0e58\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" Oct 30 16:25:15.234060 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.234025 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4acfbd91-1540-4ea5-8d50-0f3ca16f0e58-manager-config\") pod \"lws-controller-manager-6584f6464-9p2gk\" (UID: \"4acfbd91-1540-4ea5-8d50-0f3ca16f0e58\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" Oct 30 16:25:15.236018 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.235999 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4acfbd91-1540-4ea5-8d50-0f3ca16f0e58-cert\") pod \"lws-controller-manager-6584f6464-9p2gk\" (UID: \"4acfbd91-1540-4ea5-8d50-0f3ca16f0e58\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" Oct 30 16:25:15.236125 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.236108 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4acfbd91-1540-4ea5-8d50-0f3ca16f0e58-metrics-cert\") pod \"lws-controller-manager-6584f6464-9p2gk\" (UID: \"4acfbd91-1540-4ea5-8d50-0f3ca16f0e58\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" Oct 30 16:25:15.257972 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.257952 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdv6r\" (UniqueName: \"kubernetes.io/projected/4acfbd91-1540-4ea5-8d50-0f3ca16f0e58-kube-api-access-bdv6r\") pod \"lws-controller-manager-6584f6464-9p2gk\" (UID: \"4acfbd91-1540-4ea5-8d50-0f3ca16f0e58\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" Oct 30 16:25:15.359890 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.359855 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" Oct 30 16:25:15.650628 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:15.650598 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk"] Oct 30 16:25:15.651380 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:25:15.651355 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4acfbd91_1540_4ea5_8d50_0f3ca16f0e58.slice/crio-f92c1bd3d19422437733ddc8c25731889ff4fef792b1dc2cff5cfcda9a9fff4c WatchSource:0}: Error finding container f92c1bd3d19422437733ddc8c25731889ff4fef792b1dc2cff5cfcda9a9fff4c: Status 404 returned error can't find the container with id f92c1bd3d19422437733ddc8c25731889ff4fef792b1dc2cff5cfcda9a9fff4c Oct 30 16:25:16.247559 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:16.247524 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" event={"ID":"4acfbd91-1540-4ea5-8d50-0f3ca16f0e58","Type":"ContainerStarted","Data":"f92c1bd3d19422437733ddc8c25731889ff4fef792b1dc2cff5cfcda9a9fff4c"} Oct 30 16:25:18.256296 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:18.256199 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" event={"ID":"4acfbd91-1540-4ea5-8d50-0f3ca16f0e58","Type":"ContainerStarted","Data":"708245c457d9d2a74e10ef1ad5bd0e568ad543301a81b7defbf90291110fd994"} Oct 30 16:25:18.256665 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:18.256328 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" Oct 30 16:25:18.299639 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:18.299596 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" podStartSLOduration=0.984421418 podStartE2EDuration="3.299585239s" totalImagesPullingTime="2.315163821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:25:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:25:18.298965635 +0000 UTC m=+232.009420380" watchObservedRunningTime="2026-10-30 16:25:18.299585239 +0000 UTC m=+232.010039982" Oct 30 16:25:24.192298 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:24.192255 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:24.306746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:24.306709 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:24.791174 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:24.791144 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:24.927647 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:24.927612 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:24.993595 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:24.993568 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw"] Oct 30 16:25:24.997041 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:24.997027 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.012610 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.012589 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw"] Oct 30 16:25:25.018186 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.018165 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6e0ab79-e56f-453b-8159-97d62ccbeb59-tmp\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xbsxw\" (UID: \"b6e0ab79-e56f-453b-8159-97d62ccbeb59\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.018294 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.018202 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrmvm\" (UniqueName: \"kubernetes.io/projected/b6e0ab79-e56f-453b-8159-97d62ccbeb59-kube-api-access-mrmvm\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xbsxw\" (UID: \"b6e0ab79-e56f-453b-8159-97d62ccbeb59\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.018294 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.018221 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6e0ab79-e56f-453b-8159-97d62ccbeb59-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xbsxw\" (UID: \"b6e0ab79-e56f-453b-8159-97d62ccbeb59\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.018294 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.018241 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6e0ab79-e56f-453b-8159-97d62ccbeb59-webhook-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xbsxw\" (UID: \"b6e0ab79-e56f-453b-8159-97d62ccbeb59\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.018410 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.018356 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6e0ab79-e56f-453b-8159-97d62ccbeb59-manifests\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xbsxw\" (UID: \"b6e0ab79-e56f-453b-8159-97d62ccbeb59\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.018410 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.018378 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6e0ab79-e56f-453b-8159-97d62ccbeb59-metrics-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xbsxw\" (UID: \"b6e0ab79-e56f-453b-8159-97d62ccbeb59\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.071068 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.071011 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 16:25:25.071179 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.071074 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 16:25:25.071216 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.071187 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 16:25:25.071259 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.071227 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wc8qt\"" Oct 30 16:25:25.072240 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.072225 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 16:25:25.072326 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.072265 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 16:25:25.118807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.118750 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrmvm\" (UniqueName: \"kubernetes.io/projected/b6e0ab79-e56f-453b-8159-97d62ccbeb59-kube-api-access-mrmvm\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xbsxw\" (UID: \"b6e0ab79-e56f-453b-8159-97d62ccbeb59\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.118881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.118825 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6e0ab79-e56f-453b-8159-97d62ccbeb59-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xbsxw\" (UID: \"b6e0ab79-e56f-453b-8159-97d62ccbeb59\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.118881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.118858 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6e0ab79-e56f-453b-8159-97d62ccbeb59-webhook-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xbsxw\" (UID: \"b6e0ab79-e56f-453b-8159-97d62ccbeb59\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.118970 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.118915 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6e0ab79-e56f-453b-8159-97d62ccbeb59-manifests\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xbsxw\" (UID: \"b6e0ab79-e56f-453b-8159-97d62ccbeb59\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.119089 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.119062 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6e0ab79-e56f-453b-8159-97d62ccbeb59-metrics-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xbsxw\" (UID: \"b6e0ab79-e56f-453b-8159-97d62ccbeb59\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.119218 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.119144 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6e0ab79-e56f-453b-8159-97d62ccbeb59-tmp\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xbsxw\" (UID: \"b6e0ab79-e56f-453b-8159-97d62ccbeb59\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.119283 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.119258 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6e0ab79-e56f-453b-8159-97d62ccbeb59-manifests\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xbsxw\" (UID: \"b6e0ab79-e56f-453b-8159-97d62ccbeb59\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.120189 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.120160 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:25.121380 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.121359 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6e0ab79-e56f-453b-8159-97d62ccbeb59-tmp\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xbsxw\" (UID: \"b6e0ab79-e56f-453b-8159-97d62ccbeb59\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.121700 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.121682 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6e0ab79-e56f-453b-8159-97d62ccbeb59-metrics-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xbsxw\" (UID: \"b6e0ab79-e56f-453b-8159-97d62ccbeb59\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.121791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.121678 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6e0ab79-e56f-453b-8159-97d62ccbeb59-webhook-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xbsxw\" (UID: \"b6e0ab79-e56f-453b-8159-97d62ccbeb59\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.121791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.121745 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6e0ab79-e56f-453b-8159-97d62ccbeb59-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xbsxw\" (UID: \"b6e0ab79-e56f-453b-8159-97d62ccbeb59\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.143874 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.143840 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrmvm\" (UniqueName: \"kubernetes.io/projected/b6e0ab79-e56f-453b-8159-97d62ccbeb59-kube-api-access-mrmvm\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xbsxw\" (UID: \"b6e0ab79-e56f-453b-8159-97d62ccbeb59\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.290097 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.290064 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:25.311133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.311092 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:25.420124 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.420099 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:25.656575 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:25.656548 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw"] Oct 30 16:25:25.657396 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:25:25.657365 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e0ab79_e56f_453b_8159_97d62ccbeb59.slice/crio-fae1be4a54a3d2acc6e153e119d7b6e4f2305ecd278ee67c5007cc55301f39fd WatchSource:0}: Error finding container fae1be4a54a3d2acc6e153e119d7b6e4f2305ecd278ee67c5007cc55301f39fd: Status 404 returned error can't find the container with id fae1be4a54a3d2acc6e153e119d7b6e4f2305ecd278ee67c5007cc55301f39fd Oct 30 16:25:26.286486 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:26.286453 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" event={"ID":"b6e0ab79-e56f-453b-8159-97d62ccbeb59","Type":"ContainerStarted","Data":"fae1be4a54a3d2acc6e153e119d7b6e4f2305ecd278ee67c5007cc55301f39fd"} Oct 30 16:25:29.263054 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:29.262976 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6584f6464-9p2gk" Oct 30 16:25:29.306210 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:29.306178 6202 generic.go:356] "Generic (PLEG): container finished" podID="b6e0ab79-e56f-453b-8159-97d62ccbeb59" containerID="cece66d94bea6c9e28a934248bbdbf9fe4273400188f872d08cd2db07dd13521" exitCode=0 Oct 30 16:25:29.306364 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:29.306261 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" event={"ID":"b6e0ab79-e56f-453b-8159-97d62ccbeb59","Type":"ContainerDied","Data":"cece66d94bea6c9e28a934248bbdbf9fe4273400188f872d08cd2db07dd13521"} Oct 30 16:25:30.312151 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:30.312111 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" event={"ID":"b6e0ab79-e56f-453b-8159-97d62ccbeb59","Type":"ContainerStarted","Data":"4a40088b23210d56f3011ca24d419683621932feeedf52be5cc4016600a6aeae"} Oct 30 16:25:30.312726 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:30.312223 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:30.404925 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:30.404871 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" podStartSLOduration=2.723171712 podStartE2EDuration="6.404855776s" totalImagesPullingTime="3.242451316s" totalInitContainerRuntime="439.232748ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:25:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:25:30.390743821 +0000 UTC m=+244.101198565" watchObservedRunningTime="2026-10-30 16:25:30.404855776 +0000 UTC m=+244.115310517" Oct 30 16:25:35.188921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:35.188884 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:41.317888 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:41.317860 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xbsxw" Oct 30 16:25:44.065581 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:25:44.065540 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:25:44.753605 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:44.753569 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:44.884554 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:44.884521 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:45.247710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.247676 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54d485b749-54hv6"] Oct 30 16:25:45.252434 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.252409 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:45.255274 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.255258 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54d485b749-54hv6" Oct 30 16:25:45.278980 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.278932 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 16:25:45.279652 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.279633 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54d485b749-54hv6"] Oct 30 16:25:45.290136 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.290110 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 16:25:45.362674 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.362650 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 16:25:45.362674 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.362652 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 16:25:45.362674 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.362652 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7ptt6\"" Oct 30 16:25:45.372038 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.372011 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:45.377065 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.377048 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/785b392b-2225-4f7c-97dd-74ba17073238-tls-certs\") pod \"kube-auth-proxy-54d485b749-54hv6\" (UID: \"785b392b-2225-4f7c-97dd-74ba17073238\") " pod="openshift-ingress/kube-auth-proxy-54d485b749-54hv6" Oct 30 16:25:45.377132 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.377091 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvl48\" (UniqueName: \"kubernetes.io/projected/785b392b-2225-4f7c-97dd-74ba17073238-kube-api-access-nvl48\") pod \"kube-auth-proxy-54d485b749-54hv6\" (UID: \"785b392b-2225-4f7c-97dd-74ba17073238\") " pod="openshift-ingress/kube-auth-proxy-54d485b749-54hv6" Oct 30 16:25:45.377132 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.377115 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/785b392b-2225-4f7c-97dd-74ba17073238-tmp\") pod \"kube-auth-proxy-54d485b749-54hv6\" (UID: \"785b392b-2225-4f7c-97dd-74ba17073238\") " pod="openshift-ingress/kube-auth-proxy-54d485b749-54hv6" Oct 30 16:25:45.477617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.477586 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/785b392b-2225-4f7c-97dd-74ba17073238-tls-certs\") pod \"kube-auth-proxy-54d485b749-54hv6\" (UID: \"785b392b-2225-4f7c-97dd-74ba17073238\") " pod="openshift-ingress/kube-auth-proxy-54d485b749-54hv6" Oct 30 16:25:45.477820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.477629 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvl48\" (UniqueName: \"kubernetes.io/projected/785b392b-2225-4f7c-97dd-74ba17073238-kube-api-access-nvl48\") pod \"kube-auth-proxy-54d485b749-54hv6\" (UID: \"785b392b-2225-4f7c-97dd-74ba17073238\") " pod="openshift-ingress/kube-auth-proxy-54d485b749-54hv6" Oct 30 16:25:45.477820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.477660 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/785b392b-2225-4f7c-97dd-74ba17073238-tmp\") pod \"kube-auth-proxy-54d485b749-54hv6\" (UID: \"785b392b-2225-4f7c-97dd-74ba17073238\") " pod="openshift-ingress/kube-auth-proxy-54d485b749-54hv6" Oct 30 16:25:45.480000 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.479965 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/785b392b-2225-4f7c-97dd-74ba17073238-tmp\") pod \"kube-auth-proxy-54d485b749-54hv6\" (UID: \"785b392b-2225-4f7c-97dd-74ba17073238\") " pod="openshift-ingress/kube-auth-proxy-54d485b749-54hv6" Oct 30 16:25:45.480139 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.480121 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/785b392b-2225-4f7c-97dd-74ba17073238-tls-certs\") pod \"kube-auth-proxy-54d485b749-54hv6\" (UID: \"785b392b-2225-4f7c-97dd-74ba17073238\") " pod="openshift-ingress/kube-auth-proxy-54d485b749-54hv6" Oct 30 16:25:45.502577 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.502522 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvl48\" (UniqueName: \"kubernetes.io/projected/785b392b-2225-4f7c-97dd-74ba17073238-kube-api-access-nvl48\") pod \"kube-auth-proxy-54d485b749-54hv6\" (UID: \"785b392b-2225-4f7c-97dd-74ba17073238\") " pod="openshift-ingress/kube-auth-proxy-54d485b749-54hv6" Oct 30 16:25:45.566602 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.566576 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54d485b749-54hv6" Oct 30 16:25:45.855014 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:45.854986 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54d485b749-54hv6"] Oct 30 16:25:45.856383 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:25:45.856358 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785b392b_2225_4f7c_97dd_74ba17073238.slice/crio-4c3c7d6ca68ae374a37e5bb48db504cfc3f099cf5f649f4f9e1efd14734ca4fe WatchSource:0}: Error finding container 4c3c7d6ca68ae374a37e5bb48db504cfc3f099cf5f649f4f9e1efd14734ca4fe: Status 404 returned error can't find the container with id 4c3c7d6ca68ae374a37e5bb48db504cfc3f099cf5f649f4f9e1efd14734ca4fe Oct 30 16:25:46.372213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:46.372024 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54d485b749-54hv6" event={"ID":"785b392b-2225-4f7c-97dd-74ba17073238","Type":"ContainerStarted","Data":"4c3c7d6ca68ae374a37e5bb48db504cfc3f099cf5f649f4f9e1efd14734ca4fe"} Oct 30 16:25:46.918684 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:46.918649 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:47.043422 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:47.043387 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:48.057270 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:48.057243 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:48.215341 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:48.215303 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:48.373019 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:48.372987 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:48.381178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:48.381152 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54d485b749-54hv6" event={"ID":"785b392b-2225-4f7c-97dd-74ba17073238","Type":"ContainerStarted","Data":"9cfa04d401e71265808a27086509335ddf1e6145210d1f86c07547155ecb23fc"} Oct 30 16:25:48.424284 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:48.424244 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54d485b749-54hv6" podStartSLOduration=1.2564792040000001 podStartE2EDuration="3.424232785s" totalImagesPullingTime="2.167753581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:25:48.423366009 +0000 UTC m=+262.133820752" watchObservedRunningTime="2026-10-30 16:25:48.424232785 +0000 UTC m=+262.134687529" Oct 30 16:25:48.832972 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:48.832936 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:50.666254 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:50.666219 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:50.822348 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:50.822310 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:51.618188 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:51.618153 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:51.744103 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:51.744067 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:52.032659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:52.032627 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:52.150103 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:25:52.150074 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:00.919882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:26:00.919845 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:01.092030 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:26:01.091987 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:07.263571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:26:07.263489 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:07.855056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:26:07.855009 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:14.066136 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:26:14.066094 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:26:35.327055 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:26:35.327013 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:44.066239 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:26:44.066196 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:26:47.664132 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:26:47.664100 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:47.819901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:26:47.819871 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:47.955021 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:26:47.954953 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:48.159165 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:26:48.159130 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:48.283559 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:26:48.283482 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:48.417126 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:26:48.417093 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:50.249534 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:26:50.249504 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:50.370729 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:26:50.370699 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:50.489891 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:26:50.489856 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:50.605480 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:26:50.605453 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:11.118378 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:27:11.118343 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:12.449090 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:27:12.449055 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:14.067035 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:27:14.067000 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:27:35.435058 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:27:35.434985 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:44.067137 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:27:44.067098 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:27:46.460271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:27:46.460235 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:47.156586 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:27:47.156551 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:48.385381 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:27:48.385349 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:48.629066 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:27:48.629034 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:58.514969 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:27:58.514933 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:59.300842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:27:59.300806 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:09.575782 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:09.575734 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:10.366958 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:10.366922 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:11.162220 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:11.162188 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:14.065893 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:28:14.065859 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:28:14.383281 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:14.383249 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:14.512792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:14.512746 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:14.781213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:14.781138 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:16.786462 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:16.786422 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:16.908184 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:16.908150 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:18.074571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:18.074523 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:18.359321 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:18.359219 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-nzbcq"] Oct 30 16:28:18.362521 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:18.362503 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nzbcq" Oct 30 16:28:18.381043 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:18.381020 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 16:28:18.392048 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:18.392029 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 16:28:18.392171 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:18.392087 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-rtk88\"" Oct 30 16:28:18.393562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:18.393544 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-nzbcq"] Oct 30 16:28:18.404932 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:18.404905 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzv8j\" (UniqueName: \"kubernetes.io/projected/34745160-9b04-4309-bb7b-87cb8f4aca06-kube-api-access-qzv8j\") pod \"dns-operator-controller-manager-65bfd79598-nzbcq\" (UID: \"34745160-9b04-4309-bb7b-87cb8f4aca06\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nzbcq" Oct 30 16:28:18.480527 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:18.480509 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 16:28:18.506010 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:18.505989 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzv8j\" (UniqueName: \"kubernetes.io/projected/34745160-9b04-4309-bb7b-87cb8f4aca06-kube-api-access-qzv8j\") pod \"dns-operator-controller-manager-65bfd79598-nzbcq\" (UID: \"34745160-9b04-4309-bb7b-87cb8f4aca06\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nzbcq" Oct 30 16:28:18.532483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:18.532462 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzv8j\" (UniqueName: \"kubernetes.io/projected/34745160-9b04-4309-bb7b-87cb8f4aca06-kube-api-access-qzv8j\") pod \"dns-operator-controller-manager-65bfd79598-nzbcq\" (UID: \"34745160-9b04-4309-bb7b-87cb8f4aca06\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nzbcq" Oct 30 16:28:18.675333 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:18.675253 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nzbcq" Oct 30 16:28:18.792843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:18.792808 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:18.965931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:18.965908 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-nzbcq"] Oct 30 16:28:18.968441 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:28:18.968202 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34745160_9b04_4309_bb7b_87cb8f4aca06.slice/crio-4e25333cc36e3d4b9526242889a40a5b15f7b5218cfa692113dea59473437352 WatchSource:0}: Error finding container 4e25333cc36e3d4b9526242889a40a5b15f7b5218cfa692113dea59473437352: Status 404 returned error can't find the container with id 4e25333cc36e3d4b9526242889a40a5b15f7b5218cfa692113dea59473437352 Oct 30 16:28:19.162448 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:19.162414 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:19.917743 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:19.917698 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nzbcq" event={"ID":"34745160-9b04-4309-bb7b-87cb8f4aca06","Type":"ContainerStarted","Data":"4e25333cc36e3d4b9526242889a40a5b15f7b5218cfa692113dea59473437352"} Oct 30 16:28:20.232145 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:20.232056 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:21.274159 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:21.274128 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-nsfnw"] Oct 30 16:28:21.278112 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:21.278093 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-nsfnw" Oct 30 16:28:21.293621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:21.293598 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-djl7b\"" Oct 30 16:28:21.306015 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:21.305998 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-nsfnw"] Oct 30 16:28:21.329305 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:21.329274 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8cj8\" (UniqueName: \"kubernetes.io/projected/118b3526-1a2c-42dc-9061-310ccd01d554-kube-api-access-l8cj8\") pod \"authorino-operator-68d6dd76dc-nsfnw\" (UID: \"118b3526-1a2c-42dc-9061-310ccd01d554\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-nsfnw" Oct 30 16:28:21.430892 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:21.430861 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8cj8\" (UniqueName: \"kubernetes.io/projected/118b3526-1a2c-42dc-9061-310ccd01d554-kube-api-access-l8cj8\") pod \"authorino-operator-68d6dd76dc-nsfnw\" (UID: \"118b3526-1a2c-42dc-9061-310ccd01d554\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-nsfnw" Oct 30 16:28:21.462625 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:21.462593 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8cj8\" (UniqueName: \"kubernetes.io/projected/118b3526-1a2c-42dc-9061-310ccd01d554-kube-api-access-l8cj8\") pod \"authorino-operator-68d6dd76dc-nsfnw\" (UID: \"118b3526-1a2c-42dc-9061-310ccd01d554\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-nsfnw" Oct 30 16:28:21.590705 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:21.590671 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-nsfnw" Oct 30 16:28:21.890957 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:21.890927 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-nsfnw"] Oct 30 16:28:21.893498 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:28:21.893464 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118b3526_1a2c_42dc_9061_310ccd01d554.slice/crio-595e70b1ec317dc1a277f53c24f7d4088e4b8848eea992bebe90141817228226 WatchSource:0}: Error finding container 595e70b1ec317dc1a277f53c24f7d4088e4b8848eea992bebe90141817228226: Status 404 returned error can't find the container with id 595e70b1ec317dc1a277f53c24f7d4088e4b8848eea992bebe90141817228226 Oct 30 16:28:21.926976 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:21.926943 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-nsfnw" event={"ID":"118b3526-1a2c-42dc-9061-310ccd01d554","Type":"ContainerStarted","Data":"595e70b1ec317dc1a277f53c24f7d4088e4b8848eea992bebe90141817228226"} Oct 30 16:28:21.928547 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:21.928522 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nzbcq" event={"ID":"34745160-9b04-4309-bb7b-87cb8f4aca06","Type":"ContainerStarted","Data":"65c5c7392dbaf2565af12961da59729c523bf9ff1362abd43b2106b44e45797e"} Oct 30 16:28:21.928655 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:21.928639 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nzbcq" Oct 30 16:28:21.971446 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:21.971403 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nzbcq" podStartSLOduration=1.639361757 podStartE2EDuration="3.971390927s" totalImagesPullingTime="2.33202917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:21.971058667 +0000 UTC m=+415.681513412" watchObservedRunningTime="2026-10-30 16:28:21.971390927 +0000 UTC m=+415.681845682" Oct 30 16:28:22.164597 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:22.164524 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:22.288916 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:22.288886 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:23.940173 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:23.940136 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-nsfnw" event={"ID":"118b3526-1a2c-42dc-9061-310ccd01d554","Type":"ContainerStarted","Data":"b223b2988bdbe496e2af562dc9f286fad107e1b51a704fc770a70c161a6c3b46"} Oct 30 16:28:23.940576 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:23.940271 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-nsfnw" Oct 30 16:28:23.982473 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:23.982419 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-nsfnw" podStartSLOduration=1.689349783 podStartE2EDuration="2.982408612s" totalImagesPullingTime="1.293058829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:23.98197193 +0000 UTC m=+417.692426671" watchObservedRunningTime="2026-10-30 16:28:23.982408612 +0000 UTC m=+417.692863356" Oct 30 16:28:24.780590 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:24.780556 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:24.895347 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:24.895316 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:25.172480 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:25.172454 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:25.289362 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:25.289327 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:25.568985 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:25.568909 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:25.721038 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:25.721010 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:31.857285 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:31.857253 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:31.978218 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:31.978182 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:32.937111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:32.937075 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-nzbcq" Oct 30 16:28:34.290216 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:34.290181 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:34.417548 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:34.417517 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:34.946500 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:34.946469 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-nsfnw" Oct 30 16:28:35.635844 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:35.635810 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:40.810865 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:40.810833 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:40.923432 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:40.923396 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:44.066391 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:28:44.066342 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:28:46.007108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:46.007073 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:46.123831 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:46.123803 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:46.427975 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:46.427943 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:46.606992 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:46.606961 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:46.753515 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:46.753413 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:49.357705 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:49.357673 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq"] Oct 30 16:28:49.360793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:49.360754 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq" Oct 30 16:28:49.387336 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:49.387313 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gwddn\"" Oct 30 16:28:49.388243 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:49.388223 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq"] Oct 30 16:28:49.455324 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:49.455291 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrwgk\" (UniqueName: \"kubernetes.io/projected/07e83efa-0d59-4133-8588-133fe009c226-kube-api-access-zrwgk\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qh4lq\" (UID: \"07e83efa-0d59-4133-8588-133fe009c226\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq" Oct 30 16:28:49.455484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:49.455355 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07e83efa-0d59-4133-8588-133fe009c226-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qh4lq\" (UID: \"07e83efa-0d59-4133-8588-133fe009c226\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq" Oct 30 16:28:49.556685 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:49.556649 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrwgk\" (UniqueName: \"kubernetes.io/projected/07e83efa-0d59-4133-8588-133fe009c226-kube-api-access-zrwgk\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qh4lq\" (UID: \"07e83efa-0d59-4133-8588-133fe009c226\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq" Oct 30 16:28:49.556849 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:49.556727 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07e83efa-0d59-4133-8588-133fe009c226-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qh4lq\" (UID: \"07e83efa-0d59-4133-8588-133fe009c226\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq" Oct 30 16:28:49.557108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:49.557089 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07e83efa-0d59-4133-8588-133fe009c226-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qh4lq\" (UID: \"07e83efa-0d59-4133-8588-133fe009c226\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq" Oct 30 16:28:49.592451 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:49.592429 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrwgk\" (UniqueName: \"kubernetes.io/projected/07e83efa-0d59-4133-8588-133fe009c226-kube-api-access-zrwgk\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qh4lq\" (UID: \"07e83efa-0d59-4133-8588-133fe009c226\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq" Oct 30 16:28:49.673357 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:49.673297 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq" Oct 30 16:28:49.990049 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:49.990019 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq"] Oct 30 16:28:49.993021 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:28:49.992994 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e83efa_0d59_4133_8588_133fe009c226.slice/crio-eb8abb116c953db112b49fc2f35b1f75f9b00eed97410c34a3af20f3fe5e989f WatchSource:0}: Error finding container eb8abb116c953db112b49fc2f35b1f75f9b00eed97410c34a3af20f3fe5e989f: Status 404 returned error can't find the container with id eb8abb116c953db112b49fc2f35b1f75f9b00eed97410c34a3af20f3fe5e989f Oct 30 16:28:50.038816 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:50.038787 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq" event={"ID":"07e83efa-0d59-4133-8588-133fe009c226","Type":"ContainerStarted","Data":"eb8abb116c953db112b49fc2f35b1f75f9b00eed97410c34a3af20f3fe5e989f"} Oct 30 16:28:53.215394 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:53.215347 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:53.337239 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:53.337208 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:54.060000 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:54.059960 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq" event={"ID":"07e83efa-0d59-4133-8588-133fe009c226","Type":"ContainerStarted","Data":"7b9e583263389fb1d89e7b419a2437c0d36cf8e04cc74a6924370766396ffb05"} Oct 30 16:28:54.060198 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:54.060119 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq" Oct 30 16:28:54.128056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:54.128011 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq" podStartSLOduration=1.6284565519999998 podStartE2EDuration="5.12799392s" totalImagesPullingTime="3.499537368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:54.115163213 +0000 UTC m=+447.825617957" watchObservedRunningTime="2026-10-30 16:28:54.12799392 +0000 UTC m=+447.838448668" Oct 30 16:28:55.662931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:55.662896 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:55.786988 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:55.786954 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:57.370624 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:57.370586 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:57.494647 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:28:57.494616 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:02.217592 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:29:02.217558 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:02.337206 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:29:02.337177 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:05.066603 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:29:05.066577 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq" Oct 30 16:29:14.066875 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:29:14.066836 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:29:16.577565 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:29:16.577487 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:16.748656 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:29:16.748624 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:23.727928 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:29:23.727870 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:23.848869 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:29:23.848823 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:23.967947 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:29:23.967909 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:24.661815 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:29:24.661779 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:24.780820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:29:24.780787 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:35.673729 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:29:35.673686 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:38.548133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:29:38.548097 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:41.791493 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:29:41.791460 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:41.910566 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:29:41.910537 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:42.750164 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:29:42.750126 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:44.067561 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:29:44.067527 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:29:54.950391 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:29:54.950349 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:55.355301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:29:55.355260 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:14.067041 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:30:14.067002 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:30:15.349543 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:30:15.349506 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:15.498719 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:30:15.498687 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:15.999349 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:30:15.999309 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:16.144404 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:30:16.144370 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:19.037065 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:30:19.037027 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:19.152874 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:30:19.152839 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:22.616059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:30:22.616029 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:22.794689 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:30:22.794656 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:35.811493 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:30:35.811404 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:44.067027 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:30:44.066988 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:30:55.147885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:30:55.147846 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:55.397125 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:30:55.397093 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:56.463979 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:30:56.463935 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:56.583183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:30:56.583149 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:56.702286 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:30:56.702253 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:56.814508 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:30:56.814436 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:14.002336 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:31:14.002300 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:14.066451 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:31:14.066417 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:31:15.518639 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:31:15.518606 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:32.328122 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:31:32.328086 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:32.443669 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:31:32.443620 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:35.966883 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:31:35.966849 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:36.338753 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:31:36.338726 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:40.780983 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:31:40.780952 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:40.912092 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:31:40.912049 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:44.065719 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:31:44.065678 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:31:59.178712 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:31:59.178675 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:01.109960 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.109925 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8"] Oct 30 16:32:01.114362 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.114337 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.135298 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.135269 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9702f62c-b2ea-47b1-9e67-d50be44e166e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8\" (UID: \"9702f62c-b2ea-47b1-9e67-d50be44e166e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.135455 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.135319 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9702f62c-b2ea-47b1-9e67-d50be44e166e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8\" (UID: \"9702f62c-b2ea-47b1-9e67-d50be44e166e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.135455 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.135387 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x58zc\" (UniqueName: \"kubernetes.io/projected/9702f62c-b2ea-47b1-9e67-d50be44e166e-kube-api-access-x58zc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8\" (UID: \"9702f62c-b2ea-47b1-9e67-d50be44e166e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.135455 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.135446 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9702f62c-b2ea-47b1-9e67-d50be44e166e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8\" (UID: \"9702f62c-b2ea-47b1-9e67-d50be44e166e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.135572 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.135517 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9702f62c-b2ea-47b1-9e67-d50be44e166e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8\" (UID: \"9702f62c-b2ea-47b1-9e67-d50be44e166e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.135572 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.135554 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9702f62c-b2ea-47b1-9e67-d50be44e166e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8\" (UID: \"9702f62c-b2ea-47b1-9e67-d50be44e166e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.144215 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.144168 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 16:32:01.144368 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.144348 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 16:32:01.144452 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.144388 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-w67vc\"" Oct 30 16:32:01.144452 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.144422 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 16:32:01.145918 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.145900 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8"] Oct 30 16:32:01.190575 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.190504 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:01.235951 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.235913 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9702f62c-b2ea-47b1-9e67-d50be44e166e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8\" (UID: \"9702f62c-b2ea-47b1-9e67-d50be44e166e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.236142 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.235999 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9702f62c-b2ea-47b1-9e67-d50be44e166e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8\" (UID: \"9702f62c-b2ea-47b1-9e67-d50be44e166e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.236142 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.236044 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9702f62c-b2ea-47b1-9e67-d50be44e166e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8\" (UID: \"9702f62c-b2ea-47b1-9e67-d50be44e166e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.236266 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.236218 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x58zc\" (UniqueName: \"kubernetes.io/projected/9702f62c-b2ea-47b1-9e67-d50be44e166e-kube-api-access-x58zc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8\" (UID: \"9702f62c-b2ea-47b1-9e67-d50be44e166e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.236266 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.236260 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9702f62c-b2ea-47b1-9e67-d50be44e166e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8\" (UID: \"9702f62c-b2ea-47b1-9e67-d50be44e166e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.236379 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.236299 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9702f62c-b2ea-47b1-9e67-d50be44e166e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8\" (UID: \"9702f62c-b2ea-47b1-9e67-d50be44e166e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.236379 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.236343 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9702f62c-b2ea-47b1-9e67-d50be44e166e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8\" (UID: \"9702f62c-b2ea-47b1-9e67-d50be44e166e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.236485 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.236417 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9702f62c-b2ea-47b1-9e67-d50be44e166e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8\" (UID: \"9702f62c-b2ea-47b1-9e67-d50be44e166e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.236609 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.236594 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9702f62c-b2ea-47b1-9e67-d50be44e166e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8\" (UID: \"9702f62c-b2ea-47b1-9e67-d50be44e166e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.238652 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.238626 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9702f62c-b2ea-47b1-9e67-d50be44e166e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8\" (UID: \"9702f62c-b2ea-47b1-9e67-d50be44e166e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.238804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.238751 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9702f62c-b2ea-47b1-9e67-d50be44e166e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8\" (UID: \"9702f62c-b2ea-47b1-9e67-d50be44e166e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.260581 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.260550 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x58zc\" (UniqueName: \"kubernetes.io/projected/9702f62c-b2ea-47b1-9e67-d50be44e166e-kube-api-access-x58zc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8\" (UID: \"9702f62c-b2ea-47b1-9e67-d50be44e166e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.426755 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.426663 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:01.723154 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.723129 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8"] Oct 30 16:32:01.725566 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:32:01.725539 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9702f62c_b2ea_47b1_9e67_d50be44e166e.slice/crio-a460053d1e4fe8148b3daa23bbbb58a1915affd44c55b8c1eb7d0751452a59c4 WatchSource:0}: Error finding container a460053d1e4fe8148b3daa23bbbb58a1915affd44c55b8c1eb7d0751452a59c4: Status 404 returned error can't find the container with id a460053d1e4fe8148b3daa23bbbb58a1915affd44c55b8c1eb7d0751452a59c4 Oct 30 16:32:01.726904 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.726885 6202 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:32:01.794620 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:01.794580 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" event={"ID":"9702f62c-b2ea-47b1-9e67-d50be44e166e","Type":"ContainerStarted","Data":"a460053d1e4fe8148b3daa23bbbb58a1915affd44c55b8c1eb7d0751452a59c4"} Oct 30 16:32:03.180465 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:03.180421 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:03.813228 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:03.813191 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" event={"ID":"9702f62c-b2ea-47b1-9e67-d50be44e166e","Type":"ContainerStarted","Data":"d6105f8cf2134546757e304c5d406a8fac71ed7d882b9e1de07ab2ccf9edb77c"} Oct 30 16:32:03.854618 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:03.854570 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podStartSLOduration=1.24793989 podStartE2EDuration="2.854550769s" totalImagesPullingTime="1.606610879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:03.854552881 +0000 UTC m=+637.565007628" watchObservedRunningTime="2026-10-30 16:32:03.854550769 +0000 UTC m=+637.565005507" Oct 30 16:32:04.388861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.388826 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m"] Oct 30 16:32:04.392251 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.392226 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.414404 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.414381 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 16:32:04.424086 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.424051 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m"] Oct 30 16:32:04.426747 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.426726 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:04.475049 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.475003 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:04.564156 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.564115 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c7c579a-a559-4424-b23b-cc7b91f61532-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tbc5m\" (UID: \"0c7c579a-a559-4424-b23b-cc7b91f61532\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.564360 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.564206 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c7c579a-a559-4424-b23b-cc7b91f61532-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tbc5m\" (UID: \"0c7c579a-a559-4424-b23b-cc7b91f61532\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.564360 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.564311 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c7c579a-a559-4424-b23b-cc7b91f61532-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tbc5m\" (UID: \"0c7c579a-a559-4424-b23b-cc7b91f61532\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.564360 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.564329 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c7c579a-a559-4424-b23b-cc7b91f61532-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tbc5m\" (UID: \"0c7c579a-a559-4424-b23b-cc7b91f61532\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.564360 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.564345 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdtfv\" (UniqueName: \"kubernetes.io/projected/0c7c579a-a559-4424-b23b-cc7b91f61532-kube-api-access-xdtfv\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tbc5m\" (UID: \"0c7c579a-a559-4424-b23b-cc7b91f61532\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.564573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.564486 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c7c579a-a559-4424-b23b-cc7b91f61532-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tbc5m\" (UID: \"0c7c579a-a559-4424-b23b-cc7b91f61532\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.665361 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.665280 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c7c579a-a559-4424-b23b-cc7b91f61532-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tbc5m\" (UID: \"0c7c579a-a559-4424-b23b-cc7b91f61532\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.665361 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.665310 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c7c579a-a559-4424-b23b-cc7b91f61532-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tbc5m\" (UID: \"0c7c579a-a559-4424-b23b-cc7b91f61532\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.665361 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.665330 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdtfv\" (UniqueName: \"kubernetes.io/projected/0c7c579a-a559-4424-b23b-cc7b91f61532-kube-api-access-xdtfv\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tbc5m\" (UID: \"0c7c579a-a559-4424-b23b-cc7b91f61532\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.665361 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.665361 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c7c579a-a559-4424-b23b-cc7b91f61532-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tbc5m\" (UID: \"0c7c579a-a559-4424-b23b-cc7b91f61532\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.665683 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.665550 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c7c579a-a559-4424-b23b-cc7b91f61532-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tbc5m\" (UID: \"0c7c579a-a559-4424-b23b-cc7b91f61532\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.665683 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.665603 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c7c579a-a559-4424-b23b-cc7b91f61532-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tbc5m\" (UID: \"0c7c579a-a559-4424-b23b-cc7b91f61532\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.665754 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.665738 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c7c579a-a559-4424-b23b-cc7b91f61532-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tbc5m\" (UID: \"0c7c579a-a559-4424-b23b-cc7b91f61532\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.665838 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.665817 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c7c579a-a559-4424-b23b-cc7b91f61532-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tbc5m\" (UID: \"0c7c579a-a559-4424-b23b-cc7b91f61532\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.665974 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.665957 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c7c579a-a559-4424-b23b-cc7b91f61532-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tbc5m\" (UID: \"0c7c579a-a559-4424-b23b-cc7b91f61532\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.667938 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.667906 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c7c579a-a559-4424-b23b-cc7b91f61532-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tbc5m\" (UID: \"0c7c579a-a559-4424-b23b-cc7b91f61532\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.668168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.668154 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c7c579a-a559-4424-b23b-cc7b91f61532-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tbc5m\" (UID: \"0c7c579a-a559-4424-b23b-cc7b91f61532\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.691554 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.691525 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdtfv\" (UniqueName: \"kubernetes.io/projected/0c7c579a-a559-4424-b23b-cc7b91f61532-kube-api-access-xdtfv\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tbc5m\" (UID: \"0c7c579a-a559-4424-b23b-cc7b91f61532\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:04.706404 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:04.706375 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:05.004074 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:05.004050 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m"] Oct 30 16:32:05.006223 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:32:05.006197 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7c579a_a559_4424_b23b_cc7b91f61532.slice/crio-bb14bb5faba3610f2341bd63f8783351b9d0e2f4cbdd9a92e37c8a9307ce8be7 WatchSource:0}: Error finding container bb14bb5faba3610f2341bd63f8783351b9d0e2f4cbdd9a92e37c8a9307ce8be7: Status 404 returned error can't find the container with id bb14bb5faba3610f2341bd63f8783351b9d0e2f4cbdd9a92e37c8a9307ce8be7 Oct 30 16:32:05.823601 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:05.823561 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" event={"ID":"0c7c579a-a559-4424-b23b-cc7b91f61532","Type":"ContainerStarted","Data":"22d3dae4f3d488d9841f150e6c713790df022861c73a7c10e99e5dd9e617322e"} Oct 30 16:32:05.824028 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:05.823610 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" event={"ID":"0c7c579a-a559-4424-b23b-cc7b91f61532","Type":"ContainerStarted","Data":"bb14bb5faba3610f2341bd63f8783351b9d0e2f4cbdd9a92e37c8a9307ce8be7"} Oct 30 16:32:05.866096 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:05.866043 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" podStartSLOduration=1.667144511 podStartE2EDuration="1.866030301s" totalImagesPullingTime="198.88579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:05.865891574 +0000 UTC m=+639.576346317" watchObservedRunningTime="2026-10-30 16:32:05.866030301 +0000 UTC m=+639.576485088" Oct 30 16:32:06.707538 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.707488 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:06.804406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.804372 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8"] Oct 30 16:32:06.807833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.807793 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:06.822802 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.822735 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 16:32:06.835355 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.835321 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8"] Oct 30 16:32:06.883302 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.883257 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4kdj8\" (UID: \"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:06.883302 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.883298 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4kdj8\" (UID: \"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:06.883612 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.883331 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp589\" (UniqueName: \"kubernetes.io/projected/6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4-kube-api-access-tp589\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4kdj8\" (UID: \"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:06.883697 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.883668 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4kdj8\" (UID: \"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:06.883761 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.883738 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4kdj8\" (UID: \"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:06.883856 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.883827 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4kdj8\" (UID: \"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:06.917971 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.917937 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:06.984361 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.984277 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4kdj8\" (UID: \"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:06.984361 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.984312 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4kdj8\" (UID: \"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:06.984591 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.984464 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4kdj8\" (UID: \"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:06.984591 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.984513 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4kdj8\" (UID: \"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:06.984591 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.984546 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4kdj8\" (UID: \"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:06.984591 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.984579 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp589\" (UniqueName: \"kubernetes.io/projected/6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4-kube-api-access-tp589\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4kdj8\" (UID: \"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:06.984810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.984688 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4kdj8\" (UID: \"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:06.984874 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.984859 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4kdj8\" (UID: \"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:06.984960 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.984938 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4kdj8\" (UID: \"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:06.986876 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.986855 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4kdj8\" (UID: \"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:06.987222 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:06.987203 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4kdj8\" (UID: \"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:07.024540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:07.024505 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp589\" (UniqueName: \"kubernetes.io/projected/6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4-kube-api-access-tp589\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4kdj8\" (UID: \"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:07.123553 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:07.123522 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:07.427003 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:07.426975 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8"] Oct 30 16:32:07.429639 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:32:07.429608 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7bd43f_a7fb_449d_bfe3_a07c2aff68a4.slice/crio-4cc9c092ed2136974f4750fd38eb18ef2b00b7f3c70a7e02b30e2f6447a0bbd7 WatchSource:0}: Error finding container 4cc9c092ed2136974f4750fd38eb18ef2b00b7f3c70a7e02b30e2f6447a0bbd7: Status 404 returned error can't find the container with id 4cc9c092ed2136974f4750fd38eb18ef2b00b7f3c70a7e02b30e2f6447a0bbd7 Oct 30 16:32:07.832929 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:07.832890 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" event={"ID":"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4","Type":"ContainerStarted","Data":"7821c5ca973f1f6081c63236271965fca30687590025f10a9faf81b107756e3e"} Oct 30 16:32:07.833164 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:07.832951 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" event={"ID":"6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4","Type":"ContainerStarted","Data":"4cc9c092ed2136974f4750fd38eb18ef2b00b7f3c70a7e02b30e2f6447a0bbd7"} Oct 30 16:32:07.879713 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:07.879646 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" podStartSLOduration=1.6913567760000001 podStartE2EDuration="1.879624603s" totalImagesPullingTime="188.267827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:07.878344524 +0000 UTC m=+641.588799266" watchObservedRunningTime="2026-10-30 16:32:07.879624603 +0000 UTC m=+641.590079349" Oct 30 16:32:08.124102 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:08.124067 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:08.601684 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:08.601651 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:10.321071 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:10.321029 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:11.427837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:11.427801 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:11.441148 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:11.441120 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:11.861580 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:11.861553 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:32:14.067114 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:32:14.067068 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:32:14.707341 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:14.707309 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:14.719706 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:14.719682 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:14.880171 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:14.880138 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tbc5m" Oct 30 16:32:17.123714 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:17.123688 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:17.136337 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:17.136319 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:17.888597 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:17.888557 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4kdj8" Oct 30 16:32:27.253249 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:27.253213 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:29.480646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:29.480609 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:29.522690 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:29.522655 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8"] Oct 30 16:32:29.528068 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:29.528042 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8" Oct 30 16:32:29.542821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:29.542791 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 16:32:29.542966 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:29.542938 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-x95nw\"" Oct 30 16:32:29.554440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:29.554410 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 16:32:29.555662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:29.555635 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8"] Oct 30 16:32:29.588261 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:29.588221 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw9tl\" (UniqueName: \"kubernetes.io/projected/6998c08b-0952-44ac-a708-3c4309686b70-kube-api-access-qw9tl\") pod \"ai-gateway-controller-6cf7bcbfb6-s4pl8\" (UID: \"6998c08b-0952-44ac-a708-3c4309686b70\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8" Oct 30 16:32:29.588423 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:29.588279 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6998c08b-0952-44ac-a708-3c4309686b70-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-s4pl8\" (UID: \"6998c08b-0952-44ac-a708-3c4309686b70\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8" Oct 30 16:32:29.610338 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:29.610308 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:29.689486 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:29.689450 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw9tl\" (UniqueName: \"kubernetes.io/projected/6998c08b-0952-44ac-a708-3c4309686b70-kube-api-access-qw9tl\") pod \"ai-gateway-controller-6cf7bcbfb6-s4pl8\" (UID: \"6998c08b-0952-44ac-a708-3c4309686b70\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8" Oct 30 16:32:29.689486 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:29.689491 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6998c08b-0952-44ac-a708-3c4309686b70-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-s4pl8\" (UID: \"6998c08b-0952-44ac-a708-3c4309686b70\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8" Oct 30 16:32:29.692084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:29.692054 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6998c08b-0952-44ac-a708-3c4309686b70-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-s4pl8\" (UID: \"6998c08b-0952-44ac-a708-3c4309686b70\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8" Oct 30 16:32:29.713005 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:29.712979 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw9tl\" (UniqueName: \"kubernetes.io/projected/6998c08b-0952-44ac-a708-3c4309686b70-kube-api-access-qw9tl\") pod \"ai-gateway-controller-6cf7bcbfb6-s4pl8\" (UID: \"6998c08b-0952-44ac-a708-3c4309686b70\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8" Oct 30 16:32:29.840255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:29.840222 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8" Oct 30 16:32:30.137746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:30.137720 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8"] Oct 30 16:32:30.140912 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:32:30.140877 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6998c08b_0952_44ac_a708_3c4309686b70.slice/crio-4a6224a6fe87428af3398b43a02a612b131c858f5572ef980d72c78eaf8b1826 WatchSource:0}: Error finding container 4a6224a6fe87428af3398b43a02a612b131c858f5572ef980d72c78eaf8b1826: Status 404 returned error can't find the container with id 4a6224a6fe87428af3398b43a02a612b131c858f5572ef980d72c78eaf8b1826 Oct 30 16:32:30.940918 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:30.940884 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8" event={"ID":"6998c08b-0952-44ac-a708-3c4309686b70","Type":"ContainerStarted","Data":"4a6224a6fe87428af3398b43a02a612b131c858f5572ef980d72c78eaf8b1826"} Oct 30 16:32:31.726779 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.726728 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx"] Oct 30 16:32:31.731375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.731350 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.753224 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.753194 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj"] Oct 30 16:32:31.756816 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.756794 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.763829 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.763798 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 16:32:31.763969 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.763798 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nztr5\"" Oct 30 16:32:31.765336 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.765313 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx"] Oct 30 16:32:31.786437 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.786391 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj"] Oct 30 16:32:31.806666 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.806632 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.806666 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.806668 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4591587-01a3-4160-a8b7-d25450be9054-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.806911 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.806685 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2z2l\" (UniqueName: \"kubernetes.io/projected/f4591587-01a3-4160-a8b7-d25450be9054-kube-api-access-v2z2l\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.806911 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.806711 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47902ba0-362e-40c1-9589-c2181308e89a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.806911 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.806860 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.806911 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.806882 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggt4b\" (UniqueName: \"kubernetes.io/projected/47902ba0-362e-40c1-9589-c2181308e89a-kube-api-access-ggt4b\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.807056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.806919 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4591587-01a3-4160-a8b7-d25450be9054-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.807056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.807019 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47902ba0-362e-40c1-9589-c2181308e89a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.807056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.807049 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4591587-01a3-4160-a8b7-d25450be9054-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.807158 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.807078 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4591587-01a3-4160-a8b7-d25450be9054-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.807158 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.807094 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4591587-01a3-4160-a8b7-d25450be9054-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.807235 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.807181 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47902ba0-362e-40c1-9589-c2181308e89a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.807235 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.807213 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.807235 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.807230 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4591587-01a3-4160-a8b7-d25450be9054-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.807333 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.807249 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.807333 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.807299 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4591587-01a3-4160-a8b7-d25450be9054-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.807406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.807349 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.807406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.807367 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4591587-01a3-4160-a8b7-d25450be9054-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.908804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.908754 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.908804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.908804 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4591587-01a3-4160-a8b7-d25450be9054-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.909044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.908830 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.909044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.908850 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4591587-01a3-4160-a8b7-d25450be9054-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.909044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.908868 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.909044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.908946 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4591587-01a3-4160-a8b7-d25450be9054-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.909044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.908991 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.909044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.909015 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4591587-01a3-4160-a8b7-d25450be9054-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.909044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.909042 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2z2l\" (UniqueName: \"kubernetes.io/projected/f4591587-01a3-4160-a8b7-d25450be9054-kube-api-access-v2z2l\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.909457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.909073 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47902ba0-362e-40c1-9589-c2181308e89a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.909457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.909130 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.909457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.909159 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggt4b\" (UniqueName: \"kubernetes.io/projected/47902ba0-362e-40c1-9589-c2181308e89a-kube-api-access-ggt4b\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.909457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.909213 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4591587-01a3-4160-a8b7-d25450be9054-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.909457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.909247 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47902ba0-362e-40c1-9589-c2181308e89a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.909457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.909281 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4591587-01a3-4160-a8b7-d25450be9054-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.909457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.909289 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.909457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.909326 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.909457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.909339 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4591587-01a3-4160-a8b7-d25450be9054-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.909457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.909368 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4591587-01a3-4160-a8b7-d25450be9054-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.910141 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.909632 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4591587-01a3-4160-a8b7-d25450be9054-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.910141 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.909249 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.910141 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.909657 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4591587-01a3-4160-a8b7-d25450be9054-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.910141 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.909790 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4591587-01a3-4160-a8b7-d25450be9054-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.910141 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.909757 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47902ba0-362e-40c1-9589-c2181308e89a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.910141 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.910028 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.910446 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.910405 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4591587-01a3-4160-a8b7-d25450be9054-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.910588 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.910526 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47902ba0-362e-40c1-9589-c2181308e89a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.910863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.910840 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4591587-01a3-4160-a8b7-d25450be9054-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.911846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.911825 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4591587-01a3-4160-a8b7-d25450be9054-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.912078 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.912057 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.912543 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.912518 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47902ba0-362e-40c1-9589-c2181308e89a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.912639 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.912577 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4591587-01a3-4160-a8b7-d25450be9054-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.934375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.934340 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47902ba0-362e-40c1-9589-c2181308e89a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:31.934506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.934448 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2z2l\" (UniqueName: \"kubernetes.io/projected/f4591587-01a3-4160-a8b7-d25450be9054-kube-api-access-v2z2l\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.934789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.934743 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4591587-01a3-4160-a8b7-d25450be9054-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-sm8wx\" (UID: \"f4591587-01a3-4160-a8b7-d25450be9054\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:31.934887 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:31.934871 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggt4b\" (UniqueName: \"kubernetes.io/projected/47902ba0-362e-40c1-9589-c2181308e89a-kube-api-access-ggt4b\") pod \"maas-default-gateway-openshift-default-76f4876667-8cxrj\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:32.047413 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:32.047326 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:32.070987 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:32.070958 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:32.360864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:32.360840 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx"] Oct 30 16:32:32.362830 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:32:32.362802 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4591587_01a3_4160_a8b7_d25450be9054.slice/crio-a9c0db467e1a46f139735684d441ee6d9c874b2b8836280af4bff04aa9758729 WatchSource:0}: Error finding container a9c0db467e1a46f139735684d441ee6d9c874b2b8836280af4bff04aa9758729: Status 404 returned error can't find the container with id a9c0db467e1a46f139735684d441ee6d9c874b2b8836280af4bff04aa9758729 Oct 30 16:32:32.385705 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:32.385682 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj"] Oct 30 16:32:32.387418 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:32:32.387395 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47902ba0_362e_40c1_9589_c2181308e89a.slice/crio-17c520598ac215898a66d414a18bfb597894eb17639d637f869b725468e80975 WatchSource:0}: Error finding container 17c520598ac215898a66d414a18bfb597894eb17639d637f869b725468e80975: Status 404 returned error can't find the container with id 17c520598ac215898a66d414a18bfb597894eb17639d637f869b725468e80975 Oct 30 16:32:32.956376 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:32.956333 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" event={"ID":"f4591587-01a3-4160-a8b7-d25450be9054","Type":"ContainerStarted","Data":"a9c0db467e1a46f139735684d441ee6d9c874b2b8836280af4bff04aa9758729"} Oct 30 16:32:32.961635 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:32.961569 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" event={"ID":"47902ba0-362e-40c1-9589-c2181308e89a","Type":"ContainerStarted","Data":"17c520598ac215898a66d414a18bfb597894eb17639d637f869b725468e80975"} Oct 30 16:32:35.416668 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:35.416629 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:32:35.416969 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:35.416725 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:32:35.416969 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:35.416794 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:32:35.518460 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:35.518427 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:32:35.518545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:35.518502 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:32:35.518602 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:35.518540 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:32:35.985165 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:35.985080 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" event={"ID":"47902ba0-362e-40c1-9589-c2181308e89a","Type":"ContainerStarted","Data":"f21ac59678585f01e3ee373b97fe1cb50511979baa4559d7e400aaa6459eb741"} Oct 30 16:32:35.988286 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:35.987833 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" event={"ID":"f4591587-01a3-4160-a8b7-d25450be9054","Type":"ContainerStarted","Data":"4d25b86c1c982307b38fe1d0cf19b9d9b2474ee74a233d710b632737defa1ad5"} Oct 30 16:32:36.049162 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:36.049078 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:36.051010 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:36.050928 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-sm8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Oct 30 16:32:36.051160 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:36.051041 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" podUID="f4591587-01a3-4160-a8b7-d25450be9054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Oct 30 16:32:36.051821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:36.051795 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:36.057787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:36.056391 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" podStartSLOduration=1.927977437 podStartE2EDuration="5.05637484s" totalImagesPullingTime="3.128397403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:36.042683769 +0000 UTC m=+669.753138513" watchObservedRunningTime="2026-10-30 16:32:36.05637484 +0000 UTC m=+669.766829584" Oct 30 16:32:36.076792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:36.072149 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:36.077323 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:36.077296 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8cxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Oct 30 16:32:36.077419 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:36.077354 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" podUID="47902ba0-362e-40c1-9589-c2181308e89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Oct 30 16:32:37.067810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:37.059201 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-sm8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Oct 30 16:32:37.067810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:37.059301 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" podUID="f4591587-01a3-4160-a8b7-d25450be9054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Oct 30 16:32:37.084227 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:37.084180 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8cxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Oct 30 16:32:37.084422 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:37.084264 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" podUID="47902ba0-362e-40c1-9589-c2181308e89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Oct 30 16:32:38.060793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:38.050157 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-sm8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Oct 30 16:32:38.060793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:38.050219 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" podUID="f4591587-01a3-4160-a8b7-d25450be9054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Oct 30 16:32:38.077811 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:38.071558 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8cxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Oct 30 16:32:38.077811 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:38.071610 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" podUID="47902ba0-362e-40c1-9589-c2181308e89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Oct 30 16:32:39.050566 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:39.050126 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-sm8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Oct 30 16:32:39.050566 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:39.050187 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" podUID="f4591587-01a3-4160-a8b7-d25450be9054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Oct 30 16:32:39.075207 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:39.075098 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8cxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Oct 30 16:32:39.075207 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:39.075154 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" podUID="47902ba0-362e-40c1-9589-c2181308e89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Oct 30 16:32:40.010907 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.010867 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8" event={"ID":"6998c08b-0952-44ac-a708-3c4309686b70","Type":"ContainerStarted","Data":"a2fac3d34c0547edc29a67d5f1968ba23e74daf6e7733a43df38cdd98c9a4c8b"} Oct 30 16:32:40.011346 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.011006 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8" Oct 30 16:32:40.055793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.051148 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-sm8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Oct 30 16:32:40.056065 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.056030 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" podUID="f4591587-01a3-4160-a8b7-d25450be9054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Oct 30 16:32:40.069802 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.066840 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8" podStartSLOduration=2.079678321 podStartE2EDuration="11.066814407s" totalImagesPullingTime="8.987136086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:40.056048965 +0000 UTC m=+673.766503737" watchObservedRunningTime="2026-10-30 16:32:40.066814407 +0000 UTC m=+673.777269182" Oct 30 16:32:40.069802 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.066994 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" podStartSLOduration=6.016171576 podStartE2EDuration="9.066985088s" totalImagesPullingTime="3.050813512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:36.115234757 +0000 UTC m=+669.825689503" watchObservedRunningTime="2026-10-30 16:32:40.066985088 +0000 UTC m=+673.777439833" Oct 30 16:32:40.079792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.076021 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8cxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Oct 30 16:32:40.079792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.076084 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" podUID="47902ba0-362e-40c1-9589-c2181308e89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Oct 30 16:32:40.597380 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.597341 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:40.715335 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.715279 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:40.760792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.749750 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-57rw9"] Oct 30 16:32:40.766840 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.766081 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-57rw9" Oct 30 16:32:40.787863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.786746 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-57rw9"] Oct 30 16:32:40.792368 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.792243 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 16:32:40.792368 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.792326 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-xwsp9\"" Oct 30 16:32:40.831648 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.831623 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a21ffcf-b1ed-4709-934e-f88b301ea0f4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-57rw9\" (UID: \"8a21ffcf-b1ed-4709-934e-f88b301ea0f4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-57rw9" Oct 30 16:32:40.831797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.831700 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmq64\" (UniqueName: \"kubernetes.io/projected/8a21ffcf-b1ed-4709-934e-f88b301ea0f4-kube-api-access-gmq64\") pod \"payload-pre-processing-7b98bb678c-57rw9\" (UID: \"8a21ffcf-b1ed-4709-934e-f88b301ea0f4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-57rw9" Oct 30 16:32:40.932697 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.932606 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmq64\" (UniqueName: \"kubernetes.io/projected/8a21ffcf-b1ed-4709-934e-f88b301ea0f4-kube-api-access-gmq64\") pod \"payload-pre-processing-7b98bb678c-57rw9\" (UID: \"8a21ffcf-b1ed-4709-934e-f88b301ea0f4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-57rw9" Oct 30 16:32:40.933057 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.933035 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a21ffcf-b1ed-4709-934e-f88b301ea0f4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-57rw9\" (UID: \"8a21ffcf-b1ed-4709-934e-f88b301ea0f4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-57rw9" Oct 30 16:32:40.933650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.933624 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a21ffcf-b1ed-4709-934e-f88b301ea0f4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-57rw9\" (UID: \"8a21ffcf-b1ed-4709-934e-f88b301ea0f4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-57rw9" Oct 30 16:32:40.988991 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:40.988923 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmq64\" (UniqueName: \"kubernetes.io/projected/8a21ffcf-b1ed-4709-934e-f88b301ea0f4-kube-api-access-gmq64\") pod \"payload-pre-processing-7b98bb678c-57rw9\" (UID: \"8a21ffcf-b1ed-4709-934e-f88b301ea0f4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-57rw9" Oct 30 16:32:41.060511 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:41.060164 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-sm8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Oct 30 16:32:41.060511 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:41.060240 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" podUID="f4591587-01a3-4160-a8b7-d25450be9054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Oct 30 16:32:41.073226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:41.073098 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8cxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Oct 30 16:32:41.073226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:41.073168 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" podUID="47902ba0-362e-40c1-9589-c2181308e89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Oct 30 16:32:41.104461 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:41.104430 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-57rw9" Oct 30 16:32:41.483976 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:41.478967 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-57rw9"] Oct 30 16:32:41.489820 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:32:41.489297 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a21ffcf_b1ed_4709_934e_f88b301ea0f4.slice/crio-f1e3673d3f6ae6ce09f27c7c1285e90d1905b3a4d69e4b30688cfc178760010e WatchSource:0}: Error finding container f1e3673d3f6ae6ce09f27c7c1285e90d1905b3a4d69e4b30688cfc178760010e: Status 404 returned error can't find the container with id f1e3673d3f6ae6ce09f27c7c1285e90d1905b3a4d69e4b30688cfc178760010e Oct 30 16:32:42.024009 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:42.023963 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-57rw9" event={"ID":"8a21ffcf-b1ed-4709-934e-f88b301ea0f4","Type":"ContainerStarted","Data":"f1e3673d3f6ae6ce09f27c7c1285e90d1905b3a4d69e4b30688cfc178760010e"} Oct 30 16:32:42.048589 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:42.048553 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-sm8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Oct 30 16:32:42.048785 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:42.048622 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" podUID="f4591587-01a3-4160-a8b7-d25450be9054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Oct 30 16:32:42.071576 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:42.071542 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8cxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Oct 30 16:32:42.072052 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:42.071601 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" podUID="47902ba0-362e-40c1-9589-c2181308e89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Oct 30 16:32:43.048623 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:43.048586 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-sm8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Oct 30 16:32:43.048839 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:43.048652 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" podUID="f4591587-01a3-4160-a8b7-d25450be9054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Oct 30 16:32:43.072083 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:43.072045 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8cxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Oct 30 16:32:43.072557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:43.072109 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" podUID="47902ba0-362e-40c1-9589-c2181308e89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Oct 30 16:32:44.036892 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:44.036786 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-57rw9" event={"ID":"8a21ffcf-b1ed-4709-934e-f88b301ea0f4","Type":"ContainerStarted","Data":"4d78ca20583b9f822c3dd8c9e69673c98fe8e4b542d0cff33e6fbe90dde369fa"} Oct 30 16:32:44.037052 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:44.036903 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-57rw9" Oct 30 16:32:44.079731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:44.079665 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-57rw9" podStartSLOduration=1.9849382100000001 podStartE2EDuration="4.079643324s" totalImagesPullingTime="2.094705114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:44.078236535 +0000 UTC m=+677.788691315" watchObservedRunningTime="2026-10-30 16:32:44.079643324 +0000 UTC m=+677.790098068" Oct 30 16:32:44.083031 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:32:44.082995 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:32:45.048474 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:45.048436 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-sm8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:32:45.048665 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:45.048503 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" podUID="f4591587-01a3-4160-a8b7-d25450be9054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:32:45.073123 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:45.073074 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8cxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:32:45.073299 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:45.073159 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" podUID="47902ba0-362e-40c1-9589-c2181308e89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:32:46.089967 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:46.089931 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-sm8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:32:46.090621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:46.090585 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" podUID="f4591587-01a3-4160-a8b7-d25450be9054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:32:46.114904 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:46.114875 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8cxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:32:46.115047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:46.114918 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" podUID="47902ba0-362e-40c1-9589-c2181308e89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:32:46.467703 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:46.467624 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:46.468127 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:46.468105 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:46.710192 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:46.710164 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" Oct 30 16:32:47.060424 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:47.060395 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:47.060694 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:47.060669 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:47.061342 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:47.061325 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 16:32:51.017227 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:51.017197 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8" Oct 30 16:32:51.320487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:51.320400 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6d48fc876f-4g6d7"] Oct 30 16:32:51.324096 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:51.324074 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6d48fc876f-4g6d7" Oct 30 16:32:51.344463 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:51.344434 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6d48fc876f-4g6d7"] Oct 30 16:32:51.354755 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:51.354729 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m4qn\" (UniqueName: \"kubernetes.io/projected/1d405c38-8a59-446b-9517-6804f6e29cdc-kube-api-access-9m4qn\") pod \"ai-gateway-controller-6d48fc876f-4g6d7\" (UID: \"1d405c38-8a59-446b-9517-6804f6e29cdc\") " pod="opendatahub/ai-gateway-controller-6d48fc876f-4g6d7" Oct 30 16:32:51.354885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:51.354783 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d405c38-8a59-446b-9517-6804f6e29cdc-metrics-certs\") pod \"ai-gateway-controller-6d48fc876f-4g6d7\" (UID: \"1d405c38-8a59-446b-9517-6804f6e29cdc\") " pod="opendatahub/ai-gateway-controller-6d48fc876f-4g6d7" Oct 30 16:32:51.456249 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:51.456225 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4qn\" (UniqueName: \"kubernetes.io/projected/1d405c38-8a59-446b-9517-6804f6e29cdc-kube-api-access-9m4qn\") pod \"ai-gateway-controller-6d48fc876f-4g6d7\" (UID: \"1d405c38-8a59-446b-9517-6804f6e29cdc\") " pod="opendatahub/ai-gateway-controller-6d48fc876f-4g6d7" Oct 30 16:32:51.456366 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:51.456260 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d405c38-8a59-446b-9517-6804f6e29cdc-metrics-certs\") pod \"ai-gateway-controller-6d48fc876f-4g6d7\" (UID: \"1d405c38-8a59-446b-9517-6804f6e29cdc\") " pod="opendatahub/ai-gateway-controller-6d48fc876f-4g6d7" Oct 30 16:32:51.458938 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:51.458917 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d405c38-8a59-446b-9517-6804f6e29cdc-metrics-certs\") pod \"ai-gateway-controller-6d48fc876f-4g6d7\" (UID: \"1d405c38-8a59-446b-9517-6804f6e29cdc\") " pod="opendatahub/ai-gateway-controller-6d48fc876f-4g6d7" Oct 30 16:32:51.479641 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:51.479611 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m4qn\" (UniqueName: \"kubernetes.io/projected/1d405c38-8a59-446b-9517-6804f6e29cdc-kube-api-access-9m4qn\") pod \"ai-gateway-controller-6d48fc876f-4g6d7\" (UID: \"1d405c38-8a59-446b-9517-6804f6e29cdc\") " pod="opendatahub/ai-gateway-controller-6d48fc876f-4g6d7" Oct 30 16:32:51.636078 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:51.636054 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6d48fc876f-4g6d7" Oct 30 16:32:51.926150 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:51.926123 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6d48fc876f-4g6d7"] Oct 30 16:32:51.928629 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:32:51.928602 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d405c38_8a59_446b_9517_6804f6e29cdc.slice/crio-6990e742b3f724d65a2cfb5898bd481520674faa30240b2a06141a3e190d50fa WatchSource:0}: Error finding container 6990e742b3f724d65a2cfb5898bd481520674faa30240b2a06141a3e190d50fa: Status 404 returned error can't find the container with id 6990e742b3f724d65a2cfb5898bd481520674faa30240b2a06141a3e190d50fa Oct 30 16:32:52.075275 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:52.075242 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6d48fc876f-4g6d7" event={"ID":"1d405c38-8a59-446b-9517-6804f6e29cdc","Type":"ContainerStarted","Data":"6990e742b3f724d65a2cfb5898bd481520674faa30240b2a06141a3e190d50fa"} Oct 30 16:32:53.080923 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:53.080888 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6d48fc876f-4g6d7" event={"ID":"1d405c38-8a59-446b-9517-6804f6e29cdc","Type":"ContainerStarted","Data":"5faa1baf7b543aeff9cd0f1b43838936e102be7c477c5c66f9e0592686e52c33"} Oct 30 16:32:53.081374 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:53.080977 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6d48fc876f-4g6d7" Oct 30 16:32:53.121417 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:53.121370 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6d48fc876f-4g6d7" podStartSLOduration=1.705261154 podStartE2EDuration="2.121358481s" totalImagesPullingTime="416.097327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:53.121067723 +0000 UTC m=+686.831522467" watchObservedRunningTime="2026-10-30 16:32:53.121358481 +0000 UTC m=+686.831813224" Oct 30 16:32:55.046373 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:32:55.046347 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-57rw9" Oct 30 16:33:04.088225 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:04.088195 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6d48fc876f-4g6d7" Oct 30 16:33:04.161677 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:04.161651 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8"] Oct 30 16:33:04.161893 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:04.161870 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8" podUID="6998c08b-0952-44ac-a708-3c4309686b70" containerName="manager" containerID="cri-o://a2fac3d34c0547edc29a67d5f1968ba23e74daf6e7733a43df38cdd98c9a4c8b" gracePeriod=30 Oct 30 16:33:04.414225 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:04.414200 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8" Oct 30 16:33:04.575603 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:04.575574 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6998c08b-0952-44ac-a708-3c4309686b70-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6998c08b-0952-44ac-a708-3c4309686b70-metrics-certs\") pod \"6998c08b-0952-44ac-a708-3c4309686b70\" (UID: \"6998c08b-0952-44ac-a708-3c4309686b70\") " Oct 30 16:33:04.575802 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:04.575614 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6998c08b-0952-44ac-a708-3c4309686b70-kube-api-access-qw9tl\" (UniqueName: \"kubernetes.io/projected/6998c08b-0952-44ac-a708-3c4309686b70-kube-api-access-qw9tl\") pod \"6998c08b-0952-44ac-a708-3c4309686b70\" (UID: \"6998c08b-0952-44ac-a708-3c4309686b70\") " Oct 30 16:33:04.577986 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:04.577952 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6998c08b-0952-44ac-a708-3c4309686b70-kube-api-access-qw9tl" pod "6998c08b-0952-44ac-a708-3c4309686b70" (UID: "6998c08b-0952-44ac-a708-3c4309686b70"). InnerVolumeSpecName "kube-api-access-qw9tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:33:04.578109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:04.578026 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6998c08b-0952-44ac-a708-3c4309686b70-metrics-certs" pod "6998c08b-0952-44ac-a708-3c4309686b70" (UID: "6998c08b-0952-44ac-a708-3c4309686b70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:33:04.676854 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:04.676786 6202 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6998c08b-0952-44ac-a708-3c4309686b70-metrics-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:33:04.676854 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:04.676810 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw9tl\" (UniqueName: \"kubernetes.io/projected/6998c08b-0952-44ac-a708-3c4309686b70-kube-api-access-qw9tl\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:33:05.135489 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:05.135455 6202 generic.go:356] "Generic (PLEG): container finished" podID="6998c08b-0952-44ac-a708-3c4309686b70" containerID="a2fac3d34c0547edc29a67d5f1968ba23e74daf6e7733a43df38cdd98c9a4c8b" exitCode=0 Oct 30 16:33:05.136006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:05.135505 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8" Oct 30 16:33:05.136006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:05.135521 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8" event={"ID":"6998c08b-0952-44ac-a708-3c4309686b70","Type":"ContainerDied","Data":"a2fac3d34c0547edc29a67d5f1968ba23e74daf6e7733a43df38cdd98c9a4c8b"} Oct 30 16:33:05.136006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:05.135562 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8" event={"ID":"6998c08b-0952-44ac-a708-3c4309686b70","Type":"ContainerDied","Data":"4a6224a6fe87428af3398b43a02a612b131c858f5572ef980d72c78eaf8b1826"} Oct 30 16:33:05.136006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:05.135583 6202 scope.go:122] "RemoveContainer" containerID="a2fac3d34c0547edc29a67d5f1968ba23e74daf6e7733a43df38cdd98c9a4c8b" Oct 30 16:33:05.144738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:05.144722 6202 scope.go:122] "RemoveContainer" containerID="a2fac3d34c0547edc29a67d5f1968ba23e74daf6e7733a43df38cdd98c9a4c8b" Oct 30 16:33:05.145011 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:33:05.144992 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2fac3d34c0547edc29a67d5f1968ba23e74daf6e7733a43df38cdd98c9a4c8b\": container with ID starting with a2fac3d34c0547edc29a67d5f1968ba23e74daf6e7733a43df38cdd98c9a4c8b not found: ID does not exist" containerID="a2fac3d34c0547edc29a67d5f1968ba23e74daf6e7733a43df38cdd98c9a4c8b" Oct 30 16:33:05.145056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:05.145021 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2fac3d34c0547edc29a67d5f1968ba23e74daf6e7733a43df38cdd98c9a4c8b"} err="failed to get container status \"a2fac3d34c0547edc29a67d5f1968ba23e74daf6e7733a43df38cdd98c9a4c8b\": rpc error: code = NotFound desc = could not find container \"a2fac3d34c0547edc29a67d5f1968ba23e74daf6e7733a43df38cdd98c9a4c8b\": container with ID starting with a2fac3d34c0547edc29a67d5f1968ba23e74daf6e7733a43df38cdd98c9a4c8b not found: ID does not exist" Oct 30 16:33:05.194937 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:05.194913 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8"] Oct 30 16:33:05.206899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:05.206873 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-s4pl8"] Oct 30 16:33:06.016373 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:06.016338 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:06.408881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:06.408844 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6998c08b-0952-44ac-a708-3c4309686b70" path="/var/lib/kubelet/pods/6998c08b-0952-44ac-a708-3c4309686b70/volumes" Oct 30 16:33:06.587074 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:06.587040 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:08.356370 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:08.356338 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:08.481530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:08.481499 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:10.195758 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:10.195717 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:14.067196 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:33:14.067160 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:33:20.059649 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:20.059613 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:20.185970 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:20.185939 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:36.172491 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:33:36.172417 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:44.066880 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:33:44.066844 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:34:02.209846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.209812 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:02.332386 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.332328 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:02.445947 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.445890 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:02.823498 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.823459 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb"] Oct 30 16:34:02.823897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.823881 6202 cpu_manager.go:414] "containerMap: removing container" podUID="6998c08b-0952-44ac-a708-3c4309686b70" containerName="manager" Oct 30 16:34:02.823897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.823896 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6998c08b-0952-44ac-a708-3c4309686b70" containerName="manager" Oct 30 16:34:02.824038 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.823974 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="6998c08b-0952-44ac-a708-3c4309686b70" containerName="manager" Oct 30 16:34:02.827124 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.827103 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:02.844549 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.844527 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-750dcb-openshift-default-dockercfg-fbrrg\"" Oct 30 16:34:02.858166 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.858133 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb"] Oct 30 16:34:02.901299 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.901261 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:02.954387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.954326 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8"] Oct 30 16:34:02.958303 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.958275 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:02.974538 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.974505 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3e2f96-openshift-default-dockercfg-v5cl2\"" Oct 30 16:34:02.983415 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.983385 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8"] Oct 30 16:34:02.990905 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.990880 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:02.990905 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.990917 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:02.991132 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.990994 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac56359c-6664-4198-88ef-2208c355cd11-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:02.991204 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.991128 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac56359c-6664-4198-88ef-2208c355cd11-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:02.991204 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.991176 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swsfq\" (UniqueName: \"kubernetes.io/projected/ac56359c-6664-4198-88ef-2208c355cd11-kube-api-access-swsfq\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:02.991325 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.991233 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:02.991325 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.991268 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:02.991325 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.991289 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:02.991472 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:02.991334 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac56359c-6664-4198-88ef-2208c355cd11-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.028462 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.028426 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:03.092476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.092390 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.092476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.092447 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.092881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.092855 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.093793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.093028 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lcbr\" (UniqueName: \"kubernetes.io/projected/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-kube-api-access-8lcbr\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.093793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.093090 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.093793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.093133 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.093793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.093156 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac56359c-6664-4198-88ef-2208c355cd11-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.093793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.093182 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.093793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.093225 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.093793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.093247 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac56359c-6664-4198-88ef-2208c355cd11-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.093793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.093268 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swsfq\" (UniqueName: \"kubernetes.io/projected/ac56359c-6664-4198-88ef-2208c355cd11-kube-api-access-swsfq\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.093793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.093287 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.093793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.093324 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.093793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.093345 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.093793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.093367 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.093793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.093514 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.093793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.093795 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.094648 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.093922 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.094648 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.093985 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.094648 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.094059 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac56359c-6664-4198-88ef-2208c355cd11-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.094648 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.094330 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.094648 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.094556 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.095506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.095417 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac56359c-6664-4198-88ef-2208c355cd11-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.097899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.097856 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac56359c-6664-4198-88ef-2208c355cd11-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.101040 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.101017 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.117067 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.117038 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swsfq\" (UniqueName: \"kubernetes.io/projected/ac56359c-6664-4198-88ef-2208c355cd11-kube-api-access-swsfq\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.121715 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.121690 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac56359c-6664-4198-88ef-2208c355cd11-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.140564 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.140538 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:03.149988 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.149961 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:03.196663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.195545 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.196663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.195629 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lcbr\" (UniqueName: \"kubernetes.io/projected/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-kube-api-access-8lcbr\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.196663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.195672 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.196663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.195711 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.196663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.195736 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.196663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.195798 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.196663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.195830 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.196663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.195873 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.196663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.195903 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.197964 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.197915 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.198105 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.198061 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.199898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.198309 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.199898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.198436 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.199898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.198502 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.200354 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.200273 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.203530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.203336 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.224564 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.224524 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c"] Oct 30 16:34:03.229905 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.229879 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.231114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.231093 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lcbr\" (UniqueName: \"kubernetes.io/projected/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-kube-api-access-8lcbr\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.232229 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.232204 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.252324 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.252295 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c"] Oct 30 16:34:03.266202 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.266170 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:03.276978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.276955 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:03.293421 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.293402 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-d693d0-openshift-default-dockercfg-qggxm\"" Oct 30 16:34:03.396215 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.396122 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:03.397310 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.397288 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.397397 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.397320 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.397397 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.397344 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81b2492-7d46-44ac-a92b-7cd698769695-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.397503 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.397482 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.397551 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.397535 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.397623 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.397609 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81b2492-7d46-44ac-a92b-7cd698769695-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.397678 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.397663 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.397721 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.397709 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a81b2492-7d46-44ac-a92b-7cd698769695-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.397755 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.397735 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trlp2\" (UniqueName: \"kubernetes.io/projected/a81b2492-7d46-44ac-a92b-7cd698769695-kube-api-access-trlp2\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.471281 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.471254 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb"] Oct 30 16:34:03.472872 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:34:03.472835 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac56359c_6664_4198_88ef_2208c355cd11.slice/crio-2bf945ea40fd87e5401d0392f90ad43c07cbe413b4821996528e8bafeaab98da WatchSource:0}: Error finding container 2bf945ea40fd87e5401d0392f90ad43c07cbe413b4821996528e8bafeaab98da: Status 404 returned error can't find the container with id 2bf945ea40fd87e5401d0392f90ad43c07cbe413b4821996528e8bafeaab98da Oct 30 16:34:03.474986 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.474955 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:03.475079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.475015 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:03.475079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.475045 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:03.499245 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.499221 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.499339 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.499256 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.499339 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.499287 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81b2492-7d46-44ac-a92b-7cd698769695-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.499339 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.499315 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.499454 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.499370 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.499454 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.499429 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81b2492-7d46-44ac-a92b-7cd698769695-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.499549 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.499478 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.499549 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.499508 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a81b2492-7d46-44ac-a92b-7cd698769695-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.499660 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.499544 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trlp2\" (UniqueName: \"kubernetes.io/projected/a81b2492-7d46-44ac-a92b-7cd698769695-kube-api-access-trlp2\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.499660 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.499620 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.499748 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.499701 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.499954 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.499915 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.500071 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.499997 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.500235 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.500218 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81b2492-7d46-44ac-a92b-7cd698769695-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.502009 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.501988 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.502106 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.502090 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81b2492-7d46-44ac-a92b-7cd698769695-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.526054 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.526029 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a81b2492-7d46-44ac-a92b-7cd698769695-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.527447 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.527421 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trlp2\" (UniqueName: \"kubernetes.io/projected/a81b2492-7d46-44ac-a92b-7cd698769695-kube-api-access-trlp2\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.546358 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.546334 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:03.548010 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.547987 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9"] Oct 30 16:34:03.551734 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.551717 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.567116 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.567094 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-263f00-openshift-default-dockercfg-nr2tc\"" Oct 30 16:34:03.579014 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.578934 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9"] Oct 30 16:34:03.588457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.588332 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:03.591616 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.591411 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8"] Oct 30 16:34:03.602342 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:34:03.602299 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3d175f_3803_4ef3_b5bc_2b3c55d88a7a.slice/crio-7e55c2a1d44d5631d87d7e10b0bcb7a97d75fba270d5a50b9e72f8f5fa3c9c7c WatchSource:0}: Error finding container 7e55c2a1d44d5631d87d7e10b0bcb7a97d75fba270d5a50b9e72f8f5fa3c9c7c: Status 404 returned error can't find the container with id 7e55c2a1d44d5631d87d7e10b0bcb7a97d75fba270d5a50b9e72f8f5fa3c9c7c Oct 30 16:34:03.606240 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.605550 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:03.606240 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.605629 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:03.606240 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.605670 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:03.702421 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.702378 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.702576 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.702439 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119c2424-9656-41b0-9202-477228e632c2-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.702576 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.702499 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119c2424-9656-41b0-9202-477228e632c2-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.702576 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.702555 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119c2424-9656-41b0-9202-477228e632c2-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.702759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.702624 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.702759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.702673 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.702759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.702731 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.703555 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.702799 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq52x\" (UniqueName: \"kubernetes.io/projected/119c2424-9656-41b0-9202-477228e632c2-kube-api-access-jq52x\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.703555 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.702835 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.706274 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.706249 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:03.806101 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.806064 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.806284 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.806114 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq52x\" (UniqueName: \"kubernetes.io/projected/119c2424-9656-41b0-9202-477228e632c2-kube-api-access-jq52x\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.806284 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.806154 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.806284 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.806195 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.806473 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.806340 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119c2424-9656-41b0-9202-477228e632c2-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.806473 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.806426 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119c2424-9656-41b0-9202-477228e632c2-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.806576 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.806492 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119c2424-9656-41b0-9202-477228e632c2-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.806632 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.806574 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.806632 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.806593 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.806740 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.806629 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.806740 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.806720 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.807039 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.807017 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.807259 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.807232 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.807401 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.807263 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119c2424-9656-41b0-9202-477228e632c2-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.810544 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.810511 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119c2424-9656-41b0-9202-477228e632c2-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.811620 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.811596 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.829559 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.829530 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119c2424-9656-41b0-9202-477228e632c2-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.829667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.829635 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq52x\" (UniqueName: \"kubernetes.io/projected/119c2424-9656-41b0-9202-477228e632c2-kube-api-access-jq52x\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.867825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.867540 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:03.869986 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.869959 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c"] Oct 30 16:34:03.879069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.879032 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:03.879217 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.879114 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:03.879217 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:03.879159 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:04.175161 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.175133 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9"] Oct 30 16:34:04.177122 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:34:04.177088 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod119c2424_9656_41b0_9202_477228e632c2.slice/crio-9f55cced3ff9a6fe54360611a71c5a1e525e79315aab8963fe186c7fc5a69420 WatchSource:0}: Error finding container 9f55cced3ff9a6fe54360611a71c5a1e525e79315aab8963fe186c7fc5a69420: Status 404 returned error can't find the container with id 9f55cced3ff9a6fe54360611a71c5a1e525e79315aab8963fe186c7fc5a69420 Oct 30 16:34:04.179637 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.179593 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:04.179720 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.179685 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:04.179759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.179728 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:04.401359 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.401319 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" event={"ID":"a81b2492-7d46-44ac-a92b-7cd698769695","Type":"ContainerStarted","Data":"ac89cb3f5cacedb924bfd0c405e25b2741c138215753220e7bc234e19dcfc803"} Oct 30 16:34:04.401813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.401389 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" event={"ID":"a81b2492-7d46-44ac-a92b-7cd698769695","Type":"ContainerStarted","Data":"3f6226b3f1317817da896e05ecd421dd91e94a7d67255e6fef3ad5a6dd46a27c"} Oct 30 16:34:04.411777 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.411736 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" event={"ID":"ac56359c-6664-4198-88ef-2208c355cd11","Type":"ContainerStarted","Data":"9fc6e07fb0b1ae86bcbee07e00402a998487aec7d191ffb882dd8c5806ac3409"} Oct 30 16:34:04.411915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.411801 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" event={"ID":"ac56359c-6664-4198-88ef-2208c355cd11","Type":"ContainerStarted","Data":"2bf945ea40fd87e5401d0392f90ad43c07cbe413b4821996528e8bafeaab98da"} Oct 30 16:34:04.411915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.411818 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" event={"ID":"119c2424-9656-41b0-9202-477228e632c2","Type":"ContainerStarted","Data":"d7f4f4fe6abbe924891c9c8c3ce5557bd6a4f3c51b69a12ab2a24b9c06358c7b"} Oct 30 16:34:04.411915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.411834 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" event={"ID":"119c2424-9656-41b0-9202-477228e632c2","Type":"ContainerStarted","Data":"9f55cced3ff9a6fe54360611a71c5a1e525e79315aab8963fe186c7fc5a69420"} Oct 30 16:34:04.411915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.411848 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" event={"ID":"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a","Type":"ContainerStarted","Data":"529709dde15bdd2c696871fd51f5c2e62bf14f13c662a87fcc8b61f2db84a52e"} Oct 30 16:34:04.411915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.411868 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" event={"ID":"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a","Type":"ContainerStarted","Data":"7e55c2a1d44d5631d87d7e10b0bcb7a97d75fba270d5a50b9e72f8f5fa3c9c7c"} Oct 30 16:34:04.446989 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.446881 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" podStartSLOduration=1.446860177 podStartE2EDuration="1.446860177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:04.445123294 +0000 UTC m=+758.155578037" watchObservedRunningTime="2026-10-30 16:34:04.446860177 +0000 UTC m=+758.157314923" Oct 30 16:34:04.490510 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.490460 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" podStartSLOduration=2.490446199 podStartE2EDuration="2.490446199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:04.490108822 +0000 UTC m=+758.200563561" watchObservedRunningTime="2026-10-30 16:34:04.490446199 +0000 UTC m=+758.200900993" Oct 30 16:34:04.536573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.536513 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" podStartSLOduration=2.536494646 podStartE2EDuration="2.536494646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:04.534983326 +0000 UTC m=+758.245438069" watchObservedRunningTime="2026-10-30 16:34:04.536494646 +0000 UTC m=+758.246949394" Oct 30 16:34:04.547378 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.547350 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:04.552317 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.552295 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:04.580022 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.579974 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" podStartSLOduration=1.579959121 podStartE2EDuration="1.579959121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:04.579088885 +0000 UTC m=+758.289543830" watchObservedRunningTime="2026-10-30 16:34:04.579959121 +0000 UTC m=+758.290413865" Oct 30 16:34:04.867925 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:04.867895 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:05.141651 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:05.141550 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:05.146543 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:05.146517 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:05.277968 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:05.277927 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:05.283553 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:05.283527 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:05.415145 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:05.415052 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:05.415621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:05.415559 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:05.415621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:05.415605 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:05.416409 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:05.416386 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:05.416631 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:05.416617 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:05.416707 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:05.416675 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:05.873295 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:05.873265 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:06.419787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:06.419737 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:06.420873 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:06.420850 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:07.062182 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:07.062148 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:07.177531 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:07.177497 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:07.294604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:07.294568 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:07.470889 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:07.470858 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:07.604345 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:07.604312 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:07.744442 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:07.744363 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:08.652761 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:08.652722 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:08.769173 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:08.769134 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:08.840602 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:08.840572 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj"] Oct 30 16:34:08.844163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:08.844141 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj" Oct 30 16:34:08.859603 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:08.859575 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-263f00\"" Oct 30 16:34:08.861238 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:08.861215 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce1d5787-a563-4b1e-822f-b34c2603346b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj\" (UID: \"ce1d5787-a563-4b1e-822f-b34c2603346b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj" Oct 30 16:34:08.861337 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:08.861288 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sngw\" (UniqueName: \"kubernetes.io/projected/ce1d5787-a563-4b1e-822f-b34c2603346b-kube-api-access-9sngw\") pod \"payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj\" (UID: \"ce1d5787-a563-4b1e-822f-b34c2603346b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj" Oct 30 16:34:08.872407 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:08.872380 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj"] Oct 30 16:34:08.892311 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:08.892276 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:08.962882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:08.962794 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce1d5787-a563-4b1e-822f-b34c2603346b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj\" (UID: \"ce1d5787-a563-4b1e-822f-b34c2603346b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj" Oct 30 16:34:08.962882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:08.962838 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sngw\" (UniqueName: \"kubernetes.io/projected/ce1d5787-a563-4b1e-822f-b34c2603346b-kube-api-access-9sngw\") pod \"payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj\" (UID: \"ce1d5787-a563-4b1e-822f-b34c2603346b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj" Oct 30 16:34:08.964032 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:08.964012 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce1d5787-a563-4b1e-822f-b34c2603346b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj\" (UID: \"ce1d5787-a563-4b1e-822f-b34c2603346b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj" Oct 30 16:34:09.005654 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:09.002805 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sngw\" (UniqueName: \"kubernetes.io/projected/ce1d5787-a563-4b1e-822f-b34c2603346b-kube-api-access-9sngw\") pod \"payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj\" (UID: \"ce1d5787-a563-4b1e-822f-b34c2603346b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj" Oct 30 16:34:09.157599 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:09.157561 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj" Oct 30 16:34:09.451123 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:09.451093 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj"] Oct 30 16:34:09.453871 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:34:09.453840 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1d5787_a563_4b1e_822f_b34c2603346b.slice/crio-c23e2ff0fda83be872e72ccc4dc96877eee0b82298279f779864e78444ecac74 WatchSource:0}: Error finding container c23e2ff0fda83be872e72ccc4dc96877eee0b82298279f779864e78444ecac74: Status 404 returned error can't find the container with id c23e2ff0fda83be872e72ccc4dc96877eee0b82298279f779864e78444ecac74 Oct 30 16:34:10.261994 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.261952 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:10.376780 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.376742 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth"] Oct 30 16:34:10.379831 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.379803 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:10.380254 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.380236 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" Oct 30 16:34:10.397212 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.397194 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-3e2f96-dockercfg-2dk7v\"" Oct 30 16:34:10.404028 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.404006 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-3e2f96\"" Oct 30 16:34:10.412080 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.412055 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth"] Oct 30 16:34:10.432184 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.432156 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf"] Oct 30 16:34:10.435909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.435885 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" Oct 30 16:34:10.442240 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.442210 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj" event={"ID":"ce1d5787-a563-4b1e-822f-b34c2603346b","Type":"ContainerStarted","Data":"de870726e83e7d7e42e1e4ae895588ac0d0891c80d52efe10b7560387fd533f3"} Oct 30 16:34:10.442390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.442269 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj" event={"ID":"ce1d5787-a563-4b1e-822f-b34c2603346b","Type":"ContainerStarted","Data":"c23e2ff0fda83be872e72ccc4dc96877eee0b82298279f779864e78444ecac74"} Oct 30 16:34:10.442390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.442352 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj" Oct 30 16:34:10.459500 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.459475 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf"] Oct 30 16:34:10.475400 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.475363 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfwrb\" (UniqueName: \"kubernetes.io/projected/19c858ea-1556-446d-ac02-5c3b2cc6f709-kube-api-access-jfwrb\") pod \"payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth\" (UID: \"19c858ea-1556-446d-ac02-5c3b2cc6f709\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" Oct 30 16:34:10.475568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.475441 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d9wz\" (UniqueName: \"kubernetes.io/projected/dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5-kube-api-access-4d9wz\") pod \"payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf\" (UID: \"dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" Oct 30 16:34:10.475625 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.475606 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf\" (UID: \"dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" Oct 30 16:34:10.475716 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.475698 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c858ea-1556-446d-ac02-5c3b2cc6f709-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth\" (UID: \"19c858ea-1556-446d-ac02-5c3b2cc6f709\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" Oct 30 16:34:10.535059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.534939 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj" podStartSLOduration=2.5349175600000002 podStartE2EDuration="2.53491756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:10.5335065 +0000 UTC m=+764.243961245" watchObservedRunningTime="2026-10-30 16:34:10.53491756 +0000 UTC m=+764.245372305" Oct 30 16:34:10.571089 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.571059 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:10.576487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.576446 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d9wz\" (UniqueName: \"kubernetes.io/projected/dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5-kube-api-access-4d9wz\") pod \"payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf\" (UID: \"dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" Oct 30 16:34:10.576598 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.576547 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf\" (UID: \"dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" Oct 30 16:34:10.576664 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.576606 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c858ea-1556-446d-ac02-5c3b2cc6f709-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth\" (UID: \"19c858ea-1556-446d-ac02-5c3b2cc6f709\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" Oct 30 16:34:10.576722 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.576673 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfwrb\" (UniqueName: \"kubernetes.io/projected/19c858ea-1556-446d-ac02-5c3b2cc6f709-kube-api-access-jfwrb\") pod \"payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth\" (UID: \"19c858ea-1556-446d-ac02-5c3b2cc6f709\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" Oct 30 16:34:10.577313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.577289 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c858ea-1556-446d-ac02-5c3b2cc6f709-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth\" (UID: \"19c858ea-1556-446d-ac02-5c3b2cc6f709\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" Oct 30 16:34:10.577401 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.577332 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf\" (UID: \"dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" Oct 30 16:34:10.600702 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.600667 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfwrb\" (UniqueName: \"kubernetes.io/projected/19c858ea-1556-446d-ac02-5c3b2cc6f709-kube-api-access-jfwrb\") pod \"payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth\" (UID: \"19c858ea-1556-446d-ac02-5c3b2cc6f709\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" Oct 30 16:34:10.605423 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.605395 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d9wz\" (UniqueName: \"kubernetes.io/projected/dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5-kube-api-access-4d9wz\") pod \"payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf\" (UID: \"dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" Oct 30 16:34:10.692097 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.692064 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:10.692314 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.692275 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" Oct 30 16:34:10.752871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:10.752834 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" Oct 30 16:34:11.012219 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.012189 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth"] Oct 30 16:34:11.013917 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:34:11.013893 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c858ea_1556_446d_ac02_5c3b2cc6f709.slice/crio-6d450e48f0faf2c1e62b66751d0fbcfff168c80ceead8a96e1cc2d4abc86379d WatchSource:0}: Error finding container 6d450e48f0faf2c1e62b66751d0fbcfff168c80ceead8a96e1cc2d4abc86379d: Status 404 returned error can't find the container with id 6d450e48f0faf2c1e62b66751d0fbcfff168c80ceead8a96e1cc2d4abc86379d Oct 30 16:34:11.072193 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.072125 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf"] Oct 30 16:34:11.073684 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:34:11.073655 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfbacf8f_cd50_4ed5_a2ed_e7ad2a9a18a5.slice/crio-d58b963c9c8a62581fc991f1f232a3e056370767eaa80797fc7281559901b2c5 WatchSource:0}: Error finding container d58b963c9c8a62581fc991f1f232a3e056370767eaa80797fc7281559901b2c5: Status 404 returned error can't find the container with id d58b963c9c8a62581fc991f1f232a3e056370767eaa80797fc7281559901b2c5 Oct 30 16:34:11.126911 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.126858 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:11.242920 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.242889 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw"] Oct 30 16:34:11.252267 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.252221 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:11.252502 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.252485 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" Oct 30 16:34:11.269307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.269240 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-750dcb\"" Oct 30 16:34:11.269307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.269300 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-750dcb-dockercfg-4rf4d\"" Oct 30 16:34:11.271085 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.271062 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw"] Oct 30 16:34:11.284556 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.284527 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a46e1ced-667f-4218-9fe5-4dafa958b55e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw\" (UID: \"a46e1ced-667f-4218-9fe5-4dafa958b55e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" Oct 30 16:34:11.284730 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.284599 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blngh\" (UniqueName: \"kubernetes.io/projected/a46e1ced-667f-4218-9fe5-4dafa958b55e-kube-api-access-blngh\") pod \"payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw\" (UID: \"a46e1ced-667f-4218-9fe5-4dafa958b55e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" Oct 30 16:34:11.299435 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.299405 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj"] Oct 30 16:34:11.303304 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.303281 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" Oct 30 16:34:11.324495 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.324435 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj"] Oct 30 16:34:11.387282 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.387049 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a46e1ced-667f-4218-9fe5-4dafa958b55e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw\" (UID: \"a46e1ced-667f-4218-9fe5-4dafa958b55e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" Oct 30 16:34:11.388205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.387954 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a46e1ced-667f-4218-9fe5-4dafa958b55e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw\" (UID: \"a46e1ced-667f-4218-9fe5-4dafa958b55e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" Oct 30 16:34:11.388419 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.388136 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b428320-6e0b-463f-be2e-74ec2cfda977-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj\" (UID: \"5b428320-6e0b-463f-be2e-74ec2cfda977\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" Oct 30 16:34:11.388561 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.388495 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blngh\" (UniqueName: \"kubernetes.io/projected/a46e1ced-667f-4218-9fe5-4dafa958b55e-kube-api-access-blngh\") pod \"payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw\" (UID: \"a46e1ced-667f-4218-9fe5-4dafa958b55e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" Oct 30 16:34:11.388561 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.388524 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8m57\" (UniqueName: \"kubernetes.io/projected/5b428320-6e0b-463f-be2e-74ec2cfda977-kube-api-access-h8m57\") pod \"payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj\" (UID: \"5b428320-6e0b-463f-be2e-74ec2cfda977\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" Oct 30 16:34:11.412150 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.412126 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blngh\" (UniqueName: \"kubernetes.io/projected/a46e1ced-667f-4218-9fe5-4dafa958b55e-kube-api-access-blngh\") pod \"payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw\" (UID: \"a46e1ced-667f-4218-9fe5-4dafa958b55e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" Oct 30 16:34:11.449845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.449807 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" event={"ID":"dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5","Type":"ContainerStarted","Data":"340e823abc09bb194758ad312e8dd977383f446bbfcf391d454d4135b024443e"} Oct 30 16:34:11.450016 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.449870 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" event={"ID":"dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5","Type":"ContainerStarted","Data":"d58b963c9c8a62581fc991f1f232a3e056370767eaa80797fc7281559901b2c5"} Oct 30 16:34:11.450016 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.449897 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" Oct 30 16:34:11.451442 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.451414 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" event={"ID":"19c858ea-1556-446d-ac02-5c3b2cc6f709","Type":"ContainerStarted","Data":"657c26198569ee21c59d233188a82629bed1150f4e579a0ca1bb0f43d2c75ef8"} Oct 30 16:34:11.451578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.451460 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" event={"ID":"19c858ea-1556-446d-ac02-5c3b2cc6f709","Type":"ContainerStarted","Data":"6d450e48f0faf2c1e62b66751d0fbcfff168c80ceead8a96e1cc2d4abc86379d"} Oct 30 16:34:11.489342 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.489307 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8m57\" (UniqueName: \"kubernetes.io/projected/5b428320-6e0b-463f-be2e-74ec2cfda977-kube-api-access-h8m57\") pod \"payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj\" (UID: \"5b428320-6e0b-463f-be2e-74ec2cfda977\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" Oct 30 16:34:11.490164 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.489792 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b428320-6e0b-463f-be2e-74ec2cfda977-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj\" (UID: \"5b428320-6e0b-463f-be2e-74ec2cfda977\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" Oct 30 16:34:11.490606 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.490522 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b428320-6e0b-463f-be2e-74ec2cfda977-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj\" (UID: \"5b428320-6e0b-463f-be2e-74ec2cfda977\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" Oct 30 16:34:11.491373 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.491331 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" podStartSLOduration=1.4913196420000001 podStartE2EDuration="1.491319642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:11.490036874 +0000 UTC m=+765.200491616" watchObservedRunningTime="2026-10-30 16:34:11.491319642 +0000 UTC m=+765.201774384" Oct 30 16:34:11.512823 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.512789 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8m57\" (UniqueName: \"kubernetes.io/projected/5b428320-6e0b-463f-be2e-74ec2cfda977-kube-api-access-h8m57\") pod \"payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj\" (UID: \"5b428320-6e0b-463f-be2e-74ec2cfda977\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" Oct 30 16:34:11.530170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.530131 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" podStartSLOduration=1.530120563 podStartE2EDuration="1.530120563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:11.529325695 +0000 UTC m=+765.239780434" watchObservedRunningTime="2026-10-30 16:34:11.530120563 +0000 UTC m=+765.240575304" Oct 30 16:34:11.566738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.566715 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" Oct 30 16:34:11.619194 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.619159 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" Oct 30 16:34:11.859043 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.858950 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw"] Oct 30 16:34:11.872935 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:34:11.872908 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46e1ced_667f_4218_9fe5_4dafa958b55e.slice/crio-ba67980ca4502ecc1b4ed4e7755b2c0c144e0e8e62330a4e7b040cc2d79d2603 WatchSource:0}: Error finding container ba67980ca4502ecc1b4ed4e7755b2c0c144e0e8e62330a4e7b040cc2d79d2603: Status 404 returned error can't find the container with id ba67980ca4502ecc1b4ed4e7755b2c0c144e0e8e62330a4e7b040cc2d79d2603 Oct 30 16:34:11.916211 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:11.916191 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj"] Oct 30 16:34:11.918301 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:34:11.918275 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b428320_6e0b_463f_be2e_74ec2cfda977.slice/crio-f3a8c366e3c89e827924543e10831945073e882d0978b9659d47a5f3b7cc154e WatchSource:0}: Error finding container f3a8c366e3c89e827924543e10831945073e882d0978b9659d47a5f3b7cc154e: Status 404 returned error can't find the container with id f3a8c366e3c89e827924543e10831945073e882d0978b9659d47a5f3b7cc154e Oct 30 16:34:12.251915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.251837 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:12.371826 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.371263 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:12.457811 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.457753 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" event={"ID":"5b428320-6e0b-463f-be2e-74ec2cfda977","Type":"ContainerStarted","Data":"ad737be37cbc8782fdb8841ac5d2ecb9a056d237074637a76214be8e40184441"} Oct 30 16:34:12.457997 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.457831 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" event={"ID":"5b428320-6e0b-463f-be2e-74ec2cfda977","Type":"ContainerStarted","Data":"f3a8c366e3c89e827924543e10831945073e882d0978b9659d47a5f3b7cc154e"} Oct 30 16:34:12.457997 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.457868 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" Oct 30 16:34:12.459243 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.459221 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" event={"ID":"a46e1ced-667f-4218-9fe5-4dafa958b55e","Type":"ContainerStarted","Data":"c28e76120e6e4732e94b57d46b9af757d11697d2663091c37de9ea1bf377fae5"} Oct 30 16:34:12.459354 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.459299 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" Oct 30 16:34:12.459354 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.459337 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" event={"ID":"a46e1ced-667f-4218-9fe5-4dafa958b55e","Type":"ContainerStarted","Data":"ba67980ca4502ecc1b4ed4e7755b2c0c144e0e8e62330a4e7b040cc2d79d2603"} Oct 30 16:34:12.459514 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.459501 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" Oct 30 16:34:12.482513 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.482484 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:12.505874 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.505800 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" podStartSLOduration=1.505754898 podStartE2EDuration="1.505754898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:12.504385047 +0000 UTC m=+766.214839790" watchObservedRunningTime="2026-10-30 16:34:12.505754898 +0000 UTC m=+766.216209637" Oct 30 16:34:12.556793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.556312 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" podStartSLOduration=1.556291811 podStartE2EDuration="1.556291811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:12.55300641 +0000 UTC m=+766.263461154" watchObservedRunningTime="2026-10-30 16:34:12.556291811 +0000 UTC m=+766.266746556" Oct 30 16:34:12.691487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.691458 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:12.725498 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.725472 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf"] Oct 30 16:34:12.729218 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.729197 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" Oct 30 16:34:12.765927 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.765860 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf"] Oct 30 16:34:12.766070 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.766057 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-0b2348\"" Oct 30 16:34:12.784190 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.784160 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn"] Oct 30 16:34:12.788642 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.788622 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" Oct 30 16:34:12.805783 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.805745 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f750fe64-42e5-4d6f-91f7-071c1493c881-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn\" (UID: \"f750fe64-42e5-4d6f-91f7-071c1493c881\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" Oct 30 16:34:12.805921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.805896 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b94338f-f1d5-4e7c-880d-905f46fd9c7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf\" (UID: \"1b94338f-f1d5-4e7c-880d-905f46fd9c7e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" Oct 30 16:34:12.806019 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.806001 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2p77\" (UniqueName: \"kubernetes.io/projected/f750fe64-42e5-4d6f-91f7-071c1493c881-kube-api-access-m2p77\") pod \"payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn\" (UID: \"f750fe64-42e5-4d6f-91f7-071c1493c881\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" Oct 30 16:34:12.806057 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.806037 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pndw\" (UniqueName: \"kubernetes.io/projected/1b94338f-f1d5-4e7c-880d-905f46fd9c7e-kube-api-access-7pndw\") pod \"payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf\" (UID: \"1b94338f-f1d5-4e7c-880d-905f46fd9c7e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" Oct 30 16:34:12.806313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.806286 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-0b2348-dockercfg-8zvrg\"" Oct 30 16:34:12.821207 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.821179 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn"] Oct 30 16:34:12.907434 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.907403 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f750fe64-42e5-4d6f-91f7-071c1493c881-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn\" (UID: \"f750fe64-42e5-4d6f-91f7-071c1493c881\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" Oct 30 16:34:12.907607 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.907463 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b94338f-f1d5-4e7c-880d-905f46fd9c7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf\" (UID: \"1b94338f-f1d5-4e7c-880d-905f46fd9c7e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" Oct 30 16:34:12.907607 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.907527 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2p77\" (UniqueName: \"kubernetes.io/projected/f750fe64-42e5-4d6f-91f7-071c1493c881-kube-api-access-m2p77\") pod \"payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn\" (UID: \"f750fe64-42e5-4d6f-91f7-071c1493c881\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" Oct 30 16:34:12.907607 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.907558 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pndw\" (UniqueName: \"kubernetes.io/projected/1b94338f-f1d5-4e7c-880d-905f46fd9c7e-kube-api-access-7pndw\") pod \"payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf\" (UID: \"1b94338f-f1d5-4e7c-880d-905f46fd9c7e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" Oct 30 16:34:12.908194 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.908172 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f750fe64-42e5-4d6f-91f7-071c1493c881-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn\" (UID: \"f750fe64-42e5-4d6f-91f7-071c1493c881\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" Oct 30 16:34:12.908309 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.908245 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b94338f-f1d5-4e7c-880d-905f46fd9c7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf\" (UID: \"1b94338f-f1d5-4e7c-880d-905f46fd9c7e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" Oct 30 16:34:12.932138 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.932114 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pndw\" (UniqueName: \"kubernetes.io/projected/1b94338f-f1d5-4e7c-880d-905f46fd9c7e-kube-api-access-7pndw\") pod \"payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf\" (UID: \"1b94338f-f1d5-4e7c-880d-905f46fd9c7e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" Oct 30 16:34:12.938679 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.938653 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2p77\" (UniqueName: \"kubernetes.io/projected/f750fe64-42e5-4d6f-91f7-071c1493c881-kube-api-access-m2p77\") pod \"payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn\" (UID: \"f750fe64-42e5-4d6f-91f7-071c1493c881\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" Oct 30 16:34:12.953571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:12.953543 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:13.052201 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:13.052121 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" Oct 30 16:34:13.075159 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:13.075126 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:13.102632 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:13.102591 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" Oct 30 16:34:13.214974 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:13.214020 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:13.354648 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:13.354569 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf"] Oct 30 16:34:13.359498 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:34:13.359465 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b94338f_f1d5_4e7c_880d_905f46fd9c7e.slice/crio-1b00410367b983a4983853129c3023a732cb84c0a42d16626183f928ae657ad4 WatchSource:0}: Error finding container 1b00410367b983a4983853129c3023a732cb84c0a42d16626183f928ae657ad4: Status 404 returned error can't find the container with id 1b00410367b983a4983853129c3023a732cb84c0a42d16626183f928ae657ad4 Oct 30 16:34:13.420413 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:13.419481 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn"] Oct 30 16:34:13.429944 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:34:13.429915 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf750fe64_42e5_4d6f_91f7_071c1493c881.slice/crio-7e633b9ffe15bd888354d2c699b780a71c10d316541f219d36cd7470017c1e8b WatchSource:0}: Error finding container 7e633b9ffe15bd888354d2c699b780a71c10d316541f219d36cd7470017c1e8b: Status 404 returned error can't find the container with id 7e633b9ffe15bd888354d2c699b780a71c10d316541f219d36cd7470017c1e8b Oct 30 16:34:13.475483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:13.475416 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" event={"ID":"f750fe64-42e5-4d6f-91f7-071c1493c881","Type":"ContainerStarted","Data":"7e633b9ffe15bd888354d2c699b780a71c10d316541f219d36cd7470017c1e8b"} Oct 30 16:34:13.477781 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:13.477724 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" event={"ID":"1b94338f-f1d5-4e7c-880d-905f46fd9c7e","Type":"ContainerStarted","Data":"1b00410367b983a4983853129c3023a732cb84c0a42d16626183f928ae657ad4"} Oct 30 16:34:14.097866 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:34:14.097805 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:34:14.124093 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:14.124061 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:14.254551 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:14.254517 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:14.365968 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:14.365882 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:14.484088 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:14.484054 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" event={"ID":"1b94338f-f1d5-4e7c-880d-905f46fd9c7e","Type":"ContainerStarted","Data":"f778a7ac0ea6b96d017c00add1efc0dbb8b7fa2135dae671c99b669e524868aa"} Oct 30 16:34:14.484591 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:14.484207 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" Oct 30 16:34:14.485654 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:14.485631 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" event={"ID":"f750fe64-42e5-4d6f-91f7-071c1493c881","Type":"ContainerStarted","Data":"704f712bbb2dbfd40ae22408a6c69ba1e475c9ff4d79039bced0825f04ae1a49"} Oct 30 16:34:14.485814 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:14.485793 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" Oct 30 16:34:14.525901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:14.525859 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" podStartSLOduration=2.525846354 podStartE2EDuration="2.525846354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:14.524187136 +0000 UTC m=+768.234641876" watchObservedRunningTime="2026-10-30 16:34:14.525846354 +0000 UTC m=+768.236301096" Oct 30 16:34:14.582420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:14.582372 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" podStartSLOduration=2.58235919 podStartE2EDuration="2.58235919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:14.58197754 +0000 UTC m=+768.292432284" watchObservedRunningTime="2026-10-30 16:34:14.58235919 +0000 UTC m=+768.292813933" Oct 30 16:34:15.443097 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:15.443061 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:15.569028 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:15.568995 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:15.701922 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:15.701840 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:16.435348 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:16.435315 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:16.594036 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:16.593991 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:17.583780 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:17.583729 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:17.719792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:17.719636 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:17.755789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:17.746582 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j"] Oct 30 16:34:17.834850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:17.816390 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j"] Oct 30 16:34:17.834850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:17.817212 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" Oct 30 16:34:17.848800 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:17.847415 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/040124e8-ee08-4658-8088-a7563fd57c1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j\" (UID: \"040124e8-ee08-4658-8088-a7563fd57c1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" Oct 30 16:34:17.848800 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:17.847487 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4qpn\" (UniqueName: \"kubernetes.io/projected/040124e8-ee08-4658-8088-a7563fd57c1b-kube-api-access-v4qpn\") pod \"payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j\" (UID: \"040124e8-ee08-4658-8088-a7563fd57c1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" Oct 30 16:34:17.880813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:17.879528 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-d693d0\"" Oct 30 16:34:17.880813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:17.879658 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-d693d0-dockercfg-9ztsg\"" Oct 30 16:34:17.880813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:17.879908 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn"] Oct 30 16:34:17.904154 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:17.903161 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" Oct 30 16:34:17.926973 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:17.922656 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn"] Oct 30 16:34:17.989793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:17.989361 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64359b58-0df8-41d1-9461-c87f3f6e6fa6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn\" (UID: \"64359b58-0df8-41d1-9461-c87f3f6e6fa6\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" Oct 30 16:34:17.989793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:17.989539 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/040124e8-ee08-4658-8088-a7563fd57c1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j\" (UID: \"040124e8-ee08-4658-8088-a7563fd57c1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" Oct 30 16:34:18.003917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:17.990175 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6zfn\" (UniqueName: \"kubernetes.io/projected/64359b58-0df8-41d1-9461-c87f3f6e6fa6-kube-api-access-z6zfn\") pod \"payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn\" (UID: \"64359b58-0df8-41d1-9461-c87f3f6e6fa6\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" Oct 30 16:34:18.003917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:17.990226 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4qpn\" (UniqueName: \"kubernetes.io/projected/040124e8-ee08-4658-8088-a7563fd57c1b-kube-api-access-v4qpn\") pod \"payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j\" (UID: \"040124e8-ee08-4658-8088-a7563fd57c1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" Oct 30 16:34:18.023391 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:18.004259 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/040124e8-ee08-4658-8088-a7563fd57c1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j\" (UID: \"040124e8-ee08-4658-8088-a7563fd57c1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" Oct 30 16:34:18.105531 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:18.105430 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64359b58-0df8-41d1-9461-c87f3f6e6fa6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn\" (UID: \"64359b58-0df8-41d1-9461-c87f3f6e6fa6\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" Oct 30 16:34:18.106921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:18.106895 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6zfn\" (UniqueName: \"kubernetes.io/projected/64359b58-0df8-41d1-9461-c87f3f6e6fa6-kube-api-access-z6zfn\") pod \"payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn\" (UID: \"64359b58-0df8-41d1-9461-c87f3f6e6fa6\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" Oct 30 16:34:18.119792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:18.106731 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64359b58-0df8-41d1-9461-c87f3f6e6fa6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn\" (UID: \"64359b58-0df8-41d1-9461-c87f3f6e6fa6\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" Oct 30 16:34:18.120871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:18.120817 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4qpn\" (UniqueName: \"kubernetes.io/projected/040124e8-ee08-4658-8088-a7563fd57c1b-kube-api-access-v4qpn\") pod \"payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j\" (UID: \"040124e8-ee08-4658-8088-a7563fd57c1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" Oct 30 16:34:18.134257 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:18.134182 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6zfn\" (UniqueName: \"kubernetes.io/projected/64359b58-0df8-41d1-9461-c87f3f6e6fa6-kube-api-access-z6zfn\") pod \"payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn\" (UID: \"64359b58-0df8-41d1-9461-c87f3f6e6fa6\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" Oct 30 16:34:18.175253 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:18.169554 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" Oct 30 16:34:18.261793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:18.255410 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" Oct 30 16:34:18.419689 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:18.419574 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:19.127373 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:19.127348 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j"] Oct 30 16:34:19.129473 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:34:19.129449 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040124e8_ee08_4658_8088_a7563fd57c1b.slice/crio-9c0d2dcc7cbd4ae050ce62ed76c0977a31f81c942b1907f65e0d19abf8b64917 WatchSource:0}: Error finding container 9c0d2dcc7cbd4ae050ce62ed76c0977a31f81c942b1907f65e0d19abf8b64917: Status 404 returned error can't find the container with id 9c0d2dcc7cbd4ae050ce62ed76c0977a31f81c942b1907f65e0d19abf8b64917 Oct 30 16:34:19.232629 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:19.232598 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn"] Oct 30 16:34:19.270843 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:34:19.270808 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64359b58_0df8_41d1_9461_c87f3f6e6fa6.slice/crio-d651b8ee5cc90b651db6f027b66e824b4a91289f8515d7a618a293a6650be80b WatchSource:0}: Error finding container d651b8ee5cc90b651db6f027b66e824b4a91289f8515d7a618a293a6650be80b: Status 404 returned error can't find the container with id d651b8ee5cc90b651db6f027b66e824b4a91289f8515d7a618a293a6650be80b Oct 30 16:34:19.799823 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:19.798728 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" event={"ID":"64359b58-0df8-41d1-9461-c87f3f6e6fa6","Type":"ContainerStarted","Data":"0cbd64b6238566314d62f047e7d9d3bdb37424e497a9f43c9c13f2ec9197a7ab"} Oct 30 16:34:19.799823 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:19.798818 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" event={"ID":"64359b58-0df8-41d1-9461-c87f3f6e6fa6","Type":"ContainerStarted","Data":"d651b8ee5cc90b651db6f027b66e824b4a91289f8515d7a618a293a6650be80b"} Oct 30 16:34:19.799823 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:19.799676 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" Oct 30 16:34:19.802934 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:19.802232 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" event={"ID":"040124e8-ee08-4658-8088-a7563fd57c1b","Type":"ContainerStarted","Data":"6049d01c230b5f2608e81db63a56dffe38bb0b26517faab3b19bdf32476325e0"} Oct 30 16:34:19.802934 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:19.802276 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" event={"ID":"040124e8-ee08-4658-8088-a7563fd57c1b","Type":"ContainerStarted","Data":"9c0d2dcc7cbd4ae050ce62ed76c0977a31f81c942b1907f65e0d19abf8b64917"} Oct 30 16:34:19.802934 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:19.802871 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" Oct 30 16:34:19.900912 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:19.895710 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" podStartSLOduration=2.895691085 podStartE2EDuration="2.895691085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:19.845917203 +0000 UTC m=+773.556371948" watchObservedRunningTime="2026-10-30 16:34:19.895691085 +0000 UTC m=+773.606145829" Oct 30 16:34:19.919621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:19.919584 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:20.035427 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.035212 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" podStartSLOduration=3.035191574 podStartE2EDuration="3.035191574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:19.898190494 +0000 UTC m=+773.608645239" watchObservedRunningTime="2026-10-30 16:34:20.035191574 +0000 UTC m=+773.745646318" Oct 30 16:34:20.035634 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.035609 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:20.036667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.036377 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6"] Oct 30 16:34:20.063146 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.063053 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6" Oct 30 16:34:20.075418 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.069910 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6"] Oct 30 16:34:20.097853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.096014 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-967fac-dockercfg-9xz57\"" Oct 30 16:34:20.097853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.096742 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-967fac\"" Oct 30 16:34:20.126472 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.126442 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m"] Oct 30 16:34:20.130281 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.130260 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m" Oct 30 16:34:20.200886 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.200850 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6733b6dd-929a-48c6-a8d8-ada293bc5079-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6\" (UID: \"6733b6dd-929a-48c6-a8d8-ada293bc5079\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6" Oct 30 16:34:20.201076 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.200943 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f97td\" (UniqueName: \"kubernetes.io/projected/6733b6dd-929a-48c6-a8d8-ada293bc5079-kube-api-access-f97td\") pod \"payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6\" (UID: \"6733b6dd-929a-48c6-a8d8-ada293bc5079\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6" Oct 30 16:34:20.201076 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.200990 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e64229e-1a05-4bdd-8ebc-f01a3661584e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m\" (UID: \"8e64229e-1a05-4bdd-8ebc-f01a3661584e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m" Oct 30 16:34:20.201076 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.201018 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d42pl\" (UniqueName: \"kubernetes.io/projected/8e64229e-1a05-4bdd-8ebc-f01a3661584e-kube-api-access-d42pl\") pod \"payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m\" (UID: \"8e64229e-1a05-4bdd-8ebc-f01a3661584e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m" Oct 30 16:34:20.201623 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.201595 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m"] Oct 30 16:34:20.203860 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.203583 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:20.304527 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.303601 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f97td\" (UniqueName: \"kubernetes.io/projected/6733b6dd-929a-48c6-a8d8-ada293bc5079-kube-api-access-f97td\") pod \"payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6\" (UID: \"6733b6dd-929a-48c6-a8d8-ada293bc5079\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6" Oct 30 16:34:20.304527 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.303662 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e64229e-1a05-4bdd-8ebc-f01a3661584e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m\" (UID: \"8e64229e-1a05-4bdd-8ebc-f01a3661584e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m" Oct 30 16:34:20.304527 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.303694 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d42pl\" (UniqueName: \"kubernetes.io/projected/8e64229e-1a05-4bdd-8ebc-f01a3661584e-kube-api-access-d42pl\") pod \"payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m\" (UID: \"8e64229e-1a05-4bdd-8ebc-f01a3661584e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m" Oct 30 16:34:20.304527 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.303817 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6733b6dd-929a-48c6-a8d8-ada293bc5079-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6\" (UID: \"6733b6dd-929a-48c6-a8d8-ada293bc5079\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6" Oct 30 16:34:20.304915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.304559 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e64229e-1a05-4bdd-8ebc-f01a3661584e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m\" (UID: \"8e64229e-1a05-4bdd-8ebc-f01a3661584e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m" Oct 30 16:34:20.304915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.304666 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6733b6dd-929a-48c6-a8d8-ada293bc5079-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6\" (UID: \"6733b6dd-929a-48c6-a8d8-ada293bc5079\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6" Oct 30 16:34:20.329275 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.329074 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d42pl\" (UniqueName: \"kubernetes.io/projected/8e64229e-1a05-4bdd-8ebc-f01a3661584e-kube-api-access-d42pl\") pod \"payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m\" (UID: \"8e64229e-1a05-4bdd-8ebc-f01a3661584e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m" Oct 30 16:34:20.329275 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.329234 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f97td\" (UniqueName: \"kubernetes.io/projected/6733b6dd-929a-48c6-a8d8-ada293bc5079-kube-api-access-f97td\") pod \"payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6\" (UID: \"6733b6dd-929a-48c6-a8d8-ada293bc5079\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6" Oct 30 16:34:20.411370 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.411333 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6" Oct 30 16:34:20.511921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.510621 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m" Oct 30 16:34:20.770971 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.762802 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6"] Oct 30 16:34:20.993844 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:20.962796 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m"] Oct 30 16:34:21.481052 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:21.481008 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj" Oct 30 16:34:21.780387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:21.780297 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:21.829405 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:21.829364 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m" event={"ID":"8e64229e-1a05-4bdd-8ebc-f01a3661584e","Type":"ContainerStarted","Data":"130539f5138f7f496c0fc17cff039960d86e8c2a05ce4f11d318618608864845"} Oct 30 16:34:21.829563 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:21.829444 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m" event={"ID":"8e64229e-1a05-4bdd-8ebc-f01a3661584e","Type":"ContainerStarted","Data":"7f45730e72738319bf55b16d8a0f177acd6039cdf7682c95894caa27dfcf2b80"} Oct 30 16:34:21.829872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:21.829851 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m" Oct 30 16:34:21.835642 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:21.835605 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6" event={"ID":"6733b6dd-929a-48c6-a8d8-ada293bc5079","Type":"ContainerStarted","Data":"1b9656704c2b23703d73f46621851b1e2b24bc016e8aa3ffa6c52369d3f8e759"} Oct 30 16:34:21.835810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:21.835711 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6" event={"ID":"6733b6dd-929a-48c6-a8d8-ada293bc5079","Type":"ContainerStarted","Data":"c234be4059fe1487d97af1fe926364c58f8eb6db3b0705dccdf4c7179985bd30"} Oct 30 16:34:21.835878 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:21.835863 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6" Oct 30 16:34:21.902420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:21.902380 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:21.974794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:21.962599 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6" podStartSLOduration=1.962578294 podStartE2EDuration="1.962578294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:21.960424929 +0000 UTC m=+775.670879670" watchObservedRunningTime="2026-10-30 16:34:21.962578294 +0000 UTC m=+775.673033039" Oct 30 16:34:21.975285 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:21.975234 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m" podStartSLOduration=1.975219489 podStartE2EDuration="1.975219489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:21.893201252 +0000 UTC m=+775.603655996" watchObservedRunningTime="2026-10-30 16:34:21.975219489 +0000 UTC m=+775.685674233" Oct 30 16:34:21.982059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:21.981995 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs"] Oct 30 16:34:21.988863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:21.988838 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs"] Oct 30 16:34:21.989173 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:21.989110 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs" Oct 30 16:34:22.005071 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.004987 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-a8c22d\"" Oct 30 16:34:22.005196 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.005133 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-a8c22d-dockercfg-d5mlc\"" Oct 30 16:34:22.030758 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.030672 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtwk6\" (UniqueName: \"kubernetes.io/projected/9a88a4db-f16f-4663-b7b3-acce72983c69-kube-api-access-xtwk6\") pod \"payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs\" (UID: \"9a88a4db-f16f-4663-b7b3-acce72983c69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs" Oct 30 16:34:22.030758 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.030723 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a88a4db-f16f-4663-b7b3-acce72983c69-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs\" (UID: \"9a88a4db-f16f-4663-b7b3-acce72983c69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs" Oct 30 16:34:22.053418 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.052959 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5"] Oct 30 16:34:22.079126 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.079084 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5" Oct 30 16:34:22.083366 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.083331 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5"] Oct 30 16:34:22.098288 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.098203 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:22.132286 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.131885 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02f9f64-d9f3-4fe2-b2fb-a994a92796aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5\" (UID: \"f02f9f64-d9f3-4fe2-b2fb-a994a92796aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5" Oct 30 16:34:22.132286 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.131924 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwmd2\" (UniqueName: \"kubernetes.io/projected/f02f9f64-d9f3-4fe2-b2fb-a994a92796aa-kube-api-access-gwmd2\") pod \"payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5\" (UID: \"f02f9f64-d9f3-4fe2-b2fb-a994a92796aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5" Oct 30 16:34:22.132286 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.131971 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtwk6\" (UniqueName: \"kubernetes.io/projected/9a88a4db-f16f-4663-b7b3-acce72983c69-kube-api-access-xtwk6\") pod \"payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs\" (UID: \"9a88a4db-f16f-4663-b7b3-acce72983c69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs" Oct 30 16:34:22.132286 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.132034 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a88a4db-f16f-4663-b7b3-acce72983c69-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs\" (UID: \"9a88a4db-f16f-4663-b7b3-acce72983c69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs" Oct 30 16:34:22.132910 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.132883 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a88a4db-f16f-4663-b7b3-acce72983c69-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs\" (UID: \"9a88a4db-f16f-4663-b7b3-acce72983c69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs" Oct 30 16:34:22.174794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.169956 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtwk6\" (UniqueName: \"kubernetes.io/projected/9a88a4db-f16f-4663-b7b3-acce72983c69-kube-api-access-xtwk6\") pod \"payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs\" (UID: \"9a88a4db-f16f-4663-b7b3-acce72983c69\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs" Oct 30 16:34:22.233291 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.233257 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02f9f64-d9f3-4fe2-b2fb-a994a92796aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5\" (UID: \"f02f9f64-d9f3-4fe2-b2fb-a994a92796aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5" Oct 30 16:34:22.233473 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.233299 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwmd2\" (UniqueName: \"kubernetes.io/projected/f02f9f64-d9f3-4fe2-b2fb-a994a92796aa-kube-api-access-gwmd2\") pod \"payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5\" (UID: \"f02f9f64-d9f3-4fe2-b2fb-a994a92796aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5" Oct 30 16:34:22.234211 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.234181 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02f9f64-d9f3-4fe2-b2fb-a994a92796aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5\" (UID: \"f02f9f64-d9f3-4fe2-b2fb-a994a92796aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5" Oct 30 16:34:22.261940 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.261876 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwmd2\" (UniqueName: \"kubernetes.io/projected/f02f9f64-d9f3-4fe2-b2fb-a994a92796aa-kube-api-access-gwmd2\") pod \"payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5\" (UID: \"f02f9f64-d9f3-4fe2-b2fb-a994a92796aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5" Oct 30 16:34:22.308247 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.308133 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:22.312080 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.312053 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs" Oct 30 16:34:22.409114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.409085 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5" Oct 30 16:34:22.463332 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.462632 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" Oct 30 16:34:22.463332 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.463068 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:34:22.463332 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.463117 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" podUID="119c2424-9656-41b0-9202-477228e632c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:34:22.626615 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.626583 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs"] Oct 30 16:34:22.628028 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:34:22.627996 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a88a4db_f16f_4663_b7b3_acce72983c69.slice/crio-520a53e3821bd2939f57f1de2a1b7c911408b889643a8e29a5b4db72c318e264 WatchSource:0}: Error finding container 520a53e3821bd2939f57f1de2a1b7c911408b889643a8e29a5b4db72c318e264: Status 404 returned error can't find the container with id 520a53e3821bd2939f57f1de2a1b7c911408b889643a8e29a5b4db72c318e264 Oct 30 16:34:22.754083 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.754052 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5"] Oct 30 16:34:22.785738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.785708 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:22.853749 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.851911 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5" event={"ID":"f02f9f64-d9f3-4fe2-b2fb-a994a92796aa","Type":"ContainerStarted","Data":"18908458f95ad16dfa16ab9ead851c033912044c516345cb51193ed4db04a70a"} Oct 30 16:34:22.853749 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.852005 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5" event={"ID":"f02f9f64-d9f3-4fe2-b2fb-a994a92796aa","Type":"ContainerStarted","Data":"f24415a61a3e970c1ea872d1d0c9147fe3ae9116598555268cabd2c4686244ec"} Oct 30 16:34:22.856238 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.856041 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5" Oct 30 16:34:22.858720 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.858696 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs" event={"ID":"9a88a4db-f16f-4663-b7b3-acce72983c69","Type":"ContainerStarted","Data":"1c11d50b0915ece3776001fbd9d79322054587ef0ca9b233cded56bb3beaa93e"} Oct 30 16:34:22.859650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.859629 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs" event={"ID":"9a88a4db-f16f-4663-b7b3-acce72983c69","Type":"ContainerStarted","Data":"520a53e3821bd2939f57f1de2a1b7c911408b889643a8e29a5b4db72c318e264"} Oct 30 16:34:22.859855 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.859841 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs" Oct 30 16:34:22.895134 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.895023 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5" podStartSLOduration=0.895001632 podStartE2EDuration="895.001632ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:22.893854085 +0000 UTC m=+776.604308830" watchObservedRunningTime="2026-10-30 16:34:22.895001632 +0000 UTC m=+776.605456377" Oct 30 16:34:22.957508 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:22.957418 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:23.097514 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:23.097469 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:23.482032 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:23.481996 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" Oct 30 16:34:23.482237 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:23.482213 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" Oct 30 16:34:23.482317 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:23.482292 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" Oct 30 16:34:23.527791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:23.527621 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs" podStartSLOduration=2.527600131 podStartE2EDuration="2.527600131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:22.958725942 +0000 UTC m=+776.669180680" watchObservedRunningTime="2026-10-30 16:34:23.527600131 +0000 UTC m=+777.238054877" Oct 30 16:34:24.346248 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:24.346203 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:24.539207 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:24.539123 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:24.781517 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:24.781416 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:25.496654 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:25.496607 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" Oct 30 16:34:26.937579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:25.496699 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" Oct 30 16:34:26.937579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:26.427600 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:26.937579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:26.542475 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:26.937579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:26.656944 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:26.937579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:26.890818 6202 generic.go:356] "Generic (PLEG): container finished" podID="119c2424-9656-41b0-9202-477228e632c2" containerID="d7f4f4fe6abbe924891c9c8c3ce5557bd6a4f3c51b69a12ab2a24b9c06358c7b" exitCode=0 Oct 30 16:34:26.937579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:26.890863 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" event={"ID":"119c2424-9656-41b0-9202-477228e632c2","Type":"ContainerDied","Data":"d7f4f4fe6abbe924891c9c8c3ce5557bd6a4f3c51b69a12ab2a24b9c06358c7b"} Oct 30 16:34:26.937579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:26.891211 6202 scope.go:122] "RemoveContainer" containerID="d7f4f4fe6abbe924891c9c8c3ce5557bd6a4f3c51b69a12ab2a24b9c06358c7b" Oct 30 16:34:26.937579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:26.892459 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:26.937579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:26.892524 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:26.937579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:26.892553 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:27.257601 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:27.257491 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:27.898544 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:27.898508 6202 generic.go:356] "Generic (PLEG): container finished" podID="ac56359c-6664-4198-88ef-2208c355cd11" containerID="9fc6e07fb0b1ae86bcbee07e00402a998487aec7d191ffb882dd8c5806ac3409" exitCode=0 Oct 30 16:34:27.898544 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:27.898536 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" event={"ID":"ac56359c-6664-4198-88ef-2208c355cd11","Type":"ContainerDied","Data":"9fc6e07fb0b1ae86bcbee07e00402a998487aec7d191ffb882dd8c5806ac3409"} Oct 30 16:34:27.899049 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:27.899032 6202 scope.go:122] "RemoveContainer" containerID="9fc6e07fb0b1ae86bcbee07e00402a998487aec7d191ffb882dd8c5806ac3409" Oct 30 16:34:27.900524 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:27.900486 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:27.900674 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:27.900574 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:27.900674 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:27.900615 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:27.901917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:27.901890 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" event={"ID":"119c2424-9656-41b0-9202-477228e632c2","Type":"ContainerStarted","Data":"d469be6f7acbb8a9449e1ca032cade02dc23e7c0279d005974cd930e5645dc40"} Oct 30 16:34:27.903732 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:27.903708 6202 generic.go:356] "Generic (PLEG): container finished" podID="dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" containerID="529709dde15bdd2c696871fd51f5c2e62bf14f13c662a87fcc8b61f2db84a52e" exitCode=0 Oct 30 16:34:27.903870 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:27.903782 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" event={"ID":"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a","Type":"ContainerDied","Data":"529709dde15bdd2c696871fd51f5c2e62bf14f13c662a87fcc8b61f2db84a52e"} Oct 30 16:34:27.904151 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:27.904133 6202 scope.go:122] "RemoveContainer" containerID="529709dde15bdd2c696871fd51f5c2e62bf14f13c662a87fcc8b61f2db84a52e" Oct 30 16:34:27.911531 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:27.911496 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:27.911611 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:27.911582 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:27.911661 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:27.911628 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:28.098973 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:28.097871 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:28.869182 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:28.869147 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:28.869533 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:28.869507 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 30 16:34:28.869682 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:28.869659 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" podUID="119c2424-9656-41b0-9202-477228e632c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 30 16:34:28.924794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:28.920940 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" event={"ID":"ac56359c-6664-4198-88ef-2208c355cd11","Type":"ContainerStarted","Data":"9254c8218a3a8994970b2d2cdd37035c6e03cdb3863de34b26ad6a3f1b377ce6"} Oct 30 16:34:28.929818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:28.925334 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" event={"ID":"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a","Type":"ContainerStarted","Data":"62766045165b4eaf44f22b983bc3d8b585d76a3109bf8bb6d94b4d3187ee0252"} Oct 30 16:34:29.151847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:29.149835 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:29.151847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:29.149953 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 16:34:29.151847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:29.150017 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" podUID="ac56359c-6664-4198-88ef-2208c355cd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 16:34:29.284794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:29.282744 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:29.284794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:29.282909 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 16:34:29.284794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:29.282955 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" podUID="dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 16:34:29.815559 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:29.815407 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:29.868200 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:29.868164 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 30 16:34:29.868459 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:29.868433 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" podUID="119c2424-9656-41b0-9202-477228e632c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 30 16:34:29.996340 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:29.993673 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:30.145806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:30.142173 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 16:34:30.145806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:30.142235 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" podUID="ac56359c-6664-4198-88ef-2208c355cd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 16:34:30.220535 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:30.219701 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:30.282792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:30.278317 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 16:34:30.282792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:30.278395 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" podUID="dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 16:34:30.890041 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:30.889907 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 30 16:34:30.890041 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:30.889967 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" podUID="119c2424-9656-41b0-9202-477228e632c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 30 16:34:31.145850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:31.144227 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 16:34:31.145850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:31.144292 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" podUID="ac56359c-6664-4198-88ef-2208c355cd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 16:34:31.286405 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:31.286367 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 16:34:31.287033 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:31.286997 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" podUID="dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 16:34:31.816317 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:31.816280 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:31.861226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:31.861196 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" Oct 30 16:34:31.870316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:31.870265 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 30 16:34:31.870605 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:31.870574 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" podUID="119c2424-9656-41b0-9202-477228e632c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 30 16:34:31.871286 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:31.871269 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" Oct 30 16:34:32.142825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:32.142273 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 16:34:32.142825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:32.142336 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" podUID="ac56359c-6664-4198-88ef-2208c355cd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 16:34:32.279797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:32.279532 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 16:34:32.279797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:32.279606 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" podUID="dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 16:34:32.466844 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:32.466424 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:32.877793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:32.876951 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m" Oct 30 16:34:32.881792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:32.878101 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 30 16:34:32.881792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:32.878162 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" podUID="119c2424-9656-41b0-9202-477228e632c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 30 16:34:33.139859 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:33.138578 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:33.142799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:33.142087 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 16:34:33.142799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:33.142139 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" podUID="ac56359c-6664-4198-88ef-2208c355cd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 16:34:33.261554 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:33.261517 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:33.278183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:33.278145 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 16:34:33.279356 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:33.279325 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" podUID="dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 16:34:33.410852 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:33.402006 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:33.709963 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:33.709869 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:33.887029 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:33.886992 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6" Oct 30 16:34:34.149492 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:34.149461 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 16:34:34.149799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:34.149745 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" podUID="ac56359c-6664-4198-88ef-2208c355cd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 16:34:34.306805 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:34.290877 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 16:34:34.306805 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:34.290935 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" podUID="dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 16:34:34.918738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:34.916703 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs" Oct 30 16:34:34.920878 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:34.920856 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5" Oct 30 16:34:34.957483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:34.956876 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:34:34.957483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:34.956937 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" podUID="119c2424-9656-41b0-9202-477228e632c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:34:35.028180 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:35.028150 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:35.032276 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:35.031494 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:35.052793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:35.038002 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:34:35.286127 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:35.278252 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 16:34:35.286127 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:35.278310 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" podUID="dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 16:34:35.361158 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:35.361119 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:35.378121 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:35.378093 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:35.403792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:35.400584 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 16:34:35.419043 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:35.418855 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:35.814821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:35.814788 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:36.275312 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:36.275278 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:36.282335 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:36.282312 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:36.328482 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:36.328455 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 16:34:36.497153 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:36.497120 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:37.276438 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:37.276398 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:37.635846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:37.635807 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:37.947377 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:37.947295 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:38.074845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:38.074812 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:38.192620 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:38.192576 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:39.450397 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:39.450346 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:40.350729 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:40.350698 6202 generic.go:356] "Generic (PLEG): container finished" podID="a81b2492-7d46-44ac-a92b-7cd698769695" containerID="ac89cb3f5cacedb924bfd0c405e25b2741c138215753220e7bc234e19dcfc803" exitCode=0 Oct 30 16:34:40.350956 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:40.350781 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" event={"ID":"a81b2492-7d46-44ac-a92b-7cd698769695","Type":"ContainerDied","Data":"ac89cb3f5cacedb924bfd0c405e25b2741c138215753220e7bc234e19dcfc803"} Oct 30 16:34:40.351229 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:40.351211 6202 scope.go:122] "RemoveContainer" containerID="ac89cb3f5cacedb924bfd0c405e25b2741c138215753220e7bc234e19dcfc803" Oct 30 16:34:40.352780 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:40.352725 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:40.352883 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:40.352841 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:40.352924 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:40.352889 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:41.363385 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:41.363311 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:41.371561 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:41.371523 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" event={"ID":"a81b2492-7d46-44ac-a92b-7cd698769695","Type":"ContainerStarted","Data":"fc2689089f56e955f2acb5c4baef04c6963c29e2fb519737a31bfd741045c755"} Oct 30 16:34:41.552255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:41.552219 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:41.552904 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:41.552874 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 16:34:41.553141 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:41.553114 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" podUID="a81b2492-7d46-44ac-a92b-7cd698769695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 16:34:42.551567 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:42.551132 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 16:34:42.551567 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:42.551191 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" podUID="a81b2492-7d46-44ac-a92b-7cd698769695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 16:34:43.303524 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:43.303473 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:43.553849 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:43.550158 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 16:34:43.553849 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:43.550224 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" podUID="a81b2492-7d46-44ac-a92b-7cd698769695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 16:34:44.095905 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:34:44.095868 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:34:44.564920 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:44.564819 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:44.566221 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:44.565150 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 16:34:44.566221 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:44.565577 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" podUID="a81b2492-7d46-44ac-a92b-7cd698769695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 16:34:45.101626 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:45.101582 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:45.481205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:45.481122 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:45.550211 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:45.550174 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 16:34:45.550508 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:45.550475 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" podUID="a81b2492-7d46-44ac-a92b-7cd698769695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 16:34:45.837376 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:45.837340 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:46.169829 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:46.169695 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:46.294284 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:46.294231 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:46.413290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:46.413248 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:46.581735 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:46.581706 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:46.738566 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:46.738524 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:46.806530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:46.806489 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:46.807833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:46.807810 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:46.808590 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:46.808569 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 16:34:46.861588 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:46.861497 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:48.767971 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:48.767931 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:50.373084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:50.373048 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:51.928259 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:51.928223 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:52.115107 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:52.115076 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:52.589261 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:52.589219 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:52.698436 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:52.698393 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:52.819115 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:52.819079 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:52.939665 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:52.939533 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:53.995471 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:53.995425 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:55.540006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:55.539963 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:57.264060 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:57.264014 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:58.818279 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:58.818228 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:59.835505 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:59.835470 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8"] Oct 30 16:34:59.839049 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:59.839020 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:34:59.868459 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:59.868413 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d693d0\"/\"default-dockercfg-xznkg\"" Oct 30 16:34:59.868615 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:59.868577 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d693d0\"/\"simulator-w6-d693d0-kserve-self-signed-certs\"" Oct 30 16:34:59.869307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:59.869287 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-d693d0\"/\"kube-root-ca.crt\"" Oct 30 16:34:59.869721 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:59.869288 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-d693d0\"/\"openshift-service-ca.crt\"" Oct 30 16:34:59.871541 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:59.871520 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8"] Oct 30 16:34:59.945974 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:59.945937 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:59.951216 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:59.951188 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-dshm\") pod \"simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:34:59.951330 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:59.951233 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-home\") pod \"simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:34:59.951330 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:59.951258 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-model-cache\") pod \"simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:34:59.951430 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:59.951406 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-tmp-dir\") pod \"simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:34:59.951504 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:59.951484 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zg24\" (UniqueName: \"kubernetes.io/projected/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-kube-api-access-4zg24\") pod \"simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:34:59.951592 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:34:59.951567 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-tls-certs\") pod \"simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:35:00.052168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:00.052133 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-tmp-dir\") pod \"simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:35:00.052360 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:00.052196 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zg24\" (UniqueName: \"kubernetes.io/projected/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-kube-api-access-4zg24\") pod \"simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:35:00.052360 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:00.052222 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-tls-certs\") pod \"simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:35:00.052360 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:00.052295 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-dshm\") pod \"simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:35:00.052360 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:00.052330 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-home\") pod \"simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:35:00.052360 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:00.052354 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-model-cache\") pod \"simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:35:00.052754 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:00.052711 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-tmp-dir\") pod \"simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:35:00.052754 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:00.052724 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-home\") pod \"simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:35:00.052928 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:00.052904 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-model-cache\") pod \"simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:35:00.054897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:00.054873 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-dshm\") pod \"simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:35:00.055059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:00.055038 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-tls-certs\") pod \"simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:35:00.076906 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:00.076874 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zg24\" (UniqueName: \"kubernetes.io/projected/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-kube-api-access-4zg24\") pod \"simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:35:00.175246 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:00.175162 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:35:00.415192 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:00.415155 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:00.476687 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:00.476627 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8"] Oct 30 16:35:00.479106 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:35:00.479078 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a27d606_60b7_43e4_b8ec_3ca9bb95a51c.slice/crio-1f7f5e3740dea46b86beb1afa71fc7b9b909137d9a8ed30ba6f28b7895f75eac WatchSource:0}: Error finding container 1f7f5e3740dea46b86beb1afa71fc7b9b909137d9a8ed30ba6f28b7895f75eac: Status 404 returned error can't find the container with id 1f7f5e3740dea46b86beb1afa71fc7b9b909137d9a8ed30ba6f28b7895f75eac Oct 30 16:35:00.491903 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:00.491867 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" event={"ID":"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c","Type":"ContainerStarted","Data":"1f7f5e3740dea46b86beb1afa71fc7b9b909137d9a8ed30ba6f28b7895f75eac"} Oct 30 16:35:01.135831 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:01.135749 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:01.499693 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:01.499605 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" event={"ID":"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c","Type":"ContainerStarted","Data":"6e44b6d8fb7bd1165a810c718a9e100b2abbc0ab36e99d9f7ddc2f0d91749ba0"} Oct 30 16:35:01.542455 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:01.542402 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podStartSLOduration=2.542384182 podStartE2EDuration="2.542384182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:01.54031195 +0000 UTC m=+815.250766695" watchObservedRunningTime="2026-10-30 16:35:01.542384182 +0000 UTC m=+815.252839004" Oct 30 16:35:02.077025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:02.076986 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:02.175940 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:02.175903 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:35:03.676488 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:03.676454 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:05.325869 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:05.325835 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:07.197353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:07.197317 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:07.570152 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:07.570070 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:08.961414 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:08.961383 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:10.175443 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:10.175405 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:35:10.188631 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:10.188606 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:35:10.561976 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:10.561903 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:35:10.583928 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:10.583902 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:10.990318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:10.990281 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94"] Oct 30 16:35:10.993968 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:10.993948 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.010245 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.010224 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-0b2348\"/\"openshift-service-ca.crt\"" Oct 30 16:35:11.010422 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.010401 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-0b2348\"/\"kube-root-ca.crt\"" Oct 30 16:35:11.022158 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.022135 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0b2348\"/\"default-dockercfg-j8sz5\"" Oct 30 16:35:11.022330 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.022208 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0b2348\"/\"premium-w3-0b2348-kserve-self-signed-certs\"" Oct 30 16:35:11.024229 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.024207 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94"] Oct 30 16:35:11.046516 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.046475 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:11.174819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.174779 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-tmp-dir\") pod \"premium-w3-0b2348-kserve-7c87b8c748-qnd94\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.175023 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.174914 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-dshm\") pod \"premium-w3-0b2348-kserve-7c87b8c748-qnd94\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.175023 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.175014 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-home\") pod \"premium-w3-0b2348-kserve-7c87b8c748-qnd94\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.175192 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.175081 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9b2m\" (UniqueName: \"kubernetes.io/projected/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-kube-api-access-s9b2m\") pod \"premium-w3-0b2348-kserve-7c87b8c748-qnd94\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.175192 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.175134 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-tls-certs\") pod \"premium-w3-0b2348-kserve-7c87b8c748-qnd94\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.175192 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.175176 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-model-cache\") pod \"premium-w3-0b2348-kserve-7c87b8c748-qnd94\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.276850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.276738 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-home\") pod \"premium-w3-0b2348-kserve-7c87b8c748-qnd94\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.277313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.276892 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9b2m\" (UniqueName: \"kubernetes.io/projected/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-kube-api-access-s9b2m\") pod \"premium-w3-0b2348-kserve-7c87b8c748-qnd94\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.277313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.276942 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-tls-certs\") pod \"premium-w3-0b2348-kserve-7c87b8c748-qnd94\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.277313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.276977 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-model-cache\") pod \"premium-w3-0b2348-kserve-7c87b8c748-qnd94\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.277313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.277044 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-tmp-dir\") pod \"premium-w3-0b2348-kserve-7c87b8c748-qnd94\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.277313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.277123 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-dshm\") pod \"premium-w3-0b2348-kserve-7c87b8c748-qnd94\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.277313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.277206 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-home\") pod \"premium-w3-0b2348-kserve-7c87b8c748-qnd94\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.277580 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.277318 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-model-cache\") pod \"premium-w3-0b2348-kserve-7c87b8c748-qnd94\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.277580 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.277402 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-tmp-dir\") pod \"premium-w3-0b2348-kserve-7c87b8c748-qnd94\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.279586 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.279562 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-dshm\") pod \"premium-w3-0b2348-kserve-7c87b8c748-qnd94\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.280059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.280040 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-tls-certs\") pod \"premium-w3-0b2348-kserve-7c87b8c748-qnd94\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.303113 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.303083 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9b2m\" (UniqueName: \"kubernetes.io/projected/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-kube-api-access-s9b2m\") pod \"premium-w3-0b2348-kserve-7c87b8c748-qnd94\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.306802 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.306786 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:11.604898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:11.604872 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94"] Oct 30 16:35:11.606605 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:35:11.606580 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdb161b5_f964_4fc7_95cd_cc17db3b0fbb.slice/crio-6d3794c900c6e903deba68570ec573f214e4d25fcb2348f9a4837c03741c5af5 WatchSource:0}: Error finding container 6d3794c900c6e903deba68570ec573f214e4d25fcb2348f9a4837c03741c5af5: Status 404 returned error can't find the container with id 6d3794c900c6e903deba68570ec573f214e4d25fcb2348f9a4837c03741c5af5 Oct 30 16:35:12.254219 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:12.254180 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:12.564589 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:12.564501 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" event={"ID":"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb","Type":"ContainerStarted","Data":"ce91b8ee3cf5aabf314dff53a5b7fd5fbe1d2d31f675085584b64a0fe59d4db6"} Oct 30 16:35:12.564589 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:12.564553 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" event={"ID":"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb","Type":"ContainerStarted","Data":"6d3794c900c6e903deba68570ec573f214e4d25fcb2348f9a4837c03741c5af5"} Oct 30 16:35:13.307696 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:13.307657 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:13.963690 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:13.963643 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:14.070482 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:35:14.070441 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:35:15.582167 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:15.582124 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:17.344435 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:17.344340 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:18.960916 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:18.960867 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:20.571736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.571702 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:20.611918 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.611516 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podStartSLOduration=10.611496494 podStartE2EDuration="10.611496494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:12.607176442 +0000 UTC m=+826.317631186" watchObservedRunningTime="2026-10-30 16:35:20.611496494 +0000 UTC m=+834.321951242" Oct 30 16:35:20.612660 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.612630 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9"] Oct 30 16:35:20.617498 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.617471 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.644058 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.644031 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3e2f96\"/\"default-dockercfg-vbbmf\"" Oct 30 16:35:20.644362 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.644341 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3e2f96\"/\"openshift-service-ca.crt\"" Oct 30 16:35:20.644464 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.644372 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3e2f96\"/\"premium-w0-3e2f96-kserve-self-signed-certs\"" Oct 30 16:35:20.644532 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.644513 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3e2f96\"/\"kube-root-ca.crt\"" Oct 30 16:35:20.647014 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.646985 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9"] Oct 30 16:35:20.669754 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.669729 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-model-cache\") pod \"premium-w0-3e2f96-kserve-7df5689c5f-p69q9\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.669882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.669800 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-home\") pod \"premium-w0-3e2f96-kserve-7df5689c5f-p69q9\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.670047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.670023 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnjss\" (UniqueName: \"kubernetes.io/projected/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-kube-api-access-tnjss\") pod \"premium-w0-3e2f96-kserve-7df5689c5f-p69q9\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.670147 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.670127 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-tmp-dir\") pod \"premium-w0-3e2f96-kserve-7df5689c5f-p69q9\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.670224 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.670206 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-tls-certs\") pod \"premium-w0-3e2f96-kserve-7df5689c5f-p69q9\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.670329 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.670314 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-dshm\") pod \"premium-w0-3e2f96-kserve-7df5689c5f-p69q9\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.694256 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.694228 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:20.709929 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.709902 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88"] Oct 30 16:35:20.713911 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.713895 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88" Oct 30 16:35:20.732378 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.732357 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:35:20.732378 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.732370 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:35:20.732628 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.732612 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:35:20.732799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.732785 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:35:20.733668 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.733638 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88"] Oct 30 16:35:20.771263 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.771228 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/092b952e-c47c-4ee9-9af6-29aca8023e0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-322ea210-dcc88\" (UID: \"092b952e-c47c-4ee9-9af6-29aca8023e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88" Oct 30 16:35:20.771263 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.771264 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-dshm\") pod \"premium-w0-3e2f96-kserve-7df5689c5f-p69q9\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.771495 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.771295 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-model-cache\") pod \"premium-w0-3e2f96-kserve-7df5689c5f-p69q9\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.771495 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.771364 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-home\") pod \"premium-w0-3e2f96-kserve-7df5689c5f-p69q9\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.771495 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.771475 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnjss\" (UniqueName: \"kubernetes.io/projected/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-kube-api-access-tnjss\") pod \"premium-w0-3e2f96-kserve-7df5689c5f-p69q9\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.771715 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.771514 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stqbx\" (UniqueName: \"kubernetes.io/projected/092b952e-c47c-4ee9-9af6-29aca8023e0a-kube-api-access-stqbx\") pod \"maas-api-revoke-sub-322ea210-dcc88\" (UID: \"092b952e-c47c-4ee9-9af6-29aca8023e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88" Oct 30 16:35:20.771715 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.771573 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/092b952e-c47c-4ee9-9af6-29aca8023e0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-322ea210-dcc88\" (UID: \"092b952e-c47c-4ee9-9af6-29aca8023e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88" Oct 30 16:35:20.771715 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.771608 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-tmp-dir\") pod \"premium-w0-3e2f96-kserve-7df5689c5f-p69q9\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.771892 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.771736 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-model-cache\") pod \"premium-w0-3e2f96-kserve-7df5689c5f-p69q9\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.771892 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.771756 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-tls-certs\") pod \"premium-w0-3e2f96-kserve-7df5689c5f-p69q9\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.771892 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.771804 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-home\") pod \"premium-w0-3e2f96-kserve-7df5689c5f-p69q9\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.772078 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.772055 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-tmp-dir\") pod \"premium-w0-3e2f96-kserve-7df5689c5f-p69q9\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.773673 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.773651 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-dshm\") pod \"premium-w0-3e2f96-kserve-7df5689c5f-p69q9\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.774983 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.774963 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-tls-certs\") pod \"premium-w0-3e2f96-kserve-7df5689c5f-p69q9\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.796707 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.796679 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnjss\" (UniqueName: \"kubernetes.io/projected/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-kube-api-access-tnjss\") pod \"premium-w0-3e2f96-kserve-7df5689c5f-p69q9\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:20.872725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.872632 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/092b952e-c47c-4ee9-9af6-29aca8023e0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-322ea210-dcc88\" (UID: \"092b952e-c47c-4ee9-9af6-29aca8023e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88" Oct 30 16:35:20.872725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.872712 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stqbx\" (UniqueName: \"kubernetes.io/projected/092b952e-c47c-4ee9-9af6-29aca8023e0a-kube-api-access-stqbx\") pod \"maas-api-revoke-sub-322ea210-dcc88\" (UID: \"092b952e-c47c-4ee9-9af6-29aca8023e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88" Oct 30 16:35:20.872979 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.872739 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/092b952e-c47c-4ee9-9af6-29aca8023e0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-322ea210-dcc88\" (UID: \"092b952e-c47c-4ee9-9af6-29aca8023e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88" Oct 30 16:35:20.873350 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.873326 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/092b952e-c47c-4ee9-9af6-29aca8023e0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-322ea210-dcc88\" (UID: \"092b952e-c47c-4ee9-9af6-29aca8023e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88" Oct 30 16:35:20.875525 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.875498 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/092b952e-c47c-4ee9-9af6-29aca8023e0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-322ea210-dcc88\" (UID: \"092b952e-c47c-4ee9-9af6-29aca8023e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88" Oct 30 16:35:20.897280 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.897257 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stqbx\" (UniqueName: \"kubernetes.io/projected/092b952e-c47c-4ee9-9af6-29aca8023e0a-kube-api-access-stqbx\") pod \"maas-api-revoke-sub-322ea210-dcc88\" (UID: \"092b952e-c47c-4ee9-9af6-29aca8023e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88" Oct 30 16:35:20.935045 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:20.935016 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:21.027443 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:21.027416 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88" Oct 30 16:35:21.229673 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:21.229647 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9"] Oct 30 16:35:21.230483 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:35:21.230455 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cbb0ec9_d73e_4b74_bce1_15b919acb9b4.slice/crio-05ddf959af12446be02bde15d96b8d9d0c1fe74171b03b2ac031456f837f893e WatchSource:0}: Error finding container 05ddf959af12446be02bde15d96b8d9d0c1fe74171b03b2ac031456f837f893e: Status 404 returned error can't find the container with id 05ddf959af12446be02bde15d96b8d9d0c1fe74171b03b2ac031456f837f893e Oct 30 16:35:21.307179 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:21.307149 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:21.320930 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:21.320902 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:21.332894 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:21.332872 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88"] Oct 30 16:35:21.334816 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:35:21.334756 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092b952e_c47c_4ee9_9af6_29aca8023e0a.slice/crio-5042be24f1335be740dc779565a5191213c5c05201f4cb8e3322ec2c9f6dd9b3 WatchSource:0}: Error finding container 5042be24f1335be740dc779565a5191213c5c05201f4cb8e3322ec2c9f6dd9b3: Status 404 returned error can't find the container with id 5042be24f1335be740dc779565a5191213c5c05201f4cb8e3322ec2c9f6dd9b3 Oct 30 16:35:21.621267 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:21.621217 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88" event={"ID":"092b952e-c47c-4ee9-9af6-29aca8023e0a","Type":"ContainerStarted","Data":"5042be24f1335be740dc779565a5191213c5c05201f4cb8e3322ec2c9f6dd9b3"} Oct 30 16:35:21.623007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:21.622981 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" event={"ID":"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4","Type":"ContainerStarted","Data":"cccf30eb422850080fab1fb40248b883b0e865f4c1983cf9c69298e04715e9fe"} Oct 30 16:35:21.623147 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:21.623016 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" event={"ID":"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4","Type":"ContainerStarted","Data":"05ddf959af12446be02bde15d96b8d9d0c1fe74171b03b2ac031456f837f893e"} Oct 30 16:35:21.634903 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:21.634879 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:35:21.665897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:21.665847 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podStartSLOduration=1.665833838 podStartE2EDuration="1.665833838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:21.664506274 +0000 UTC m=+835.374961018" watchObservedRunningTime="2026-10-30 16:35:21.665833838 +0000 UTC m=+835.376288581" Oct 30 16:35:21.935553 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:21.935468 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:22.188534 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:22.188452 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:23.699822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:23.699756 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:24.647827 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:24.647785 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88" event={"ID":"092b952e-c47c-4ee9-9af6-29aca8023e0a","Type":"ContainerStarted","Data":"2ea8c89386266f67a58d7eea217e87f68e1b762e1665ef5c56d4049daef08de0"} Oct 30 16:35:24.689296 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:24.689248 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88" podStartSLOduration=2.190762313 podStartE2EDuration="4.689230305s" totalImagesPullingTime="2.498467992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:24.687961394 +0000 UTC m=+838.398416141" watchObservedRunningTime="2026-10-30 16:35:24.689230305 +0000 UTC m=+838.399685050" Oct 30 16:35:25.482018 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:25.481976 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:25.658863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:25.658828 6202 generic.go:356] "Generic (PLEG): container finished" podID="092b952e-c47c-4ee9-9af6-29aca8023e0a" containerID="2ea8c89386266f67a58d7eea217e87f68e1b762e1665ef5c56d4049daef08de0" exitCode=0 Oct 30 16:35:25.659040 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:25.658911 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88" event={"ID":"092b952e-c47c-4ee9-9af6-29aca8023e0a","Type":"ContainerDied","Data":"2ea8c89386266f67a58d7eea217e87f68e1b762e1665ef5c56d4049daef08de0"} Oct 30 16:35:25.928000 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:25.927964 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:26.845993 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:26.845967 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88" Oct 30 16:35:26.948866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:26.947667 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/092b952e-c47c-4ee9-9af6-29aca8023e0a-kube-api-access-stqbx\" (UniqueName: \"kubernetes.io/projected/092b952e-c47c-4ee9-9af6-29aca8023e0a-kube-api-access-stqbx\") pod \"092b952e-c47c-4ee9-9af6-29aca8023e0a\" (UID: \"092b952e-c47c-4ee9-9af6-29aca8023e0a\") " Oct 30 16:35:26.948866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:26.947845 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/092b952e-c47c-4ee9-9af6-29aca8023e0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/092b952e-c47c-4ee9-9af6-29aca8023e0a-maas-api-xks-ca\") pod \"092b952e-c47c-4ee9-9af6-29aca8023e0a\" (UID: \"092b952e-c47c-4ee9-9af6-29aca8023e0a\") " Oct 30 16:35:26.948866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:26.947895 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/092b952e-c47c-4ee9-9af6-29aca8023e0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/092b952e-c47c-4ee9-9af6-29aca8023e0a-maas-api-openshift-ca\") pod \"092b952e-c47c-4ee9-9af6-29aca8023e0a\" (UID: \"092b952e-c47c-4ee9-9af6-29aca8023e0a\") " Oct 30 16:35:26.948866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:26.948604 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092b952e-c47c-4ee9-9af6-29aca8023e0a-maas-api-openshift-ca" pod "092b952e-c47c-4ee9-9af6-29aca8023e0a" (UID: "092b952e-c47c-4ee9-9af6-29aca8023e0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:35:26.951821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:26.951796 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092b952e-c47c-4ee9-9af6-29aca8023e0a-kube-api-access-stqbx" pod "092b952e-c47c-4ee9-9af6-29aca8023e0a" (UID: "092b952e-c47c-4ee9-9af6-29aca8023e0a"). InnerVolumeSpecName "kube-api-access-stqbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:35:26.968047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:26.967980 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/092b952e-c47c-4ee9-9af6-29aca8023e0a-maas-api-xks-ca" pod "092b952e-c47c-4ee9-9af6-29aca8023e0a" (UID: "092b952e-c47c-4ee9-9af6-29aca8023e0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:35:27.058573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:27.058526 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stqbx\" (UniqueName: \"kubernetes.io/projected/092b952e-c47c-4ee9-9af6-29aca8023e0a-kube-api-access-stqbx\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:35:27.058573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:27.058569 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/092b952e-c47c-4ee9-9af6-29aca8023e0a-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:35:27.058892 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:27.058585 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/092b952e-c47c-4ee9-9af6-29aca8023e0a-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:35:27.180794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:27.180654 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:27.672889 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:27.672851 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88" event={"ID":"092b952e-c47c-4ee9-9af6-29aca8023e0a","Type":"ContainerDied","Data":"5042be24f1335be740dc779565a5191213c5c05201f4cb8e3322ec2c9f6dd9b3"} Oct 30 16:35:27.673066 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:27.672930 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5042be24f1335be740dc779565a5191213c5c05201f4cb8e3322ec2c9f6dd9b3" Oct 30 16:35:27.673066 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:27.672861 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88" Oct 30 16:35:28.791009 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:28.790962 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:30.385244 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:30.385209 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:30.935746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:30.935716 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:30.951343 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:30.951308 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:31.708280 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:31.708246 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:35:31.974288 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:31.974181 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:33.426467 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.426429 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m"] Oct 30 16:35:33.427119 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.427103 6202 cpu_manager.go:414] "containerMap: removing container" podUID="092b952e-c47c-4ee9-9af6-29aca8023e0a" containerName="revoke-keys" Oct 30 16:35:33.427177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.427121 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="092b952e-c47c-4ee9-9af6-29aca8023e0a" containerName="revoke-keys" Oct 30 16:35:33.427212 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.427195 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="092b952e-c47c-4ee9-9af6-29aca8023e0a" containerName="revoke-keys" Oct 30 16:35:33.431507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.431493 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.446975 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.446949 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-0b2348-openshift-default-dockercfg-86t2v\"" Oct 30 16:35:33.469453 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.469429 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m"] Oct 30 16:35:33.534869 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.534819 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.534869 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.534861 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr95h\" (UniqueName: \"kubernetes.io/projected/05d8ddc3-7f1c-4c56-a906-3722b011ab90-kube-api-access-vr95h\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.535137 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.534894 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.535137 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.534954 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.535457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.535416 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.535566 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.535548 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.535629 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.535598 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.535920 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.535893 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.536144 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.536123 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.637258 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.637226 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.637487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.637291 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.637487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.637317 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.637487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.637338 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.637487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.637381 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.637487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.637437 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.637487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.637458 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr95h\" (UniqueName: \"kubernetes.io/projected/05d8ddc3-7f1c-4c56-a906-3722b011ab90-kube-api-access-vr95h\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.637806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.637695 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.637806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.637721 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.637923 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.637851 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.637923 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.637875 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.638038 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.638019 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.638252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.638200 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.638252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.638281 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.639967 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.639928 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.640075 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.640021 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.666627 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.666594 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr95h\" (UniqueName: \"kubernetes.io/projected/05d8ddc3-7f1c-4c56-a906-3722b011ab90-kube-api-access-vr95h\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.667267 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.667243 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:33.688717 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.688638 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:33.744178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:33.744138 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:34.059900 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:34.059870 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m"] Oct 30 16:35:34.060016 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:35:34.059869 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d8ddc3_7f1c_4c56_a906_3722b011ab90.slice/crio-8fdceb121748b5937659e3209b3cac789399b05b422a1535a5ff17e28b40062c WatchSource:0}: Error finding container 8fdceb121748b5937659e3209b3cac789399b05b422a1535a5ff17e28b40062c: Status 404 returned error can't find the container with id 8fdceb121748b5937659e3209b3cac789399b05b422a1535a5ff17e28b40062c Oct 30 16:35:34.062353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:34.062321 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:35:34.062455 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:34.062385 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:35:34.062455 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:34.062412 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:35:34.728506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:34.728463 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" event={"ID":"05d8ddc3-7f1c-4c56-a906-3722b011ab90","Type":"ContainerStarted","Data":"a9a7e14e8000b40aab2c3757c983b7dc0e74029fe2d759aca7b9f39db3448792"} Oct 30 16:35:34.729059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:34.728546 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" event={"ID":"05d8ddc3-7f1c-4c56-a906-3722b011ab90","Type":"ContainerStarted","Data":"8fdceb121748b5937659e3209b3cac789399b05b422a1535a5ff17e28b40062c"} Oct 30 16:35:34.752792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:34.750870 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:34.813549 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:34.813475 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" podStartSLOduration=1.813453729 podStartE2EDuration="1.813453729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:34.801509052 +0000 UTC m=+848.511963796" watchObservedRunningTime="2026-10-30 16:35:34.813453729 +0000 UTC m=+848.523908484" Oct 30 16:35:35.470586 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:35.470548 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:35.751853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:35.746881 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 16:35:35.751853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:35.746957 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" podUID="05d8ddc3-7f1c-4c56-a906-3722b011ab90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 16:35:36.406417 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.406385 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:36.668454 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.668374 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd"] Oct 30 16:35:36.673378 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.673352 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.687133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.687108 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd"] Oct 30 16:35:36.697304 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.697282 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d693d0\"/\"premium-w6-d693d0-kserve-self-signed-certs\"" Oct 30 16:35:36.729639 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.729611 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:36.765266 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.765232 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 16:35:36.765856 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.765289 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" podUID="05d8ddc3-7f1c-4c56-a906-3722b011ab90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 16:35:36.790898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.790870 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-home\") pod \"premium-w6-d693d0-kserve-787f95c794-xxrkd\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.791046 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.790902 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-dshm\") pod \"premium-w6-d693d0-kserve-787f95c794-xxrkd\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.791046 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.791026 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-model-cache\") pod \"premium-w6-d693d0-kserve-787f95c794-xxrkd\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.791145 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.791086 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f04b04a7-bcaf-43da-9379-eed11564c668-tls-certs\") pod \"premium-w6-d693d0-kserve-787f95c794-xxrkd\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.791208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.791186 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-tmp-dir\") pod \"premium-w6-d693d0-kserve-787f95c794-xxrkd\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.791250 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.791217 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvmbc\" (UniqueName: \"kubernetes.io/projected/f04b04a7-bcaf-43da-9379-eed11564c668-kube-api-access-dvmbc\") pod \"premium-w6-d693d0-kserve-787f95c794-xxrkd\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.892955 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.892907 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-home\") pod \"premium-w6-d693d0-kserve-787f95c794-xxrkd\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.893683 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.893658 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-dshm\") pod \"premium-w6-d693d0-kserve-787f95c794-xxrkd\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.894628 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.894608 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-model-cache\") pod \"premium-w6-d693d0-kserve-787f95c794-xxrkd\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.894974 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.894955 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f04b04a7-bcaf-43da-9379-eed11564c668-tls-certs\") pod \"premium-w6-d693d0-kserve-787f95c794-xxrkd\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.895818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.895786 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-home\") pod \"premium-w6-d693d0-kserve-787f95c794-xxrkd\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.896054 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.895961 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-tmp-dir\") pod \"premium-w6-d693d0-kserve-787f95c794-xxrkd\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.896214 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.896195 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvmbc\" (UniqueName: \"kubernetes.io/projected/f04b04a7-bcaf-43da-9379-eed11564c668-kube-api-access-dvmbc\") pod \"premium-w6-d693d0-kserve-787f95c794-xxrkd\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.898359 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.897047 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-model-cache\") pod \"premium-w6-d693d0-kserve-787f95c794-xxrkd\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.899274 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.899254 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-tmp-dir\") pod \"premium-w6-d693d0-kserve-787f95c794-xxrkd\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.904804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.904781 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f04b04a7-bcaf-43da-9379-eed11564c668-tls-certs\") pod \"premium-w6-d693d0-kserve-787f95c794-xxrkd\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.905502 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.904991 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-dshm\") pod \"premium-w6-d693d0-kserve-787f95c794-xxrkd\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.920270 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.920213 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvmbc\" (UniqueName: \"kubernetes.io/projected/f04b04a7-bcaf-43da-9379-eed11564c668-kube-api-access-dvmbc\") pod \"premium-w6-d693d0-kserve-787f95c794-xxrkd\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:36.987191 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:36.987161 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:37.171314 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:37.171226 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:37.330508 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:37.330479 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd"] Oct 30 16:35:37.331624 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:35:37.331597 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf04b04a7_bcaf_43da_9379_eed11564c668.slice/crio-f80d9f002a4eb65273285a20cd40bb514c0e117a8e2bd0d231b3bc164d4bebb1 WatchSource:0}: Error finding container f80d9f002a4eb65273285a20cd40bb514c0e117a8e2bd0d231b3bc164d4bebb1: Status 404 returned error can't find the container with id f80d9f002a4eb65273285a20cd40bb514c0e117a8e2bd0d231b3bc164d4bebb1 Oct 30 16:35:37.744985 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:37.744954 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 16:35:37.745234 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:37.745209 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" podUID="05d8ddc3-7f1c-4c56-a906-3722b011ab90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 16:35:37.776710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:37.776230 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" event={"ID":"f04b04a7-bcaf-43da-9379-eed11564c668","Type":"ContainerStarted","Data":"4b5cc4e94f89ebfcea44f54ca22fc470dc0b551645f369cb543c966e5e6abb70"} Oct 30 16:35:37.776710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:37.776272 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" event={"ID":"f04b04a7-bcaf-43da-9379-eed11564c668","Type":"ContainerStarted","Data":"f80d9f002a4eb65273285a20cd40bb514c0e117a8e2bd0d231b3bc164d4bebb1"} Oct 30 16:35:37.822876 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:37.822827 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podStartSLOduration=1.822813674 podStartE2EDuration="1.822813674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:37.821578622 +0000 UTC m=+851.532033366" watchObservedRunningTime="2026-10-30 16:35:37.822813674 +0000 UTC m=+851.533268417" Oct 30 16:35:37.987952 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:37.987896 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:38.759791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:38.753342 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 16:35:38.759791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:38.753442 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" podUID="05d8ddc3-7f1c-4c56-a906-3722b011ab90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 16:35:38.823166 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:38.823134 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:39.756791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:39.748129 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 16:35:39.756791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:39.748191 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" podUID="05d8ddc3-7f1c-4c56-a906-3722b011ab90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 16:35:40.451710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:40.451667 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:40.918256 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:40.918214 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:40.918571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:40.918552 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:40.919503 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:40.919487 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:35:42.058510 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:42.058458 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:43.752959 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:43.752898 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:43.924994 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:43.924943 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:44.120342 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:35:44.120251 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:35:45.377817 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:45.377784 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:46.988194 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:46.988154 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:47.001892 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:47.001854 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:47.063482 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:47.063440 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:47.851852 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:47.851819 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:35:48.538513 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.538478 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m"] Oct 30 16:35:48.542822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.542800 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m" Oct 30 16:35:48.558022 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.557998 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:35:48.569945 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.569923 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:35:48.569945 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.569934 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:35:48.570084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.569995 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:35:48.577546 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.577525 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m"] Oct 30 16:35:48.663355 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.663327 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74e99d6c-d7a8-446b-a852-6e2c62cc0016-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b29a217e-m5s8m\" (UID: \"74e99d6c-d7a8-446b-a852-6e2c62cc0016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m" Oct 30 16:35:48.663517 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.663495 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74e99d6c-d7a8-446b-a852-6e2c62cc0016-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b29a217e-m5s8m\" (UID: \"74e99d6c-d7a8-446b-a852-6e2c62cc0016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m" Oct 30 16:35:48.663560 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.663518 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptdj2\" (UniqueName: \"kubernetes.io/projected/74e99d6c-d7a8-446b-a852-6e2c62cc0016-kube-api-access-ptdj2\") pod \"maas-api-revoke-sub-b29a217e-m5s8m\" (UID: \"74e99d6c-d7a8-446b-a852-6e2c62cc0016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m" Oct 30 16:35:48.739257 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.739228 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:48.764863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.764831 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74e99d6c-d7a8-446b-a852-6e2c62cc0016-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b29a217e-m5s8m\" (UID: \"74e99d6c-d7a8-446b-a852-6e2c62cc0016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m" Oct 30 16:35:48.765029 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.765003 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74e99d6c-d7a8-446b-a852-6e2c62cc0016-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b29a217e-m5s8m\" (UID: \"74e99d6c-d7a8-446b-a852-6e2c62cc0016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m" Oct 30 16:35:48.765092 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.765032 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptdj2\" (UniqueName: \"kubernetes.io/projected/74e99d6c-d7a8-446b-a852-6e2c62cc0016-kube-api-access-ptdj2\") pod \"maas-api-revoke-sub-b29a217e-m5s8m\" (UID: \"74e99d6c-d7a8-446b-a852-6e2c62cc0016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m" Oct 30 16:35:48.765931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.765898 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74e99d6c-d7a8-446b-a852-6e2c62cc0016-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b29a217e-m5s8m\" (UID: \"74e99d6c-d7a8-446b-a852-6e2c62cc0016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m" Oct 30 16:35:48.767481 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.767460 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74e99d6c-d7a8-446b-a852-6e2c62cc0016-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b29a217e-m5s8m\" (UID: \"74e99d6c-d7a8-446b-a852-6e2c62cc0016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m" Oct 30 16:35:48.788881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.788818 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptdj2\" (UniqueName: \"kubernetes.io/projected/74e99d6c-d7a8-446b-a852-6e2c62cc0016-kube-api-access-ptdj2\") pod \"maas-api-revoke-sub-b29a217e-m5s8m\" (UID: \"74e99d6c-d7a8-446b-a852-6e2c62cc0016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m" Oct 30 16:35:48.856916 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.856878 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m" Oct 30 16:35:48.880785 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:48.880740 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:49.153322 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:49.153296 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m"] Oct 30 16:35:49.154521 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:35:49.154490 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e99d6c_d7a8_446b_a852_6e2c62cc0016.slice/crio-3a17e3c420b48deba58a17add548263dac8ea58044d76d3f584efdae7cd34501 WatchSource:0}: Error finding container 3a17e3c420b48deba58a17add548263dac8ea58044d76d3f584efdae7cd34501: Status 404 returned error can't find the container with id 3a17e3c420b48deba58a17add548263dac8ea58044d76d3f584efdae7cd34501 Oct 30 16:35:49.854593 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:49.854543 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m" event={"ID":"74e99d6c-d7a8-446b-a852-6e2c62cc0016","Type":"ContainerStarted","Data":"5314d5936cdb7b4e11a37cfd7ac1ecb1d56f9c7a1e19530e05b5228d2a8af33e"} Oct 30 16:35:49.855049 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:49.854608 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m" event={"ID":"74e99d6c-d7a8-446b-a852-6e2c62cc0016","Type":"ContainerStarted","Data":"3a17e3c420b48deba58a17add548263dac8ea58044d76d3f584efdae7cd34501"} Oct 30 16:35:49.896680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:49.896627 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m" podStartSLOduration=1.896612321 podStartE2EDuration="1.896612321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:49.893838923 +0000 UTC m=+863.604293668" watchObservedRunningTime="2026-10-30 16:35:49.896612321 +0000 UTC m=+863.607067071" Oct 30 16:35:50.483067 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:50.483033 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:50.862117 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:50.862087 6202 generic.go:356] "Generic (PLEG): container finished" podID="74e99d6c-d7a8-446b-a852-6e2c62cc0016" containerID="5314d5936cdb7b4e11a37cfd7ac1ecb1d56f9c7a1e19530e05b5228d2a8af33e" exitCode=0 Oct 30 16:35:50.862519 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:50.862154 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m" event={"ID":"74e99d6c-d7a8-446b-a852-6e2c62cc0016","Type":"ContainerDied","Data":"5314d5936cdb7b4e11a37cfd7ac1ecb1d56f9c7a1e19530e05b5228d2a8af33e"} Oct 30 16:35:50.938593 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:50.938559 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl"] Oct 30 16:35:50.943380 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:50.943355 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl" Oct 30 16:35:50.961296 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:50.961263 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl"] Oct 30 16:35:51.087557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:51.087519 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a9d020-824f-4d23-bf54-09d28522a7f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e944294-z2pwl\" (UID: \"51a9d020-824f-4d23-bf54-09d28522a7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl" Oct 30 16:35:51.087731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:51.087577 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a9d020-824f-4d23-bf54-09d28522a7f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e944294-z2pwl\" (UID: \"51a9d020-824f-4d23-bf54-09d28522a7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl" Oct 30 16:35:51.087731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:51.087661 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5zbh\" (UniqueName: \"kubernetes.io/projected/51a9d020-824f-4d23-bf54-09d28522a7f4-kube-api-access-t5zbh\") pod \"maas-api-revoke-sub-0e944294-z2pwl\" (UID: \"51a9d020-824f-4d23-bf54-09d28522a7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl" Oct 30 16:35:51.188806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:51.188705 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a9d020-824f-4d23-bf54-09d28522a7f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e944294-z2pwl\" (UID: \"51a9d020-824f-4d23-bf54-09d28522a7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl" Oct 30 16:35:51.188806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:51.188759 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a9d020-824f-4d23-bf54-09d28522a7f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e944294-z2pwl\" (UID: \"51a9d020-824f-4d23-bf54-09d28522a7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl" Oct 30 16:35:51.188806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:51.188807 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5zbh\" (UniqueName: \"kubernetes.io/projected/51a9d020-824f-4d23-bf54-09d28522a7f4-kube-api-access-t5zbh\") pod \"maas-api-revoke-sub-0e944294-z2pwl\" (UID: \"51a9d020-824f-4d23-bf54-09d28522a7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl" Oct 30 16:35:51.189578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:51.189550 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a9d020-824f-4d23-bf54-09d28522a7f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e944294-z2pwl\" (UID: \"51a9d020-824f-4d23-bf54-09d28522a7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl" Oct 30 16:35:51.191340 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:51.191322 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a9d020-824f-4d23-bf54-09d28522a7f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e944294-z2pwl\" (UID: \"51a9d020-824f-4d23-bf54-09d28522a7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl" Oct 30 16:35:51.212615 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:51.212582 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5zbh\" (UniqueName: \"kubernetes.io/projected/51a9d020-824f-4d23-bf54-09d28522a7f4-kube-api-access-t5zbh\") pod \"maas-api-revoke-sub-0e944294-z2pwl\" (UID: \"51a9d020-824f-4d23-bf54-09d28522a7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl" Oct 30 16:35:51.259139 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:51.259105 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl" Oct 30 16:35:51.568236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:51.568205 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl"] Oct 30 16:35:51.569040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:35:51.569002 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a9d020_824f_4d23_bf54_09d28522a7f4.slice/crio-bec62d4d35f0d292c0ef23960a0319a65e9b124928d124488c62a3fe6af12e93 WatchSource:0}: Error finding container bec62d4d35f0d292c0ef23960a0319a65e9b124928d124488c62a3fe6af12e93: Status 404 returned error can't find the container with id bec62d4d35f0d292c0ef23960a0319a65e9b124928d124488c62a3fe6af12e93 Oct 30 16:35:51.871451 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:51.871423 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl" event={"ID":"51a9d020-824f-4d23-bf54-09d28522a7f4","Type":"ContainerStarted","Data":"5bba2e40d0a4c474f69f51a65492b2852653daff2910d612dab11603b963b638"} Oct 30 16:35:51.871885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:51.871476 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl" event={"ID":"51a9d020-824f-4d23-bf54-09d28522a7f4","Type":"ContainerStarted","Data":"bec62d4d35f0d292c0ef23960a0319a65e9b124928d124488c62a3fe6af12e93"} Oct 30 16:35:51.915969 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:51.915913 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl" podStartSLOduration=1.91589495 podStartE2EDuration="1.91589495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:51.914246143 +0000 UTC m=+865.624700894" watchObservedRunningTime="2026-10-30 16:35:51.91589495 +0000 UTC m=+865.626349696" Oct 30 16:35:52.015928 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:52.015906 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m" Oct 30 16:35:52.096093 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:52.096064 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74e99d6c-d7a8-446b-a852-6e2c62cc0016-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74e99d6c-d7a8-446b-a852-6e2c62cc0016-maas-api-openshift-ca\") pod \"74e99d6c-d7a8-446b-a852-6e2c62cc0016\" (UID: \"74e99d6c-d7a8-446b-a852-6e2c62cc0016\") " Oct 30 16:35:52.096273 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:52.096140 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e99d6c-d7a8-446b-a852-6e2c62cc0016-kube-api-access-ptdj2\" (UniqueName: \"kubernetes.io/projected/74e99d6c-d7a8-446b-a852-6e2c62cc0016-kube-api-access-ptdj2\") pod \"74e99d6c-d7a8-446b-a852-6e2c62cc0016\" (UID: \"74e99d6c-d7a8-446b-a852-6e2c62cc0016\") " Oct 30 16:35:52.096273 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:52.096238 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74e99d6c-d7a8-446b-a852-6e2c62cc0016-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74e99d6c-d7a8-446b-a852-6e2c62cc0016-maas-api-xks-ca\") pod \"74e99d6c-d7a8-446b-a852-6e2c62cc0016\" (UID: \"74e99d6c-d7a8-446b-a852-6e2c62cc0016\") " Oct 30 16:35:52.096486 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:52.096460 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e99d6c-d7a8-446b-a852-6e2c62cc0016-maas-api-openshift-ca" pod "74e99d6c-d7a8-446b-a852-6e2c62cc0016" (UID: "74e99d6c-d7a8-446b-a852-6e2c62cc0016"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:35:52.096662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:52.096645 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74e99d6c-d7a8-446b-a852-6e2c62cc0016-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:35:52.098416 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:52.098364 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e99d6c-d7a8-446b-a852-6e2c62cc0016-kube-api-access-ptdj2" pod "74e99d6c-d7a8-446b-a852-6e2c62cc0016" (UID: "74e99d6c-d7a8-446b-a852-6e2c62cc0016"). InnerVolumeSpecName "kube-api-access-ptdj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:35:52.098484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:52.098446 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74e99d6c-d7a8-446b-a852-6e2c62cc0016-maas-api-xks-ca" pod "74e99d6c-d7a8-446b-a852-6e2c62cc0016" (UID: "74e99d6c-d7a8-446b-a852-6e2c62cc0016"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:35:52.197384 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:52.197343 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74e99d6c-d7a8-446b-a852-6e2c62cc0016-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:35:52.197384 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:52.197376 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptdj2\" (UniqueName: \"kubernetes.io/projected/74e99d6c-d7a8-446b-a852-6e2c62cc0016-kube-api-access-ptdj2\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:35:52.306977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:52.306913 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:52.881014 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:52.880966 6202 generic.go:356] "Generic (PLEG): container finished" podID="51a9d020-824f-4d23-bf54-09d28522a7f4" containerID="5bba2e40d0a4c474f69f51a65492b2852653daff2910d612dab11603b963b638" exitCode=0 Oct 30 16:35:52.881474 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:52.881074 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl" event={"ID":"51a9d020-824f-4d23-bf54-09d28522a7f4","Type":"ContainerDied","Data":"5bba2e40d0a4c474f69f51a65492b2852653daff2910d612dab11603b963b638"} Oct 30 16:35:52.882734 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:52.882698 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m" event={"ID":"74e99d6c-d7a8-446b-a852-6e2c62cc0016","Type":"ContainerDied","Data":"3a17e3c420b48deba58a17add548263dac8ea58044d76d3f584efdae7cd34501"} Oct 30 16:35:52.882734 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:52.882715 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m" Oct 30 16:35:52.882943 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:52.882743 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a17e3c420b48deba58a17add548263dac8ea58044d76d3f584efdae7cd34501" Oct 30 16:35:54.043704 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:54.043678 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl" Oct 30 16:35:54.119617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:54.119579 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51a9d020-824f-4d23-bf54-09d28522a7f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a9d020-824f-4d23-bf54-09d28522a7f4-maas-api-xks-ca\") pod \"51a9d020-824f-4d23-bf54-09d28522a7f4\" (UID: \"51a9d020-824f-4d23-bf54-09d28522a7f4\") " Oct 30 16:35:54.119818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:54.119670 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a9d020-824f-4d23-bf54-09d28522a7f4-kube-api-access-t5zbh\" (UniqueName: \"kubernetes.io/projected/51a9d020-824f-4d23-bf54-09d28522a7f4-kube-api-access-t5zbh\") pod \"51a9d020-824f-4d23-bf54-09d28522a7f4\" (UID: \"51a9d020-824f-4d23-bf54-09d28522a7f4\") " Oct 30 16:35:54.119818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:54.119756 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51a9d020-824f-4d23-bf54-09d28522a7f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a9d020-824f-4d23-bf54-09d28522a7f4-maas-api-openshift-ca\") pod \"51a9d020-824f-4d23-bf54-09d28522a7f4\" (UID: \"51a9d020-824f-4d23-bf54-09d28522a7f4\") " Oct 30 16:35:54.120358 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:54.120326 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a9d020-824f-4d23-bf54-09d28522a7f4-maas-api-openshift-ca" pod "51a9d020-824f-4d23-bf54-09d28522a7f4" (UID: "51a9d020-824f-4d23-bf54-09d28522a7f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:35:54.120738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:54.120684 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:54.122077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:54.122055 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a9d020-824f-4d23-bf54-09d28522a7f4-kube-api-access-t5zbh" pod "51a9d020-824f-4d23-bf54-09d28522a7f4" (UID: "51a9d020-824f-4d23-bf54-09d28522a7f4"). InnerVolumeSpecName "kube-api-access-t5zbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:35:54.122627 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:54.122605 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51a9d020-824f-4d23-bf54-09d28522a7f4-maas-api-xks-ca" pod "51a9d020-824f-4d23-bf54-09d28522a7f4" (UID: "51a9d020-824f-4d23-bf54-09d28522a7f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:35:54.221744 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:54.221655 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5zbh\" (UniqueName: \"kubernetes.io/projected/51a9d020-824f-4d23-bf54-09d28522a7f4-kube-api-access-t5zbh\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:35:54.221744 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:54.221686 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a9d020-824f-4d23-bf54-09d28522a7f4-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:35:54.221744 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:54.221696 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a9d020-824f-4d23-bf54-09d28522a7f4-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:35:54.897301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:54.897277 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl" Oct 30 16:35:54.897518 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:54.897271 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl" event={"ID":"51a9d020-824f-4d23-bf54-09d28522a7f4","Type":"ContainerDied","Data":"bec62d4d35f0d292c0ef23960a0319a65e9b124928d124488c62a3fe6af12e93"} Oct 30 16:35:54.897518 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:54.897433 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bec62d4d35f0d292c0ef23960a0319a65e9b124928d124488c62a3fe6af12e93" Oct 30 16:35:55.152177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.152090 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr"] Oct 30 16:35:55.152945 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.152925 6202 cpu_manager.go:414] "containerMap: removing container" podUID="74e99d6c-d7a8-446b-a852-6e2c62cc0016" containerName="revoke-keys" Oct 30 16:35:55.153008 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.152949 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74e99d6c-d7a8-446b-a852-6e2c62cc0016" containerName="revoke-keys" Oct 30 16:35:55.153008 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.152980 6202 cpu_manager.go:414] "containerMap: removing container" podUID="51a9d020-824f-4d23-bf54-09d28522a7f4" containerName="revoke-keys" Oct 30 16:35:55.153008 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.152988 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a9d020-824f-4d23-bf54-09d28522a7f4" containerName="revoke-keys" Oct 30 16:35:55.153148 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.153134 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a9d020-824f-4d23-bf54-09d28522a7f4" containerName="revoke-keys" Oct 30 16:35:55.153180 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.153164 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="74e99d6c-d7a8-446b-a852-6e2c62cc0016" containerName="revoke-keys" Oct 30 16:35:55.157274 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.157249 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr" Oct 30 16:35:55.173478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.173445 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:35:55.173610 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.173541 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:35:55.173711 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.173693 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:35:55.201297 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.193010 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:35:55.201297 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.196795 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr"] Oct 30 16:35:55.333198 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.333151 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5fbbcce-bb92-4df9-abba-0901cee17ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e33859a-2kvtr\" (UID: \"b5fbbcce-bb92-4df9-abba-0901cee17ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr" Oct 30 16:35:55.333401 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.333359 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4xw5\" (UniqueName: \"kubernetes.io/projected/b5fbbcce-bb92-4df9-abba-0901cee17ce3-kube-api-access-n4xw5\") pod \"maas-api-revoke-sub-6e33859a-2kvtr\" (UID: \"b5fbbcce-bb92-4df9-abba-0901cee17ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr" Oct 30 16:35:55.333547 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.333429 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5fbbcce-bb92-4df9-abba-0901cee17ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e33859a-2kvtr\" (UID: \"b5fbbcce-bb92-4df9-abba-0901cee17ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr" Oct 30 16:35:55.434848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.434278 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5fbbcce-bb92-4df9-abba-0901cee17ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e33859a-2kvtr\" (UID: \"b5fbbcce-bb92-4df9-abba-0901cee17ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr" Oct 30 16:35:55.434848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.434441 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4xw5\" (UniqueName: \"kubernetes.io/projected/b5fbbcce-bb92-4df9-abba-0901cee17ce3-kube-api-access-n4xw5\") pod \"maas-api-revoke-sub-6e33859a-2kvtr\" (UID: \"b5fbbcce-bb92-4df9-abba-0901cee17ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr" Oct 30 16:35:55.434848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.434510 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5fbbcce-bb92-4df9-abba-0901cee17ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e33859a-2kvtr\" (UID: \"b5fbbcce-bb92-4df9-abba-0901cee17ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr" Oct 30 16:35:55.438792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.437899 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5fbbcce-bb92-4df9-abba-0901cee17ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e33859a-2kvtr\" (UID: \"b5fbbcce-bb92-4df9-abba-0901cee17ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr" Oct 30 16:35:55.438948 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.438847 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5fbbcce-bb92-4df9-abba-0901cee17ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e33859a-2kvtr\" (UID: \"b5fbbcce-bb92-4df9-abba-0901cee17ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr" Oct 30 16:35:55.473782 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.473732 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4xw5\" (UniqueName: \"kubernetes.io/projected/b5fbbcce-bb92-4df9-abba-0901cee17ce3-kube-api-access-n4xw5\") pod \"maas-api-revoke-sub-6e33859a-2kvtr\" (UID: \"b5fbbcce-bb92-4df9-abba-0901cee17ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr" Oct 30 16:35:55.475962 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:55.475929 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr" Oct 30 16:35:56.083047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:56.082952 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr"] Oct 30 16:35:56.084515 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:35:56.084481 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5fbbcce_bb92_4df9_abba_0901cee17ce3.slice/crio-94bd7ee462a52104293bc1c9022f58ff0a5dd0bac654b9b2cf8676501508a69f WatchSource:0}: Error finding container 94bd7ee462a52104293bc1c9022f58ff0a5dd0bac654b9b2cf8676501508a69f: Status 404 returned error can't find the container with id 94bd7ee462a52104293bc1c9022f58ff0a5dd0bac654b9b2cf8676501508a69f Oct 30 16:35:56.406067 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:56.406027 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:57.014111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:57.014076 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr" event={"ID":"b5fbbcce-bb92-4df9-abba-0901cee17ce3","Type":"ContainerStarted","Data":"c4618b80ad59ba60bb3366c5aaff0fea4dbeb28d222a18d2e741964e1495b7ba"} Oct 30 16:35:57.014302 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:57.014129 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr" event={"ID":"b5fbbcce-bb92-4df9-abba-0901cee17ce3","Type":"ContainerStarted","Data":"94bd7ee462a52104293bc1c9022f58ff0a5dd0bac654b9b2cf8676501508a69f"} Oct 30 16:35:57.056655 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:57.056594 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr" podStartSLOduration=2.056577504 podStartE2EDuration="2.056577504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:57.053333643 +0000 UTC m=+870.763788387" watchObservedRunningTime="2026-10-30 16:35:57.056577504 +0000 UTC m=+870.767032269" Oct 30 16:35:58.021312 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:58.021273 6202 generic.go:356] "Generic (PLEG): container finished" podID="b5fbbcce-bb92-4df9-abba-0901cee17ce3" containerID="c4618b80ad59ba60bb3366c5aaff0fea4dbeb28d222a18d2e741964e1495b7ba" exitCode=0 Oct 30 16:35:58.021749 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:58.021311 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr" event={"ID":"b5fbbcce-bb92-4df9-abba-0901cee17ce3","Type":"ContainerDied","Data":"c4618b80ad59ba60bb3366c5aaff0fea4dbeb28d222a18d2e741964e1495b7ba"} Oct 30 16:35:58.135693 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:58.135663 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:59.180806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:59.180778 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr" Oct 30 16:35:59.237395 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:59.237368 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5fbbcce-bb92-4df9-abba-0901cee17ce3-kube-api-access-n4xw5\" (UniqueName: \"kubernetes.io/projected/b5fbbcce-bb92-4df9-abba-0901cee17ce3-kube-api-access-n4xw5\") pod \"b5fbbcce-bb92-4df9-abba-0901cee17ce3\" (UID: \"b5fbbcce-bb92-4df9-abba-0901cee17ce3\") " Oct 30 16:35:59.237574 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:59.237421 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5fbbcce-bb92-4df9-abba-0901cee17ce3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5fbbcce-bb92-4df9-abba-0901cee17ce3-maas-api-xks-ca\") pod \"b5fbbcce-bb92-4df9-abba-0901cee17ce3\" (UID: \"b5fbbcce-bb92-4df9-abba-0901cee17ce3\") " Oct 30 16:35:59.237574 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:59.237508 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5fbbcce-bb92-4df9-abba-0901cee17ce3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5fbbcce-bb92-4df9-abba-0901cee17ce3-maas-api-openshift-ca\") pod \"b5fbbcce-bb92-4df9-abba-0901cee17ce3\" (UID: \"b5fbbcce-bb92-4df9-abba-0901cee17ce3\") " Oct 30 16:35:59.237988 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:59.237949 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5fbbcce-bb92-4df9-abba-0901cee17ce3-maas-api-openshift-ca" pod "b5fbbcce-bb92-4df9-abba-0901cee17ce3" (UID: "b5fbbcce-bb92-4df9-abba-0901cee17ce3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:35:59.239938 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:59.239906 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fbbcce-bb92-4df9-abba-0901cee17ce3-kube-api-access-n4xw5" pod "b5fbbcce-bb92-4df9-abba-0901cee17ce3" (UID: "b5fbbcce-bb92-4df9-abba-0901cee17ce3"). InnerVolumeSpecName "kube-api-access-n4xw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:35:59.240287 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:59.240253 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5fbbcce-bb92-4df9-abba-0901cee17ce3-maas-api-xks-ca" pod "b5fbbcce-bb92-4df9-abba-0901cee17ce3" (UID: "b5fbbcce-bb92-4df9-abba-0901cee17ce3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:35:59.338954 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:59.338916 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4xw5\" (UniqueName: \"kubernetes.io/projected/b5fbbcce-bb92-4df9-abba-0901cee17ce3-kube-api-access-n4xw5\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:35:59.338954 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:59.338949 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5fbbcce-bb92-4df9-abba-0901cee17ce3-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:35:59.338954 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:59.338959 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5fbbcce-bb92-4df9-abba-0901cee17ce3-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:35:59.644005 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:35:59.643910 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:00.034147 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:00.034065 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr" event={"ID":"b5fbbcce-bb92-4df9-abba-0901cee17ce3","Type":"ContainerDied","Data":"94bd7ee462a52104293bc1c9022f58ff0a5dd0bac654b9b2cf8676501508a69f"} Oct 30 16:36:00.034147 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:00.034125 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94bd7ee462a52104293bc1c9022f58ff0a5dd0bac654b9b2cf8676501508a69f" Oct 30 16:36:00.034147 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:00.034077 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr" Oct 30 16:36:00.985238 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:00.985200 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl"] Oct 30 16:36:00.985846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:00.985830 6202 cpu_manager.go:414] "containerMap: removing container" podUID="b5fbbcce-bb92-4df9-abba-0901cee17ce3" containerName="revoke-keys" Oct 30 16:36:00.985846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:00.985848 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5fbbcce-bb92-4df9-abba-0901cee17ce3" containerName="revoke-keys" Oct 30 16:36:00.985962 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:00.985938 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5fbbcce-bb92-4df9-abba-0901cee17ce3" containerName="revoke-keys" Oct 30 16:36:00.988324 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:00.988309 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl" Oct 30 16:36:01.015018 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:01.014982 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:36:01.015018 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:01.015002 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:36:01.015018 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:01.015002 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:36:01.015335 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:01.015177 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:36:01.017934 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:01.017906 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl"] Oct 30 16:36:01.057109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:01.057082 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd7a75d-3328-436a-8e21-8e8088de4079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-639bde01-lzghl\" (UID: \"cfd7a75d-3328-436a-8e21-8e8088de4079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl" Oct 30 16:36:01.057255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:01.057193 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bspmk\" (UniqueName: \"kubernetes.io/projected/cfd7a75d-3328-436a-8e21-8e8088de4079-kube-api-access-bspmk\") pod \"maas-api-revoke-sub-639bde01-lzghl\" (UID: \"cfd7a75d-3328-436a-8e21-8e8088de4079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl" Oct 30 16:36:01.057317 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:01.057275 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd7a75d-3328-436a-8e21-8e8088de4079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-639bde01-lzghl\" (UID: \"cfd7a75d-3328-436a-8e21-8e8088de4079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl" Oct 30 16:36:01.158377 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:01.158341 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd7a75d-3328-436a-8e21-8e8088de4079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-639bde01-lzghl\" (UID: \"cfd7a75d-3328-436a-8e21-8e8088de4079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl" Oct 30 16:36:01.158557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:01.158441 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bspmk\" (UniqueName: \"kubernetes.io/projected/cfd7a75d-3328-436a-8e21-8e8088de4079-kube-api-access-bspmk\") pod \"maas-api-revoke-sub-639bde01-lzghl\" (UID: \"cfd7a75d-3328-436a-8e21-8e8088de4079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl" Oct 30 16:36:01.158557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:01.158490 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd7a75d-3328-436a-8e21-8e8088de4079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-639bde01-lzghl\" (UID: \"cfd7a75d-3328-436a-8e21-8e8088de4079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl" Oct 30 16:36:01.159218 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:01.159193 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd7a75d-3328-436a-8e21-8e8088de4079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-639bde01-lzghl\" (UID: \"cfd7a75d-3328-436a-8e21-8e8088de4079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl" Oct 30 16:36:01.160872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:01.160824 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd7a75d-3328-436a-8e21-8e8088de4079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-639bde01-lzghl\" (UID: \"cfd7a75d-3328-436a-8e21-8e8088de4079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl" Oct 30 16:36:01.181987 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:01.181961 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bspmk\" (UniqueName: \"kubernetes.io/projected/cfd7a75d-3328-436a-8e21-8e8088de4079-kube-api-access-bspmk\") pod \"maas-api-revoke-sub-639bde01-lzghl\" (UID: \"cfd7a75d-3328-436a-8e21-8e8088de4079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl" Oct 30 16:36:01.289011 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:01.288909 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:01.300951 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:01.300908 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl" Oct 30 16:36:01.603661 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:01.603620 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl"] Oct 30 16:36:01.606285 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:36:01.606248 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd7a75d_3328_436a_8e21_8e8088de4079.slice/crio-518c23eb2927ea8571466feb648c127f23c80bd5a2e6bce52ac88cdcfe72624a WatchSource:0}: Error finding container 518c23eb2927ea8571466feb648c127f23c80bd5a2e6bce52ac88cdcfe72624a: Status 404 returned error can't find the container with id 518c23eb2927ea8571466feb648c127f23c80bd5a2e6bce52ac88cdcfe72624a Oct 30 16:36:02.047615 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:02.047522 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl" event={"ID":"cfd7a75d-3328-436a-8e21-8e8088de4079","Type":"ContainerStarted","Data":"ff1364fd1a559a48a01974304fd15d6eb296c095fff54d67258d29c40bf99260"} Oct 30 16:36:02.047615 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:02.047583 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl" event={"ID":"cfd7a75d-3328-436a-8e21-8e8088de4079","Type":"ContainerStarted","Data":"518c23eb2927ea8571466feb648c127f23c80bd5a2e6bce52ac88cdcfe72624a"} Oct 30 16:36:02.090549 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:02.090486 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl" podStartSLOduration=2.090471775 podStartE2EDuration="2.090471775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:02.08836099 +0000 UTC m=+875.798815734" watchObservedRunningTime="2026-10-30 16:36:02.090471775 +0000 UTC m=+875.800926520" Oct 30 16:36:02.878903 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:02.878855 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:03.055148 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.055112 6202 generic.go:356] "Generic (PLEG): container finished" podID="cfd7a75d-3328-436a-8e21-8e8088de4079" containerID="ff1364fd1a559a48a01974304fd15d6eb296c095fff54d67258d29c40bf99260" exitCode=0 Oct 30 16:36:03.055537 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.055162 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl" event={"ID":"cfd7a75d-3328-436a-8e21-8e8088de4079","Type":"ContainerDied","Data":"ff1364fd1a559a48a01974304fd15d6eb296c095fff54d67258d29c40bf99260"} Oct 30 16:36:03.798744 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.798706 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk"] Oct 30 16:36:03.801963 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.801943 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.816727 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.816706 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-967fac-openshift-default-dockercfg-lfjgc\"" Oct 30 16:36:03.831872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.831828 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk"] Oct 30 16:36:03.892368 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.892338 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.892532 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.892375 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhs8h\" (UniqueName: \"kubernetes.io/projected/92906137-4350-4921-813c-cdcaf92a2143-kube-api-access-fhs8h\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.892532 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.892404 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.892659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.892542 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.892659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.892584 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92906137-4350-4921-813c-cdcaf92a2143-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.892721 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.892663 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.892721 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.892705 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92906137-4350-4921-813c-cdcaf92a2143-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.892860 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.892821 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92906137-4350-4921-813c-cdcaf92a2143-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.892923 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.892887 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.993659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.993621 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.993659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.993666 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhs8h\" (UniqueName: \"kubernetes.io/projected/92906137-4350-4921-813c-cdcaf92a2143-kube-api-access-fhs8h\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.993932 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.993690 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.993932 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.993719 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.993932 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.993736 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92906137-4350-4921-813c-cdcaf92a2143-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.993932 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.993888 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.994144 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.993962 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92906137-4350-4921-813c-cdcaf92a2143-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.994144 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.994008 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92906137-4350-4921-813c-cdcaf92a2143-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.994144 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.994057 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.994272 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.994149 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.994272 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.994188 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.994401 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.994380 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.994452 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.994401 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.994882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.994840 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92906137-4350-4921-813c-cdcaf92a2143-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.996392 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.996367 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:03.996536 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:03.996519 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92906137-4350-4921-813c-cdcaf92a2143-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:04.017626 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:04.017597 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92906137-4350-4921-813c-cdcaf92a2143-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:04.017805 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:04.017750 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhs8h\" (UniqueName: \"kubernetes.io/projected/92906137-4350-4921-813c-cdcaf92a2143-kube-api-access-fhs8h\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:04.113975 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:04.113938 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:04.211872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:04.211847 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl" Oct 30 16:36:04.301213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:04.301168 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfd7a75d-3328-436a-8e21-8e8088de4079-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd7a75d-3328-436a-8e21-8e8088de4079-maas-api-xks-ca\") pod \"cfd7a75d-3328-436a-8e21-8e8088de4079\" (UID: \"cfd7a75d-3328-436a-8e21-8e8088de4079\") " Oct 30 16:36:04.301418 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:04.301236 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd7a75d-3328-436a-8e21-8e8088de4079-kube-api-access-bspmk\" (UniqueName: \"kubernetes.io/projected/cfd7a75d-3328-436a-8e21-8e8088de4079-kube-api-access-bspmk\") pod \"cfd7a75d-3328-436a-8e21-8e8088de4079\" (UID: \"cfd7a75d-3328-436a-8e21-8e8088de4079\") " Oct 30 16:36:04.301418 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:04.301410 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfd7a75d-3328-436a-8e21-8e8088de4079-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd7a75d-3328-436a-8e21-8e8088de4079-maas-api-openshift-ca\") pod \"cfd7a75d-3328-436a-8e21-8e8088de4079\" (UID: \"cfd7a75d-3328-436a-8e21-8e8088de4079\") " Oct 30 16:36:04.301874 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:04.301847 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd7a75d-3328-436a-8e21-8e8088de4079-maas-api-openshift-ca" pod "cfd7a75d-3328-436a-8e21-8e8088de4079" (UID: "cfd7a75d-3328-436a-8e21-8e8088de4079"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:04.303489 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:04.303467 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfd7a75d-3328-436a-8e21-8e8088de4079-maas-api-xks-ca" pod "cfd7a75d-3328-436a-8e21-8e8088de4079" (UID: "cfd7a75d-3328-436a-8e21-8e8088de4079"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:04.303489 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:04.303476 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd7a75d-3328-436a-8e21-8e8088de4079-kube-api-access-bspmk" pod "cfd7a75d-3328-436a-8e21-8e8088de4079" (UID: "cfd7a75d-3328-436a-8e21-8e8088de4079"). InnerVolumeSpecName "kube-api-access-bspmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:04.403001 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:04.402965 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfd7a75d-3328-436a-8e21-8e8088de4079-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:04.403001 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:04.402993 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfd7a75d-3328-436a-8e21-8e8088de4079-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:04.403001 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:04.403003 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bspmk\" (UniqueName: \"kubernetes.io/projected/cfd7a75d-3328-436a-8e21-8e8088de4079-kube-api-access-bspmk\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:04.410265 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:04.410230 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:04.439564 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:04.439531 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk"] Oct 30 16:36:04.440220 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:36:04.440191 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92906137_4350_4921_813c_cdcaf92a2143.slice/crio-f1669976bf6d5125c0897712297a5b2dd5bab7d5555ad27a4c78af183808165e WatchSource:0}: Error finding container f1669976bf6d5125c0897712297a5b2dd5bab7d5555ad27a4c78af183808165e: Status 404 returned error can't find the container with id f1669976bf6d5125c0897712297a5b2dd5bab7d5555ad27a4c78af183808165e Oct 30 16:36:04.442779 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:04.442731 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:04.442881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:04.442860 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:04.442937 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:04.442913 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:05.073781 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.073739 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl" event={"ID":"cfd7a75d-3328-436a-8e21-8e8088de4079","Type":"ContainerDied","Data":"518c23eb2927ea8571466feb648c127f23c80bd5a2e6bce52ac88cdcfe72624a"} Oct 30 16:36:05.073986 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.073831 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="518c23eb2927ea8571466feb648c127f23c80bd5a2e6bce52ac88cdcfe72624a" Oct 30 16:36:05.073986 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.073912 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl" Oct 30 16:36:05.076389 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.076360 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" event={"ID":"92906137-4350-4921-813c-cdcaf92a2143","Type":"ContainerStarted","Data":"778e36bef68f6300f8f67641050430f9b2aa17650583af9a7334eaa26422cde3"} Oct 30 16:36:05.076546 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.076412 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" event={"ID":"92906137-4350-4921-813c-cdcaf92a2143","Type":"ContainerStarted","Data":"f1669976bf6d5125c0897712297a5b2dd5bab7d5555ad27a4c78af183808165e"} Oct 30 16:36:05.114833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.114796 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:05.116624 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.116598 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 16:36:05.116747 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.116646 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" podUID="92906137-4350-4921-813c-cdcaf92a2143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 16:36:05.122636 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.122587 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" podStartSLOduration=2.122574753 podStartE2EDuration="2.122574753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:05.121306739 +0000 UTC m=+878.831761473" watchObservedRunningTime="2026-10-30 16:36:05.122574753 +0000 UTC m=+878.833029496" Oct 30 16:36:05.479245 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.479207 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p"] Oct 30 16:36:05.480052 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.480021 6202 cpu_manager.go:414] "containerMap: removing container" podUID="cfd7a75d-3328-436a-8e21-8e8088de4079" containerName="revoke-keys" Oct 30 16:36:05.480052 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.480047 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd7a75d-3328-436a-8e21-8e8088de4079" containerName="revoke-keys" Oct 30 16:36:05.480229 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.480217 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd7a75d-3328-436a-8e21-8e8088de4079" containerName="revoke-keys" Oct 30 16:36:05.483275 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.483255 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p" Oct 30 16:36:05.508835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.499968 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:36:05.508835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.500061 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:36:05.508835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.500109 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:36:05.508835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.500140 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:36:05.508835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.505859 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p"] Oct 30 16:36:05.617923 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.617885 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gckcm\" (UniqueName: \"kubernetes.io/projected/c2770411-8647-4b26-98dc-ab9818be3d4f-kube-api-access-gckcm\") pod \"maas-api-revoke-sub-2570c8b8-m4p4p\" (UID: \"c2770411-8647-4b26-98dc-ab9818be3d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p" Oct 30 16:36:05.617923 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.617926 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2770411-8647-4b26-98dc-ab9818be3d4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2570c8b8-m4p4p\" (UID: \"c2770411-8647-4b26-98dc-ab9818be3d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p" Oct 30 16:36:05.618141 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.618017 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2770411-8647-4b26-98dc-ab9818be3d4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2570c8b8-m4p4p\" (UID: \"c2770411-8647-4b26-98dc-ab9818be3d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p" Oct 30 16:36:05.719071 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.719021 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2770411-8647-4b26-98dc-ab9818be3d4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2570c8b8-m4p4p\" (UID: \"c2770411-8647-4b26-98dc-ab9818be3d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p" Oct 30 16:36:05.719262 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.719183 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gckcm\" (UniqueName: \"kubernetes.io/projected/c2770411-8647-4b26-98dc-ab9818be3d4f-kube-api-access-gckcm\") pod \"maas-api-revoke-sub-2570c8b8-m4p4p\" (UID: \"c2770411-8647-4b26-98dc-ab9818be3d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p" Oct 30 16:36:05.719262 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.719209 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2770411-8647-4b26-98dc-ab9818be3d4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2570c8b8-m4p4p\" (UID: \"c2770411-8647-4b26-98dc-ab9818be3d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p" Oct 30 16:36:05.719797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.719759 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2770411-8647-4b26-98dc-ab9818be3d4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2570c8b8-m4p4p\" (UID: \"c2770411-8647-4b26-98dc-ab9818be3d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p" Oct 30 16:36:05.721498 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.721475 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2770411-8647-4b26-98dc-ab9818be3d4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2570c8b8-m4p4p\" (UID: \"c2770411-8647-4b26-98dc-ab9818be3d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p" Oct 30 16:36:05.759838 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.750349 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gckcm\" (UniqueName: \"kubernetes.io/projected/c2770411-8647-4b26-98dc-ab9818be3d4f-kube-api-access-gckcm\") pod \"maas-api-revoke-sub-2570c8b8-m4p4p\" (UID: \"c2770411-8647-4b26-98dc-ab9818be3d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p" Oct 30 16:36:05.798025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:05.797985 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p" Oct 30 16:36:06.114883 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:06.114850 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 16:36:06.115299 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:06.114898 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" podUID="92906137-4350-4921-813c-cdcaf92a2143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 16:36:06.130269 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:06.130238 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p"] Oct 30 16:36:06.131823 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:36:06.131794 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2770411_8647_4b26_98dc_ab9818be3d4f.slice/crio-7ff61036f87a27897f2b7f2ee57b6f2fc18fed71fa5fe9ed78e96afda4f51753 WatchSource:0}: Error finding container 7ff61036f87a27897f2b7f2ee57b6f2fc18fed71fa5fe9ed78e96afda4f51753: Status 404 returned error can't find the container with id 7ff61036f87a27897f2b7f2ee57b6f2fc18fed71fa5fe9ed78e96afda4f51753 Oct 30 16:36:06.181450 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:06.181423 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:07.090189 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:07.090145 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p" event={"ID":"c2770411-8647-4b26-98dc-ab9818be3d4f","Type":"ContainerStarted","Data":"c178c24e1666c68a9bc9a216cecb2f867d591b93259d1ef6c56f546e641a79c1"} Oct 30 16:36:07.090400 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:07.090215 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p" event={"ID":"c2770411-8647-4b26-98dc-ab9818be3d4f","Type":"ContainerStarted","Data":"7ff61036f87a27897f2b7f2ee57b6f2fc18fed71fa5fe9ed78e96afda4f51753"} Oct 30 16:36:07.115077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:07.115044 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 16:36:07.115512 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:07.115098 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" podUID="92906137-4350-4921-813c-cdcaf92a2143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 16:36:07.131494 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:07.131438 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p" podStartSLOduration=2.131419726 podStartE2EDuration="2.131419726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:07.130039451 +0000 UTC m=+880.840494196" watchObservedRunningTime="2026-10-30 16:36:07.131419726 +0000 UTC m=+880.841874470" Oct 30 16:36:08.075522 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:08.075481 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:08.099073 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:08.099041 6202 generic.go:356] "Generic (PLEG): container finished" podID="c2770411-8647-4b26-98dc-ab9818be3d4f" containerID="c178c24e1666c68a9bc9a216cecb2f867d591b93259d1ef6c56f546e641a79c1" exitCode=0 Oct 30 16:36:08.099250 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:08.099127 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p" event={"ID":"c2770411-8647-4b26-98dc-ab9818be3d4f","Type":"ContainerDied","Data":"c178c24e1666c68a9bc9a216cecb2f867d591b93259d1ef6c56f546e641a79c1"} Oct 30 16:36:08.114343 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:08.114321 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 16:36:08.114476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:08.114364 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" podUID="92906137-4350-4921-813c-cdcaf92a2143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 16:36:09.115144 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:09.115106 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 16:36:09.115596 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:09.115167 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" podUID="92906137-4350-4921-813c-cdcaf92a2143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 16:36:09.300497 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:09.300465 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p" Oct 30 16:36:09.388924 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:09.388820 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2770411-8647-4b26-98dc-ab9818be3d4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2770411-8647-4b26-98dc-ab9818be3d4f-maas-api-openshift-ca\") pod \"c2770411-8647-4b26-98dc-ab9818be3d4f\" (UID: \"c2770411-8647-4b26-98dc-ab9818be3d4f\") " Oct 30 16:36:09.389117 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:09.388947 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2770411-8647-4b26-98dc-ab9818be3d4f-kube-api-access-gckcm\" (UniqueName: \"kubernetes.io/projected/c2770411-8647-4b26-98dc-ab9818be3d4f-kube-api-access-gckcm\") pod \"c2770411-8647-4b26-98dc-ab9818be3d4f\" (UID: \"c2770411-8647-4b26-98dc-ab9818be3d4f\") " Oct 30 16:36:09.389117 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:09.389103 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2770411-8647-4b26-98dc-ab9818be3d4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2770411-8647-4b26-98dc-ab9818be3d4f-maas-api-xks-ca\") pod \"c2770411-8647-4b26-98dc-ab9818be3d4f\" (UID: \"c2770411-8647-4b26-98dc-ab9818be3d4f\") " Oct 30 16:36:09.389285 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:09.389253 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2770411-8647-4b26-98dc-ab9818be3d4f-maas-api-openshift-ca" pod "c2770411-8647-4b26-98dc-ab9818be3d4f" (UID: "c2770411-8647-4b26-98dc-ab9818be3d4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:09.389585 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:09.389561 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2770411-8647-4b26-98dc-ab9818be3d4f-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:09.391567 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:09.391527 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2770411-8647-4b26-98dc-ab9818be3d4f-maas-api-xks-ca" pod "c2770411-8647-4b26-98dc-ab9818be3d4f" (UID: "c2770411-8647-4b26-98dc-ab9818be3d4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:09.392291 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:09.392267 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2770411-8647-4b26-98dc-ab9818be3d4f-kube-api-access-gckcm" pod "c2770411-8647-4b26-98dc-ab9818be3d4f" (UID: "c2770411-8647-4b26-98dc-ab9818be3d4f"). InnerVolumeSpecName "kube-api-access-gckcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:09.490969 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:09.490928 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gckcm\" (UniqueName: \"kubernetes.io/projected/c2770411-8647-4b26-98dc-ab9818be3d4f-kube-api-access-gckcm\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:09.490969 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:09.490962 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2770411-8647-4b26-98dc-ab9818be3d4f-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:09.885959 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:09.885922 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:10.112169 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:10.112140 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p" Oct 30 16:36:10.112325 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:10.112186 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p" event={"ID":"c2770411-8647-4b26-98dc-ab9818be3d4f","Type":"ContainerDied","Data":"7ff61036f87a27897f2b7f2ee57b6f2fc18fed71fa5fe9ed78e96afda4f51753"} Oct 30 16:36:10.112325 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:10.112259 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ff61036f87a27897f2b7f2ee57b6f2fc18fed71fa5fe9ed78e96afda4f51753" Oct 30 16:36:10.114246 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:10.114225 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 16:36:10.114330 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:10.114275 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" podUID="92906137-4350-4921-813c-cdcaf92a2143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 16:36:11.580686 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:11.580642 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:12.089427 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:12.089399 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:12.089815 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:12.089760 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:12.090551 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:12.090532 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:36:13.343714 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:13.343619 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:14.069151 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:36:14.069107 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:36:14.865694 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:14.865663 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:15.225409 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.225325 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm"] Oct 30 16:36:15.226257 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.226231 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c2770411-8647-4b26-98dc-ab9818be3d4f" containerName="revoke-keys" Oct 30 16:36:15.226257 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.226258 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2770411-8647-4b26-98dc-ab9818be3d4f" containerName="revoke-keys" Oct 30 16:36:15.226422 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.226397 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2770411-8647-4b26-98dc-ab9818be3d4f" containerName="revoke-keys" Oct 30 16:36:15.229942 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.229928 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm" Oct 30 16:36:15.258790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.257625 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:36:15.258790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.257790 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:36:15.258790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.257863 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:36:15.258790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.257972 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:36:15.294830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.293848 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm"] Oct 30 16:36:15.361400 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.361361 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3445db7b-baec-4665-b2ad-274e0c0f5da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef5b0469-jw7bm\" (UID: \"3445db7b-baec-4665-b2ad-274e0c0f5da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm" Oct 30 16:36:15.361572 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.361503 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjl48\" (UniqueName: \"kubernetes.io/projected/3445db7b-baec-4665-b2ad-274e0c0f5da1-kube-api-access-tjl48\") pod \"maas-api-revoke-sub-ef5b0469-jw7bm\" (UID: \"3445db7b-baec-4665-b2ad-274e0c0f5da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm" Oct 30 16:36:15.361620 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.361599 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3445db7b-baec-4665-b2ad-274e0c0f5da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef5b0469-jw7bm\" (UID: \"3445db7b-baec-4665-b2ad-274e0c0f5da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm" Oct 30 16:36:15.463056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.463017 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjl48\" (UniqueName: \"kubernetes.io/projected/3445db7b-baec-4665-b2ad-274e0c0f5da1-kube-api-access-tjl48\") pod \"maas-api-revoke-sub-ef5b0469-jw7bm\" (UID: \"3445db7b-baec-4665-b2ad-274e0c0f5da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm" Oct 30 16:36:15.463261 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.463088 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3445db7b-baec-4665-b2ad-274e0c0f5da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef5b0469-jw7bm\" (UID: \"3445db7b-baec-4665-b2ad-274e0c0f5da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm" Oct 30 16:36:15.463329 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.463269 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3445db7b-baec-4665-b2ad-274e0c0f5da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef5b0469-jw7bm\" (UID: \"3445db7b-baec-4665-b2ad-274e0c0f5da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm" Oct 30 16:36:15.464059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.464033 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3445db7b-baec-4665-b2ad-274e0c0f5da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef5b0469-jw7bm\" (UID: \"3445db7b-baec-4665-b2ad-274e0c0f5da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm" Oct 30 16:36:15.466086 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.466063 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3445db7b-baec-4665-b2ad-274e0c0f5da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef5b0469-jw7bm\" (UID: \"3445db7b-baec-4665-b2ad-274e0c0f5da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm" Oct 30 16:36:15.487056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.486989 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjl48\" (UniqueName: \"kubernetes.io/projected/3445db7b-baec-4665-b2ad-274e0c0f5da1-kube-api-access-tjl48\") pod \"maas-api-revoke-sub-ef5b0469-jw7bm\" (UID: \"3445db7b-baec-4665-b2ad-274e0c0f5da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm" Oct 30 16:36:15.567849 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.567819 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm" Oct 30 16:36:15.889170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:15.887309 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm"] Oct 30 16:36:16.155731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:16.155694 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm" event={"ID":"3445db7b-baec-4665-b2ad-274e0c0f5da1","Type":"ContainerStarted","Data":"a70345362f46dcfd3ffe77779ed26c2fa60986c05caea13d6ee1ad00f216aedd"} Oct 30 16:36:16.825240 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:16.825205 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:17.164916 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:17.164823 6202 generic.go:356] "Generic (PLEG): container finished" podID="3445db7b-baec-4665-b2ad-274e0c0f5da1" containerID="62b7090a4f9c2201761bd0e3e2fe34ff1ee4ad3161182b1271ee06dd8dec6514" exitCode=0 Oct 30 16:36:17.165320 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:17.164903 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm" event={"ID":"3445db7b-baec-4665-b2ad-274e0c0f5da1","Type":"ContainerDied","Data":"62b7090a4f9c2201761bd0e3e2fe34ff1ee4ad3161182b1271ee06dd8dec6514"} Oct 30 16:36:17.862521 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:17.862489 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd"] Oct 30 16:36:17.866672 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:17.866651 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd" Oct 30 16:36:17.894259 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:17.894220 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd"] Oct 30 16:36:17.910603 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:17.910577 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b354d37-f9fd-4db3-9881-ea315888fd99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57a984c2-q7kmd\" (UID: \"9b354d37-f9fd-4db3-9881-ea315888fd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd" Oct 30 16:36:17.910753 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:17.910691 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b354d37-f9fd-4db3-9881-ea315888fd99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57a984c2-q7kmd\" (UID: \"9b354d37-f9fd-4db3-9881-ea315888fd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd" Oct 30 16:36:17.910753 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:17.910717 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpq2l\" (UniqueName: \"kubernetes.io/projected/9b354d37-f9fd-4db3-9881-ea315888fd99-kube-api-access-vpq2l\") pod \"maas-api-revoke-sub-57a984c2-q7kmd\" (UID: \"9b354d37-f9fd-4db3-9881-ea315888fd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd" Oct 30 16:36:18.011278 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.011248 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b354d37-f9fd-4db3-9881-ea315888fd99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57a984c2-q7kmd\" (UID: \"9b354d37-f9fd-4db3-9881-ea315888fd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd" Oct 30 16:36:18.011278 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.011281 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpq2l\" (UniqueName: \"kubernetes.io/projected/9b354d37-f9fd-4db3-9881-ea315888fd99-kube-api-access-vpq2l\") pod \"maas-api-revoke-sub-57a984c2-q7kmd\" (UID: \"9b354d37-f9fd-4db3-9881-ea315888fd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd" Oct 30 16:36:18.011494 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.011368 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b354d37-f9fd-4db3-9881-ea315888fd99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57a984c2-q7kmd\" (UID: \"9b354d37-f9fd-4db3-9881-ea315888fd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd" Oct 30 16:36:18.012045 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.012016 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b354d37-f9fd-4db3-9881-ea315888fd99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57a984c2-q7kmd\" (UID: \"9b354d37-f9fd-4db3-9881-ea315888fd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd" Oct 30 16:36:18.013872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.013851 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b354d37-f9fd-4db3-9881-ea315888fd99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57a984c2-q7kmd\" (UID: \"9b354d37-f9fd-4db3-9881-ea315888fd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd" Oct 30 16:36:18.035205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.035185 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpq2l\" (UniqueName: \"kubernetes.io/projected/9b354d37-f9fd-4db3-9881-ea315888fd99-kube-api-access-vpq2l\") pod \"maas-api-revoke-sub-57a984c2-q7kmd\" (UID: \"9b354d37-f9fd-4db3-9881-ea315888fd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd" Oct 30 16:36:18.180251 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.180215 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd" Oct 30 16:36:18.325089 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.325060 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm" Oct 30 16:36:18.414843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.414815 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3445db7b-baec-4665-b2ad-274e0c0f5da1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3445db7b-baec-4665-b2ad-274e0c0f5da1-maas-api-xks-ca\") pod \"3445db7b-baec-4665-b2ad-274e0c0f5da1\" (UID: \"3445db7b-baec-4665-b2ad-274e0c0f5da1\") " Oct 30 16:36:18.415017 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.414901 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3445db7b-baec-4665-b2ad-274e0c0f5da1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3445db7b-baec-4665-b2ad-274e0c0f5da1-maas-api-openshift-ca\") pod \"3445db7b-baec-4665-b2ad-274e0c0f5da1\" (UID: \"3445db7b-baec-4665-b2ad-274e0c0f5da1\") " Oct 30 16:36:18.415017 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.414981 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3445db7b-baec-4665-b2ad-274e0c0f5da1-kube-api-access-tjl48\" (UniqueName: \"kubernetes.io/projected/3445db7b-baec-4665-b2ad-274e0c0f5da1-kube-api-access-tjl48\") pod \"3445db7b-baec-4665-b2ad-274e0c0f5da1\" (UID: \"3445db7b-baec-4665-b2ad-274e0c0f5da1\") " Oct 30 16:36:18.415320 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.415295 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3445db7b-baec-4665-b2ad-274e0c0f5da1-maas-api-openshift-ca" pod "3445db7b-baec-4665-b2ad-274e0c0f5da1" (UID: "3445db7b-baec-4665-b2ad-274e0c0f5da1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:18.415435 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.415378 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3445db7b-baec-4665-b2ad-274e0c0f5da1-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:18.417350 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.417321 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3445db7b-baec-4665-b2ad-274e0c0f5da1-kube-api-access-tjl48" pod "3445db7b-baec-4665-b2ad-274e0c0f5da1" (UID: "3445db7b-baec-4665-b2ad-274e0c0f5da1"). InnerVolumeSpecName "kube-api-access-tjl48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:18.417741 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.417715 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3445db7b-baec-4665-b2ad-274e0c0f5da1-maas-api-xks-ca" pod "3445db7b-baec-4665-b2ad-274e0c0f5da1" (UID: "3445db7b-baec-4665-b2ad-274e0c0f5da1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:18.497304 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.497220 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:18.499113 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.499083 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd"] Oct 30 16:36:18.502372 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:36:18.502344 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b354d37_f9fd_4db3_9881_ea315888fd99.slice/crio-a47dab796299be482e382cf4b0140851044f3fb0cb387ed16cfb9ae39696cabd WatchSource:0}: Error finding container a47dab796299be482e382cf4b0140851044f3fb0cb387ed16cfb9ae39696cabd: Status 404 returned error can't find the container with id a47dab796299be482e382cf4b0140851044f3fb0cb387ed16cfb9ae39696cabd Oct 30 16:36:18.515984 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.515962 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjl48\" (UniqueName: \"kubernetes.io/projected/3445db7b-baec-4665-b2ad-274e0c0f5da1-kube-api-access-tjl48\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:18.516096 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.515985 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3445db7b-baec-4665-b2ad-274e0c0f5da1-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:18.548467 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.548429 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc"] Oct 30 16:36:18.549354 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.549319 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3445db7b-baec-4665-b2ad-274e0c0f5da1" containerName="revoke-keys" Oct 30 16:36:18.549354 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.549348 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3445db7b-baec-4665-b2ad-274e0c0f5da1" containerName="revoke-keys" Oct 30 16:36:18.549559 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.549543 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3445db7b-baec-4665-b2ad-274e0c0f5da1" containerName="revoke-keys" Oct 30 16:36:18.555000 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.554981 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.576390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.576362 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc"] Oct 30 16:36:18.616624 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.616595 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.616821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.616629 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.616821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.616680 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.616906 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.616832 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.616983 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.616901 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qvsl\" (UniqueName: \"kubernetes.io/projected/35198b3d-3a56-45c7-8d36-d4de1780a938-kube-api-access-8qvsl\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.616983 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.616933 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.617076 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.617004 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35198b3d-3a56-45c7-8d36-d4de1780a938-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.617145 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.617111 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.617230 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.617211 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.626544 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.626516 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8"] Oct 30 16:36:18.631407 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.631388 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8" Oct 30 16:36:18.650217 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.650195 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8"] Oct 30 16:36:18.718113 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.718082 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.718285 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.718118 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.718285 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.718145 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qvsl\" (UniqueName: \"kubernetes.io/projected/35198b3d-3a56-45c7-8d36-d4de1780a938-kube-api-access-8qvsl\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.718285 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.718165 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkxh5\" (UniqueName: \"kubernetes.io/projected/6bf28f2b-6a23-40aa-822b-bdf67321ae75-kube-api-access-wkxh5\") pod \"maas-api-revoke-sub-2a1c2927-2mqg8\" (UID: \"6bf28f2b-6a23-40aa-822b-bdf67321ae75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8" Oct 30 16:36:18.718285 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.718188 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf28f2b-6a23-40aa-822b-bdf67321ae75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a1c2927-2mqg8\" (UID: \"6bf28f2b-6a23-40aa-822b-bdf67321ae75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8" Oct 30 16:36:18.718285 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.718208 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.718557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.718318 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35198b3d-3a56-45c7-8d36-d4de1780a938-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.718557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.718483 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.718557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.718528 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf28f2b-6a23-40aa-822b-bdf67321ae75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a1c2927-2mqg8\" (UID: \"6bf28f2b-6a23-40aa-822b-bdf67321ae75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8" Oct 30 16:36:18.718680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.718554 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.718680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.718565 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.718680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.718613 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.718680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.718674 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.718884 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.718703 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.718884 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.718797 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.718994 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.718912 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35198b3d-3a56-45c7-8d36-d4de1780a938-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.719050 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.719033 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.738236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.738210 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.738583 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.738566 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.742072 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.742047 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qvsl\" (UniqueName: \"kubernetes.io/projected/35198b3d-3a56-45c7-8d36-d4de1780a938-kube-api-access-8qvsl\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.742183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.742152 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.820214 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.820176 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf28f2b-6a23-40aa-822b-bdf67321ae75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a1c2927-2mqg8\" (UID: \"6bf28f2b-6a23-40aa-822b-bdf67321ae75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8" Oct 30 16:36:18.820392 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.820326 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkxh5\" (UniqueName: \"kubernetes.io/projected/6bf28f2b-6a23-40aa-822b-bdf67321ae75-kube-api-access-wkxh5\") pod \"maas-api-revoke-sub-2a1c2927-2mqg8\" (UID: \"6bf28f2b-6a23-40aa-822b-bdf67321ae75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8" Oct 30 16:36:18.820392 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.820361 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf28f2b-6a23-40aa-822b-bdf67321ae75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a1c2927-2mqg8\" (UID: \"6bf28f2b-6a23-40aa-822b-bdf67321ae75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8" Oct 30 16:36:18.821028 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.821004 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf28f2b-6a23-40aa-822b-bdf67321ae75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a1c2927-2mqg8\" (UID: \"6bf28f2b-6a23-40aa-822b-bdf67321ae75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8" Oct 30 16:36:18.822747 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.822731 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf28f2b-6a23-40aa-822b-bdf67321ae75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a1c2927-2mqg8\" (UID: \"6bf28f2b-6a23-40aa-822b-bdf67321ae75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8" Oct 30 16:36:18.843429 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.843401 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkxh5\" (UniqueName: \"kubernetes.io/projected/6bf28f2b-6a23-40aa-822b-bdf67321ae75-kube-api-access-wkxh5\") pod \"maas-api-revoke-sub-2a1c2927-2mqg8\" (UID: \"6bf28f2b-6a23-40aa-822b-bdf67321ae75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8" Oct 30 16:36:18.874568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.874543 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:18.951750 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:18.946259 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8" Oct 30 16:36:19.179236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:19.179199 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd" event={"ID":"9b354d37-f9fd-4db3-9881-ea315888fd99","Type":"ContainerStarted","Data":"f288e574052a39e70dd5aa1a40ea6cd8ea655bc73446af0ef81c59a07705b1eb"} Oct 30 16:36:19.179433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:19.179285 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd" event={"ID":"9b354d37-f9fd-4db3-9881-ea315888fd99","Type":"ContainerStarted","Data":"a47dab796299be482e382cf4b0140851044f3fb0cb387ed16cfb9ae39696cabd"} Oct 30 16:36:19.180619 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:19.180588 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm" event={"ID":"3445db7b-baec-4665-b2ad-274e0c0f5da1","Type":"ContainerDied","Data":"a70345362f46dcfd3ffe77779ed26c2fa60986c05caea13d6ee1ad00f216aedd"} Oct 30 16:36:19.180994 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:19.180631 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a70345362f46dcfd3ffe77779ed26c2fa60986c05caea13d6ee1ad00f216aedd" Oct 30 16:36:19.180994 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:19.180637 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm" Oct 30 16:36:19.187878 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:19.187853 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc"] Oct 30 16:36:19.190656 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:19.190626 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:19.190736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:19.190687 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:19.190736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:19.190718 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:19.222611 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:19.222559 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd" podStartSLOduration=2.22254169 podStartE2EDuration="2.22254169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:19.218901285 +0000 UTC m=+892.929356027" watchObservedRunningTime="2026-10-30 16:36:19.22254169 +0000 UTC m=+892.932996434" Oct 30 16:36:19.251576 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:19.251549 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8"] Oct 30 16:36:19.252813 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:36:19.252789 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf28f2b_6a23_40aa_822b_bdf67321ae75.slice/crio-df7189ba6bc9cec2eb95fa9c4c26e9ec7a888de09a1bfb3425d9f228618bad98 WatchSource:0}: Error finding container df7189ba6bc9cec2eb95fa9c4c26e9ec7a888de09a1bfb3425d9f228618bad98: Status 404 returned error can't find the container with id df7189ba6bc9cec2eb95fa9c4c26e9ec7a888de09a1bfb3425d9f228618bad98 Oct 30 16:36:20.093181 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:20.093149 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:20.190673 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:20.190627 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" event={"ID":"35198b3d-3a56-45c7-8d36-d4de1780a938","Type":"ContainerStarted","Data":"6f1725e28ca712e879093ba350b2876bb755af261b2838fbf24b4787efd9ab02"} Oct 30 16:36:20.191162 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:20.190682 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" event={"ID":"35198b3d-3a56-45c7-8d36-d4de1780a938","Type":"ContainerStarted","Data":"5cc134960a6db43d8ace16eb4e0a1851ea7d514478982cb0bbb51552830b2755"} Oct 30 16:36:20.192461 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:20.192432 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8" event={"ID":"6bf28f2b-6a23-40aa-822b-bdf67321ae75","Type":"ContainerStarted","Data":"ef2f967928df90f5b291e3357d67f4f9d98e7b5a9b96f79414b0cebcdc0fa2ba"} Oct 30 16:36:20.192597 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:20.192475 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8" event={"ID":"6bf28f2b-6a23-40aa-822b-bdf67321ae75","Type":"ContainerStarted","Data":"df7189ba6bc9cec2eb95fa9c4c26e9ec7a888de09a1bfb3425d9f228618bad98"} Oct 30 16:36:20.194046 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:20.194021 6202 generic.go:356] "Generic (PLEG): container finished" podID="9b354d37-f9fd-4db3-9881-ea315888fd99" containerID="f288e574052a39e70dd5aa1a40ea6cd8ea655bc73446af0ef81c59a07705b1eb" exitCode=0 Oct 30 16:36:20.194150 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:20.194043 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd" event={"ID":"9b354d37-f9fd-4db3-9881-ea315888fd99","Type":"ContainerDied","Data":"f288e574052a39e70dd5aa1a40ea6cd8ea655bc73446af0ef81c59a07705b1eb"} Oct 30 16:36:20.248478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:20.248415 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" podStartSLOduration=2.2483912249999998 podStartE2EDuration="2.248391225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:20.236294958 +0000 UTC m=+893.946749701" watchObservedRunningTime="2026-10-30 16:36:20.248391225 +0000 UTC m=+893.958845970" Oct 30 16:36:20.284826 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:20.284758 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8" podStartSLOduration=2.28474605 podStartE2EDuration="2.28474605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:20.284074723 +0000 UTC m=+893.994529487" watchObservedRunningTime="2026-10-30 16:36:20.28474605 +0000 UTC m=+893.995200793" Oct 30 16:36:20.875188 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:20.875141 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:20.877520 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:20.877481 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 16:36:20.877659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:20.877540 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" podUID="35198b3d-3a56-45c7-8d36-d4de1780a938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 16:36:21.201375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:21.201289 6202 generic.go:356] "Generic (PLEG): container finished" podID="6bf28f2b-6a23-40aa-822b-bdf67321ae75" containerID="ef2f967928df90f5b291e3357d67f4f9d98e7b5a9b96f79414b0cebcdc0fa2ba" exitCode=0 Oct 30 16:36:21.201375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:21.201362 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8" event={"ID":"6bf28f2b-6a23-40aa-822b-bdf67321ae75","Type":"ContainerDied","Data":"ef2f967928df90f5b291e3357d67f4f9d98e7b5a9b96f79414b0cebcdc0fa2ba"} Oct 30 16:36:21.409448 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:21.409426 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd" Oct 30 16:36:21.490033 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:21.489941 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b354d37-f9fd-4db3-9881-ea315888fd99-kube-api-access-vpq2l\" (UniqueName: \"kubernetes.io/projected/9b354d37-f9fd-4db3-9881-ea315888fd99-kube-api-access-vpq2l\") pod \"9b354d37-f9fd-4db3-9881-ea315888fd99\" (UID: \"9b354d37-f9fd-4db3-9881-ea315888fd99\") " Oct 30 16:36:21.490180 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:21.490045 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b354d37-f9fd-4db3-9881-ea315888fd99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b354d37-f9fd-4db3-9881-ea315888fd99-maas-api-xks-ca\") pod \"9b354d37-f9fd-4db3-9881-ea315888fd99\" (UID: \"9b354d37-f9fd-4db3-9881-ea315888fd99\") " Oct 30 16:36:21.490180 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:21.490102 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b354d37-f9fd-4db3-9881-ea315888fd99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b354d37-f9fd-4db3-9881-ea315888fd99-maas-api-openshift-ca\") pod \"9b354d37-f9fd-4db3-9881-ea315888fd99\" (UID: \"9b354d37-f9fd-4db3-9881-ea315888fd99\") " Oct 30 16:36:21.490933 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:21.490905 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b354d37-f9fd-4db3-9881-ea315888fd99-maas-api-openshift-ca" pod "9b354d37-f9fd-4db3-9881-ea315888fd99" (UID: "9b354d37-f9fd-4db3-9881-ea315888fd99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:21.492727 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:21.492695 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b354d37-f9fd-4db3-9881-ea315888fd99-kube-api-access-vpq2l" pod "9b354d37-f9fd-4db3-9881-ea315888fd99" (UID: "9b354d37-f9fd-4db3-9881-ea315888fd99"). InnerVolumeSpecName "kube-api-access-vpq2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:21.492727 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:21.492697 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b354d37-f9fd-4db3-9881-ea315888fd99-maas-api-xks-ca" pod "9b354d37-f9fd-4db3-9881-ea315888fd99" (UID: "9b354d37-f9fd-4db3-9881-ea315888fd99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:21.592025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:21.591908 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpq2l\" (UniqueName: \"kubernetes.io/projected/9b354d37-f9fd-4db3-9881-ea315888fd99-kube-api-access-vpq2l\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:21.592025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:21.591943 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b354d37-f9fd-4db3-9881-ea315888fd99-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:21.592025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:21.591963 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b354d37-f9fd-4db3-9881-ea315888fd99-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:21.790863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:21.790750 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:21.875792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:21.875467 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 16:36:21.875792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:21.875524 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" podUID="35198b3d-3a56-45c7-8d36-d4de1780a938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 16:36:22.208987 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:22.208954 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd" Oct 30 16:36:22.209479 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:22.209289 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd" event={"ID":"9b354d37-f9fd-4db3-9881-ea315888fd99","Type":"ContainerDied","Data":"a47dab796299be482e382cf4b0140851044f3fb0cb387ed16cfb9ae39696cabd"} Oct 30 16:36:22.209479 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:22.209329 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a47dab796299be482e382cf4b0140851044f3fb0cb387ed16cfb9ae39696cabd" Oct 30 16:36:22.404106 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:22.404082 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8" Oct 30 16:36:22.504599 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:22.504512 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bf28f2b-6a23-40aa-822b-bdf67321ae75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf28f2b-6a23-40aa-822b-bdf67321ae75-maas-api-xks-ca\") pod \"6bf28f2b-6a23-40aa-822b-bdf67321ae75\" (UID: \"6bf28f2b-6a23-40aa-822b-bdf67321ae75\") " Oct 30 16:36:22.504761 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:22.504597 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bf28f2b-6a23-40aa-822b-bdf67321ae75-kube-api-access-wkxh5\" (UniqueName: \"kubernetes.io/projected/6bf28f2b-6a23-40aa-822b-bdf67321ae75-kube-api-access-wkxh5\") pod \"6bf28f2b-6a23-40aa-822b-bdf67321ae75\" (UID: \"6bf28f2b-6a23-40aa-822b-bdf67321ae75\") " Oct 30 16:36:22.505124 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:22.505087 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bf28f2b-6a23-40aa-822b-bdf67321ae75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf28f2b-6a23-40aa-822b-bdf67321ae75-maas-api-openshift-ca\") pod \"6bf28f2b-6a23-40aa-822b-bdf67321ae75\" (UID: \"6bf28f2b-6a23-40aa-822b-bdf67321ae75\") " Oct 30 16:36:22.505466 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:22.505439 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf28f2b-6a23-40aa-822b-bdf67321ae75-maas-api-openshift-ca" pod "6bf28f2b-6a23-40aa-822b-bdf67321ae75" (UID: "6bf28f2b-6a23-40aa-822b-bdf67321ae75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:22.506038 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:22.506016 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf28f2b-6a23-40aa-822b-bdf67321ae75-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:22.507672 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:22.507645 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf28f2b-6a23-40aa-822b-bdf67321ae75-kube-api-access-wkxh5" pod "6bf28f2b-6a23-40aa-822b-bdf67321ae75" (UID: "6bf28f2b-6a23-40aa-822b-bdf67321ae75"). InnerVolumeSpecName "kube-api-access-wkxh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:22.507833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:22.507814 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bf28f2b-6a23-40aa-822b-bdf67321ae75-maas-api-xks-ca" pod "6bf28f2b-6a23-40aa-822b-bdf67321ae75" (UID: "6bf28f2b-6a23-40aa-822b-bdf67321ae75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:22.607606 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:22.607577 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf28f2b-6a23-40aa-822b-bdf67321ae75-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:22.607606 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:22.607603 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkxh5\" (UniqueName: \"kubernetes.io/projected/6bf28f2b-6a23-40aa-822b-bdf67321ae75-kube-api-access-wkxh5\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:22.886786 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:22.879285 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 16:36:22.886786 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:22.879358 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" podUID="35198b3d-3a56-45c7-8d36-d4de1780a938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 16:36:23.216183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:23.216006 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8" event={"ID":"6bf28f2b-6a23-40aa-822b-bdf67321ae75","Type":"ContainerDied","Data":"df7189ba6bc9cec2eb95fa9c4c26e9ec7a888de09a1bfb3425d9f228618bad98"} Oct 30 16:36:23.216183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:23.216041 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8" Oct 30 16:36:23.216183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:23.216099 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df7189ba6bc9cec2eb95fa9c4c26e9ec7a888de09a1bfb3425d9f228618bad98" Oct 30 16:36:23.415344 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:23.415310 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:23.874933 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:23.874894 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 16:36:23.875152 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:23.874958 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" podUID="35198b3d-3a56-45c7-8d36-d4de1780a938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 16:36:24.875255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:24.875219 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 16:36:24.875846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:24.875818 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" podUID="35198b3d-3a56-45c7-8d36-d4de1780a938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 16:36:24.921247 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:24.921212 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:25.846015 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:25.845979 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24"] Oct 30 16:36:25.846557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:25.846541 6202 cpu_manager.go:414] "containerMap: removing container" podUID="6bf28f2b-6a23-40aa-822b-bdf67321ae75" containerName="revoke-keys" Oct 30 16:36:25.846601 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:25.846560 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf28f2b-6a23-40aa-822b-bdf67321ae75" containerName="revoke-keys" Oct 30 16:36:25.846601 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:25.846594 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9b354d37-f9fd-4db3-9881-ea315888fd99" containerName="revoke-keys" Oct 30 16:36:25.846601 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:25.846599 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b354d37-f9fd-4db3-9881-ea315888fd99" containerName="revoke-keys" Oct 30 16:36:25.846696 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:25.846682 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b354d37-f9fd-4db3-9881-ea315888fd99" containerName="revoke-keys" Oct 30 16:36:25.846696 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:25.846691 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf28f2b-6a23-40aa-822b-bdf67321ae75" containerName="revoke-keys" Oct 30 16:36:25.850318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:25.850296 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24" Oct 30 16:36:25.879845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:25.879817 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:36:25.880299 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:25.879886 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:36:25.880299 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:25.880052 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:36:25.883218 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:25.883191 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24"] Oct 30 16:36:25.883359 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:25.883280 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:36:25.884444 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:25.884416 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:25.884999 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:25.884983 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:25.885845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:25.885824 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 16:36:25.979796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:25.966559 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlgbh\" (UniqueName: \"kubernetes.io/projected/6b6ee8e3-149f-40f2-9e0f-586559326f88-kube-api-access-qlgbh\") pod \"maas-api-revoke-sub-2b97bb07-28j24\" (UID: \"6b6ee8e3-149f-40f2-9e0f-586559326f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24" Oct 30 16:36:26.012182 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:26.012134 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6ee8e3-149f-40f2-9e0f-586559326f88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b97bb07-28j24\" (UID: \"6b6ee8e3-149f-40f2-9e0f-586559326f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24" Oct 30 16:36:26.027792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:26.012696 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6ee8e3-149f-40f2-9e0f-586559326f88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b97bb07-28j24\" (UID: \"6b6ee8e3-149f-40f2-9e0f-586559326f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24" Oct 30 16:36:26.136403 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:26.136313 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6ee8e3-149f-40f2-9e0f-586559326f88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b97bb07-28j24\" (UID: \"6b6ee8e3-149f-40f2-9e0f-586559326f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24" Oct 30 16:36:26.137478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:26.137458 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlgbh\" (UniqueName: \"kubernetes.io/projected/6b6ee8e3-149f-40f2-9e0f-586559326f88-kube-api-access-qlgbh\") pod \"maas-api-revoke-sub-2b97bb07-28j24\" (UID: \"6b6ee8e3-149f-40f2-9e0f-586559326f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24" Oct 30 16:36:26.138050 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:26.138031 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6ee8e3-149f-40f2-9e0f-586559326f88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b97bb07-28j24\" (UID: \"6b6ee8e3-149f-40f2-9e0f-586559326f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24" Oct 30 16:36:26.144330 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:26.137344 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6ee8e3-149f-40f2-9e0f-586559326f88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b97bb07-28j24\" (UID: \"6b6ee8e3-149f-40f2-9e0f-586559326f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24" Oct 30 16:36:26.179245 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:26.179209 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6ee8e3-149f-40f2-9e0f-586559326f88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b97bb07-28j24\" (UID: \"6b6ee8e3-149f-40f2-9e0f-586559326f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24" Oct 30 16:36:26.185023 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:26.184987 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlgbh\" (UniqueName: \"kubernetes.io/projected/6b6ee8e3-149f-40f2-9e0f-586559326f88-kube-api-access-qlgbh\") pod \"maas-api-revoke-sub-2b97bb07-28j24\" (UID: \"6b6ee8e3-149f-40f2-9e0f-586559326f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24" Oct 30 16:36:26.570519 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:26.570108 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:36:26.570519 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:26.570182 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24" Oct 30 16:36:26.846740 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:26.846675 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:27.593808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:27.563870 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 16:36:27.724812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:27.721538 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:36:27.899868 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:27.898904 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:27.970326 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:27.970290 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24"] Oct 30 16:36:28.360819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:28.360784 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24" event={"ID":"6b6ee8e3-149f-40f2-9e0f-586559326f88","Type":"ContainerStarted","Data":"3497de16c23f15002d4a4df6d291a93984e2f9e710ff7b952b3c72d0babe9fd2"} Oct 30 16:36:28.703789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:28.703308 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:29.383107 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:29.383068 6202 generic.go:356] "Generic (PLEG): container finished" podID="6b6ee8e3-149f-40f2-9e0f-586559326f88" containerID="eed38e814845849b028a9145efa899f47b4c1acf2efcd0cb6d127123679e9bb3" exitCode=0 Oct 30 16:36:29.383425 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:29.383399 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24" event={"ID":"6b6ee8e3-149f-40f2-9e0f-586559326f88","Type":"ContainerDied","Data":"eed38e814845849b028a9145efa899f47b4c1acf2efcd0cb6d127123679e9bb3"} Oct 30 16:36:30.369187 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:30.369147 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:30.541445 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:30.541418 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24" Oct 30 16:36:30.629593 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:30.629514 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b6ee8e3-149f-40f2-9e0f-586559326f88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6ee8e3-149f-40f2-9e0f-586559326f88-maas-api-openshift-ca\") pod \"6b6ee8e3-149f-40f2-9e0f-586559326f88\" (UID: \"6b6ee8e3-149f-40f2-9e0f-586559326f88\") " Oct 30 16:36:30.629593 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:30.629579 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b6ee8e3-149f-40f2-9e0f-586559326f88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6ee8e3-149f-40f2-9e0f-586559326f88-maas-api-xks-ca\") pod \"6b6ee8e3-149f-40f2-9e0f-586559326f88\" (UID: \"6b6ee8e3-149f-40f2-9e0f-586559326f88\") " Oct 30 16:36:30.629855 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:30.629605 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6ee8e3-149f-40f2-9e0f-586559326f88-kube-api-access-qlgbh\" (UniqueName: \"kubernetes.io/projected/6b6ee8e3-149f-40f2-9e0f-586559326f88-kube-api-access-qlgbh\") pod \"6b6ee8e3-149f-40f2-9e0f-586559326f88\" (UID: \"6b6ee8e3-149f-40f2-9e0f-586559326f88\") " Oct 30 16:36:30.629907 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:30.629884 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6ee8e3-149f-40f2-9e0f-586559326f88-maas-api-openshift-ca" pod "6b6ee8e3-149f-40f2-9e0f-586559326f88" (UID: "6b6ee8e3-149f-40f2-9e0f-586559326f88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:30.630005 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:30.629989 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6ee8e3-149f-40f2-9e0f-586559326f88-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:30.631867 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:30.631838 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6ee8e3-149f-40f2-9e0f-586559326f88-maas-api-xks-ca" pod "6b6ee8e3-149f-40f2-9e0f-586559326f88" (UID: "6b6ee8e3-149f-40f2-9e0f-586559326f88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:30.631984 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:30.631868 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6ee8e3-149f-40f2-9e0f-586559326f88-kube-api-access-qlgbh" pod "6b6ee8e3-149f-40f2-9e0f-586559326f88" (UID: "6b6ee8e3-149f-40f2-9e0f-586559326f88"). InnerVolumeSpecName "kube-api-access-qlgbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:30.731586 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:30.731556 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6ee8e3-149f-40f2-9e0f-586559326f88-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:30.731586 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:30.731581 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlgbh\" (UniqueName: \"kubernetes.io/projected/6b6ee8e3-149f-40f2-9e0f-586559326f88-kube-api-access-qlgbh\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:31.192715 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.192674 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt"] Oct 30 16:36:31.193366 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.193344 6202 cpu_manager.go:414] "containerMap: removing container" podUID="6b6ee8e3-149f-40f2-9e0f-586559326f88" containerName="revoke-keys" Oct 30 16:36:31.193366 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.193363 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b6ee8e3-149f-40f2-9e0f-586559326f88" containerName="revoke-keys" Oct 30 16:36:31.193518 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.193508 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b6ee8e3-149f-40f2-9e0f-586559326f88" containerName="revoke-keys" Oct 30 16:36:31.197167 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.197145 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt" Oct 30 16:36:31.216519 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.216490 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt"] Oct 30 16:36:31.269801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.269755 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66"] Oct 30 16:36:31.273734 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.273716 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66" Oct 30 16:36:31.293234 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.293206 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66"] Oct 30 16:36:31.339824 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.339758 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqz7z\" (UniqueName: \"kubernetes.io/projected/77b79ab5-a82f-416b-8fbe-a4c2de84a128-kube-api-access-nqz7z\") pod \"maas-api-revoke-sub-80392282-n2vbt\" (UID: \"77b79ab5-a82f-416b-8fbe-a4c2de84a128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt" Oct 30 16:36:31.340078 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.339865 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b79ab5-a82f-416b-8fbe-a4c2de84a128-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80392282-n2vbt\" (UID: \"77b79ab5-a82f-416b-8fbe-a4c2de84a128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt" Oct 30 16:36:31.340078 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.339903 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b79ab5-a82f-416b-8fbe-a4c2de84a128-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80392282-n2vbt\" (UID: \"77b79ab5-a82f-416b-8fbe-a4c2de84a128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt" Oct 30 16:36:31.408129 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.408093 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24" event={"ID":"6b6ee8e3-149f-40f2-9e0f-586559326f88","Type":"ContainerDied","Data":"3497de16c23f15002d4a4df6d291a93984e2f9e710ff7b952b3c72d0babe9fd2"} Oct 30 16:36:31.408129 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.408129 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24" Oct 30 16:36:31.408671 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.408143 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3497de16c23f15002d4a4df6d291a93984e2f9e710ff7b952b3c72d0babe9fd2" Oct 30 16:36:31.441276 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.441243 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0253d5-274d-4e10-9331-954e74065e17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fa490f0-lmc66\" (UID: \"6e0253d5-274d-4e10-9331-954e74065e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66" Oct 30 16:36:31.441428 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.441293 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjln5\" (UniqueName: \"kubernetes.io/projected/6e0253d5-274d-4e10-9331-954e74065e17-kube-api-access-cjln5\") pod \"maas-api-revoke-sub-1fa490f0-lmc66\" (UID: \"6e0253d5-274d-4e10-9331-954e74065e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66" Oct 30 16:36:31.441428 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.441391 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqz7z\" (UniqueName: \"kubernetes.io/projected/77b79ab5-a82f-416b-8fbe-a4c2de84a128-kube-api-access-nqz7z\") pod \"maas-api-revoke-sub-80392282-n2vbt\" (UID: \"77b79ab5-a82f-416b-8fbe-a4c2de84a128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt" Oct 30 16:36:31.441554 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.441437 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0253d5-274d-4e10-9331-954e74065e17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fa490f0-lmc66\" (UID: \"6e0253d5-274d-4e10-9331-954e74065e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66" Oct 30 16:36:31.441554 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.441457 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b79ab5-a82f-416b-8fbe-a4c2de84a128-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80392282-n2vbt\" (UID: \"77b79ab5-a82f-416b-8fbe-a4c2de84a128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt" Oct 30 16:36:31.441554 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.441523 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b79ab5-a82f-416b-8fbe-a4c2de84a128-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80392282-n2vbt\" (UID: \"77b79ab5-a82f-416b-8fbe-a4c2de84a128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt" Oct 30 16:36:31.442240 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.442220 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b79ab5-a82f-416b-8fbe-a4c2de84a128-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80392282-n2vbt\" (UID: \"77b79ab5-a82f-416b-8fbe-a4c2de84a128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt" Oct 30 16:36:31.444177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.444131 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b79ab5-a82f-416b-8fbe-a4c2de84a128-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80392282-n2vbt\" (UID: \"77b79ab5-a82f-416b-8fbe-a4c2de84a128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt" Oct 30 16:36:31.465578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.465548 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqz7z\" (UniqueName: \"kubernetes.io/projected/77b79ab5-a82f-416b-8fbe-a4c2de84a128-kube-api-access-nqz7z\") pod \"maas-api-revoke-sub-80392282-n2vbt\" (UID: \"77b79ab5-a82f-416b-8fbe-a4c2de84a128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt" Oct 30 16:36:31.514363 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.514328 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt" Oct 30 16:36:31.542386 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.542350 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0253d5-274d-4e10-9331-954e74065e17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fa490f0-lmc66\" (UID: \"6e0253d5-274d-4e10-9331-954e74065e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66" Oct 30 16:36:31.542551 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.542428 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjln5\" (UniqueName: \"kubernetes.io/projected/6e0253d5-274d-4e10-9331-954e74065e17-kube-api-access-cjln5\") pod \"maas-api-revoke-sub-1fa490f0-lmc66\" (UID: \"6e0253d5-274d-4e10-9331-954e74065e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66" Oct 30 16:36:31.542551 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.542529 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0253d5-274d-4e10-9331-954e74065e17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fa490f0-lmc66\" (UID: \"6e0253d5-274d-4e10-9331-954e74065e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66" Oct 30 16:36:31.543412 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.543362 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0253d5-274d-4e10-9331-954e74065e17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fa490f0-lmc66\" (UID: \"6e0253d5-274d-4e10-9331-954e74065e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66" Oct 30 16:36:31.545437 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.545414 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0253d5-274d-4e10-9331-954e74065e17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fa490f0-lmc66\" (UID: \"6e0253d5-274d-4e10-9331-954e74065e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66" Oct 30 16:36:31.565830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.565798 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjln5\" (UniqueName: \"kubernetes.io/projected/6e0253d5-274d-4e10-9331-954e74065e17-kube-api-access-cjln5\") pod \"maas-api-revoke-sub-1fa490f0-lmc66\" (UID: \"6e0253d5-274d-4e10-9331-954e74065e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66" Oct 30 16:36:31.587106 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.587074 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66" Oct 30 16:36:31.824546 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.824515 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt"] Oct 30 16:36:31.825202 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:36:31.825176 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b79ab5_a82f_416b_8fbe_a4c2de84a128.slice/crio-8857fbf1c3b0b67ff6c4374869bd69ddaa13566b344c518be4ca1263901b971e WatchSource:0}: Error finding container 8857fbf1c3b0b67ff6c4374869bd69ddaa13566b344c518be4ca1263901b971e: Status 404 returned error can't find the container with id 8857fbf1c3b0b67ff6c4374869bd69ddaa13566b344c518be4ca1263901b971e Oct 30 16:36:31.903060 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:36:31.903029 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0253d5_274d_4e10_9331_954e74065e17.slice/crio-01ab5589c905862070ecb7848b77ac9ea6b0f297cac9431817796c0e520a3360 WatchSource:0}: Error finding container 01ab5589c905862070ecb7848b77ac9ea6b0f297cac9431817796c0e520a3360: Status 404 returned error can't find the container with id 01ab5589c905862070ecb7848b77ac9ea6b0f297cac9431817796c0e520a3360 Oct 30 16:36:31.904063 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.903969 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66"] Oct 30 16:36:31.950379 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:31.950348 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:32.415835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:32.415726 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt" event={"ID":"77b79ab5-a82f-416b-8fbe-a4c2de84a128","Type":"ContainerStarted","Data":"d12859ba897da5a354c27b2424db298bea0dbb14ceec03d31e9580d957abbd89"} Oct 30 16:36:32.415835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:32.415815 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt" event={"ID":"77b79ab5-a82f-416b-8fbe-a4c2de84a128","Type":"ContainerStarted","Data":"8857fbf1c3b0b67ff6c4374869bd69ddaa13566b344c518be4ca1263901b971e"} Oct 30 16:36:32.417470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:32.417441 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66" event={"ID":"6e0253d5-274d-4e10-9331-954e74065e17","Type":"ContainerStarted","Data":"e4911e63163c4346b39332151b898412ea73890812d62e49d18336459a3437f9"} Oct 30 16:36:32.417577 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:32.417491 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66" event={"ID":"6e0253d5-274d-4e10-9331-954e74065e17","Type":"ContainerStarted","Data":"01ab5589c905862070ecb7848b77ac9ea6b0f297cac9431817796c0e520a3360"} Oct 30 16:36:32.459073 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:32.458735 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt" podStartSLOduration=1.45871753 podStartE2EDuration="1.45871753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:32.455276014 +0000 UTC m=+906.165730755" watchObservedRunningTime="2026-10-30 16:36:32.45871753 +0000 UTC m=+906.169172274" Oct 30 16:36:32.498304 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:32.498246 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66" podStartSLOduration=1.498225984 podStartE2EDuration="1.498225984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:32.497508467 +0000 UTC m=+906.207963211" watchObservedRunningTime="2026-10-30 16:36:32.498225984 +0000 UTC m=+906.208680725" Oct 30 16:36:33.404413 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.404381 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97"] Oct 30 16:36:33.408693 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.408662 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.428520 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.428470 6202 generic.go:356] "Generic (PLEG): container finished" podID="6e0253d5-274d-4e10-9331-954e74065e17" containerID="e4911e63163c4346b39332151b898412ea73890812d62e49d18336459a3437f9" exitCode=0 Oct 30 16:36:33.429144 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.428678 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66" event={"ID":"6e0253d5-274d-4e10-9331-954e74065e17","Type":"ContainerDied","Data":"e4911e63163c4346b39332151b898412ea73890812d62e49d18336459a3437f9"} Oct 30 16:36:33.431069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.430638 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97"] Oct 30 16:36:33.432619 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.432591 6202 generic.go:356] "Generic (PLEG): container finished" podID="77b79ab5-a82f-416b-8fbe-a4c2de84a128" containerID="d12859ba897da5a354c27b2424db298bea0dbb14ceec03d31e9580d957abbd89" exitCode=0 Oct 30 16:36:33.432744 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.432719 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt" event={"ID":"77b79ab5-a82f-416b-8fbe-a4c2de84a128","Type":"ContainerDied","Data":"d12859ba897da5a354c27b2424db298bea0dbb14ceec03d31e9580d957abbd89"} Oct 30 16:36:33.438319 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.438287 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0b2348\"/\"distinct-w3-0b2348-kserve-self-signed-certs\"" Oct 30 16:36:33.473512 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.473472 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:33.565700 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.565651 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-model-cache\") pod \"distinct-w3-0b2348-kserve-7d8cd756c9-rlp97\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.565923 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.565719 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-tls-certs\") pod \"distinct-w3-0b2348-kserve-7d8cd756c9-rlp97\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.565923 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.565851 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-home\") pod \"distinct-w3-0b2348-kserve-7d8cd756c9-rlp97\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.566036 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.565950 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbpkn\" (UniqueName: \"kubernetes.io/projected/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-kube-api-access-cbpkn\") pod \"distinct-w3-0b2348-kserve-7d8cd756c9-rlp97\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.566036 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.566006 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-tmp-dir\") pod \"distinct-w3-0b2348-kserve-7d8cd756c9-rlp97\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.566145 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.566124 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-dshm\") pod \"distinct-w3-0b2348-kserve-7d8cd756c9-rlp97\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.668077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.667941 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-model-cache\") pod \"distinct-w3-0b2348-kserve-7d8cd756c9-rlp97\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.668077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.668038 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-tls-certs\") pod \"distinct-w3-0b2348-kserve-7d8cd756c9-rlp97\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.668077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.668073 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-home\") pod \"distinct-w3-0b2348-kserve-7d8cd756c9-rlp97\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.668368 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.668158 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbpkn\" (UniqueName: \"kubernetes.io/projected/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-kube-api-access-cbpkn\") pod \"distinct-w3-0b2348-kserve-7d8cd756c9-rlp97\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.668368 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.668206 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-tmp-dir\") pod \"distinct-w3-0b2348-kserve-7d8cd756c9-rlp97\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.668368 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.668243 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-dshm\") pod \"distinct-w3-0b2348-kserve-7d8cd756c9-rlp97\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.668368 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.668325 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-model-cache\") pod \"distinct-w3-0b2348-kserve-7d8cd756c9-rlp97\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.668587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.668450 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-home\") pod \"distinct-w3-0b2348-kserve-7d8cd756c9-rlp97\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.668669 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.668651 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-tmp-dir\") pod \"distinct-w3-0b2348-kserve-7d8cd756c9-rlp97\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.670710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.670688 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-tls-certs\") pod \"distinct-w3-0b2348-kserve-7d8cd756c9-rlp97\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.673055 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.673030 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-dshm\") pod \"distinct-w3-0b2348-kserve-7d8cd756c9-rlp97\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.692254 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.692218 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbpkn\" (UniqueName: \"kubernetes.io/projected/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-kube-api-access-cbpkn\") pod \"distinct-w3-0b2348-kserve-7d8cd756c9-rlp97\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.723966 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.723931 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:33.732123 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:33.732098 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:34.028540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.028512 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97"] Oct 30 16:36:34.029386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:36:34.029357 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a8a1983_6f7b_4a9c_b9d5_8beaf2a490f9.slice/crio-1a506d9a9bfbd4a15486846931af29c13f4ebca5ca83b4ee5d2b3329276d32fa WatchSource:0}: Error finding container 1a506d9a9bfbd4a15486846931af29c13f4ebca5ca83b4ee5d2b3329276d32fa: Status 404 returned error can't find the container with id 1a506d9a9bfbd4a15486846931af29c13f4ebca5ca83b4ee5d2b3329276d32fa Oct 30 16:36:34.441290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.441249 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" event={"ID":"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9","Type":"ContainerStarted","Data":"863ae4cffd7037cecc08e523b35e721f2ce6126e1eae6d51adaa000171928dbc"} Oct 30 16:36:34.441825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.441306 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" event={"ID":"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9","Type":"ContainerStarted","Data":"1a506d9a9bfbd4a15486846931af29c13f4ebca5ca83b4ee5d2b3329276d32fa"} Oct 30 16:36:34.482494 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.482363 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podStartSLOduration=1.482342799 podStartE2EDuration="1.482342799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:34.481467561 +0000 UTC m=+908.191922306" watchObservedRunningTime="2026-10-30 16:36:34.482342799 +0000 UTC m=+908.192797542" Oct 30 16:36:34.588298 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.588263 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt" Oct 30 16:36:34.637992 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.637956 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66" Oct 30 16:36:34.724926 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.724837 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:34.784531 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.784492 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77b79ab5-a82f-416b-8fbe-a4c2de84a128-kube-api-access-nqz7z\" (UniqueName: \"kubernetes.io/projected/77b79ab5-a82f-416b-8fbe-a4c2de84a128-kube-api-access-nqz7z\") pod \"77b79ab5-a82f-416b-8fbe-a4c2de84a128\" (UID: \"77b79ab5-a82f-416b-8fbe-a4c2de84a128\") " Oct 30 16:36:34.784736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.784575 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e0253d5-274d-4e10-9331-954e74065e17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0253d5-274d-4e10-9331-954e74065e17-maas-api-xks-ca\") pod \"6e0253d5-274d-4e10-9331-954e74065e17\" (UID: \"6e0253d5-274d-4e10-9331-954e74065e17\") " Oct 30 16:36:34.784736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.784624 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77b79ab5-a82f-416b-8fbe-a4c2de84a128-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b79ab5-a82f-416b-8fbe-a4c2de84a128-maas-api-xks-ca\") pod \"77b79ab5-a82f-416b-8fbe-a4c2de84a128\" (UID: \"77b79ab5-a82f-416b-8fbe-a4c2de84a128\") " Oct 30 16:36:34.784736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.784643 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e0253d5-274d-4e10-9331-954e74065e17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0253d5-274d-4e10-9331-954e74065e17-maas-api-openshift-ca\") pod \"6e0253d5-274d-4e10-9331-954e74065e17\" (UID: \"6e0253d5-274d-4e10-9331-954e74065e17\") " Oct 30 16:36:34.784950 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.784810 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77b79ab5-a82f-416b-8fbe-a4c2de84a128-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b79ab5-a82f-416b-8fbe-a4c2de84a128-maas-api-openshift-ca\") pod \"77b79ab5-a82f-416b-8fbe-a4c2de84a128\" (UID: \"77b79ab5-a82f-416b-8fbe-a4c2de84a128\") " Oct 30 16:36:34.785139 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.785093 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0253d5-274d-4e10-9331-954e74065e17-maas-api-openshift-ca" pod "6e0253d5-274d-4e10-9331-954e74065e17" (UID: "6e0253d5-274d-4e10-9331-954e74065e17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:34.785270 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.785154 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77b79ab5-a82f-416b-8fbe-a4c2de84a128-maas-api-openshift-ca" pod "77b79ab5-a82f-416b-8fbe-a4c2de84a128" (UID: "77b79ab5-a82f-416b-8fbe-a4c2de84a128"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:34.785270 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.785235 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e0253d5-274d-4e10-9331-954e74065e17-kube-api-access-cjln5\" (UniqueName: \"kubernetes.io/projected/6e0253d5-274d-4e10-9331-954e74065e17-kube-api-access-cjln5\") pod \"6e0253d5-274d-4e10-9331-954e74065e17\" (UID: \"6e0253d5-274d-4e10-9331-954e74065e17\") " Oct 30 16:36:34.785935 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.785911 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0253d5-274d-4e10-9331-954e74065e17-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:34.786022 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.785941 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b79ab5-a82f-416b-8fbe-a4c2de84a128-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:34.787011 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.786984 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e0253d5-274d-4e10-9331-954e74065e17-maas-api-xks-ca" pod "6e0253d5-274d-4e10-9331-954e74065e17" (UID: "6e0253d5-274d-4e10-9331-954e74065e17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:34.787087 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.787016 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b79ab5-a82f-416b-8fbe-a4c2de84a128-kube-api-access-nqz7z" pod "77b79ab5-a82f-416b-8fbe-a4c2de84a128" (UID: "77b79ab5-a82f-416b-8fbe-a4c2de84a128"). InnerVolumeSpecName "kube-api-access-nqz7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:34.787087 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.787059 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77b79ab5-a82f-416b-8fbe-a4c2de84a128-maas-api-xks-ca" pod "77b79ab5-a82f-416b-8fbe-a4c2de84a128" (UID: "77b79ab5-a82f-416b-8fbe-a4c2de84a128"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:34.787725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.787710 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0253d5-274d-4e10-9331-954e74065e17-kube-api-access-cjln5" pod "6e0253d5-274d-4e10-9331-954e74065e17" (UID: "6e0253d5-274d-4e10-9331-954e74065e17"). InnerVolumeSpecName "kube-api-access-cjln5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:34.886524 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.886494 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqz7z\" (UniqueName: \"kubernetes.io/projected/77b79ab5-a82f-416b-8fbe-a4c2de84a128-kube-api-access-nqz7z\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:34.886524 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.886522 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0253d5-274d-4e10-9331-954e74065e17-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:34.886524 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.886531 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b79ab5-a82f-416b-8fbe-a4c2de84a128-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:34.886801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:34.886540 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjln5\" (UniqueName: \"kubernetes.io/projected/6e0253d5-274d-4e10-9331-954e74065e17-kube-api-access-cjln5\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:35.352888 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:35.352844 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:35.449702 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:35.449618 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt" Oct 30 16:36:35.449702 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:35.449650 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt" event={"ID":"77b79ab5-a82f-416b-8fbe-a4c2de84a128","Type":"ContainerDied","Data":"8857fbf1c3b0b67ff6c4374869bd69ddaa13566b344c518be4ca1263901b971e"} Oct 30 16:36:35.450139 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:35.449707 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8857fbf1c3b0b67ff6c4374869bd69ddaa13566b344c518be4ca1263901b971e" Oct 30 16:36:35.451412 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:35.451390 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66" Oct 30 16:36:35.451544 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:35.451414 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66" event={"ID":"6e0253d5-274d-4e10-9331-954e74065e17","Type":"ContainerDied","Data":"01ab5589c905862070ecb7848b77ac9ea6b0f297cac9431817796c0e520a3360"} Oct 30 16:36:35.451544 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:35.451462 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01ab5589c905862070ecb7848b77ac9ea6b0f297cac9431817796c0e520a3360" Oct 30 16:36:36.554877 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:36.554839 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:36.970888 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:36.970851 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:38.690865 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:38.690830 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:41.012662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.012623 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:41.065327 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.065244 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7"] Oct 30 16:36:41.066168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.066144 6202 cpu_manager.go:414] "containerMap: removing container" podUID="77b79ab5-a82f-416b-8fbe-a4c2de84a128" containerName="revoke-keys" Oct 30 16:36:41.066168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.066168 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b79ab5-a82f-416b-8fbe-a4c2de84a128" containerName="revoke-keys" Oct 30 16:36:41.066391 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.066181 6202 cpu_manager.go:414] "containerMap: removing container" podUID="6e0253d5-274d-4e10-9331-954e74065e17" containerName="revoke-keys" Oct 30 16:36:41.066391 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.066189 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e0253d5-274d-4e10-9331-954e74065e17" containerName="revoke-keys" Oct 30 16:36:41.066391 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.066342 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b79ab5-a82f-416b-8fbe-a4c2de84a128" containerName="revoke-keys" Oct 30 16:36:41.066391 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.066355 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e0253d5-274d-4e10-9331-954e74065e17" containerName="revoke-keys" Oct 30 16:36:41.072357 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.072324 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7" Oct 30 16:36:41.090572 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.090535 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7"] Oct 30 16:36:41.097277 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.097245 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:36:41.097661 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.097643 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:36:41.097760 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.097704 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:36:41.098815 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.098757 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:36:41.182822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.182751 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kw84\" (UniqueName: \"kubernetes.io/projected/5be78108-87f2-41b7-bee1-0b404be3614b-kube-api-access-6kw84\") pod \"maas-api-revoke-sub-7db40c21-6w6k7\" (UID: \"5be78108-87f2-41b7-bee1-0b404be3614b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7" Oct 30 16:36:41.183019 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.183002 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be78108-87f2-41b7-bee1-0b404be3614b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7db40c21-6w6k7\" (UID: \"5be78108-87f2-41b7-bee1-0b404be3614b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7" Oct 30 16:36:41.183064 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.183032 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be78108-87f2-41b7-bee1-0b404be3614b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7db40c21-6w6k7\" (UID: \"5be78108-87f2-41b7-bee1-0b404be3614b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7" Oct 30 16:36:41.284621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.284530 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kw84\" (UniqueName: \"kubernetes.io/projected/5be78108-87f2-41b7-bee1-0b404be3614b-kube-api-access-6kw84\") pod \"maas-api-revoke-sub-7db40c21-6w6k7\" (UID: \"5be78108-87f2-41b7-bee1-0b404be3614b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7" Oct 30 16:36:41.284809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.284724 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be78108-87f2-41b7-bee1-0b404be3614b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7db40c21-6w6k7\" (UID: \"5be78108-87f2-41b7-bee1-0b404be3614b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7" Oct 30 16:36:41.284809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.284779 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be78108-87f2-41b7-bee1-0b404be3614b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7db40c21-6w6k7\" (UID: \"5be78108-87f2-41b7-bee1-0b404be3614b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7" Oct 30 16:36:41.285421 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.285389 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be78108-87f2-41b7-bee1-0b404be3614b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7db40c21-6w6k7\" (UID: \"5be78108-87f2-41b7-bee1-0b404be3614b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7" Oct 30 16:36:41.287300 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.287279 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be78108-87f2-41b7-bee1-0b404be3614b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7db40c21-6w6k7\" (UID: \"5be78108-87f2-41b7-bee1-0b404be3614b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7" Oct 30 16:36:41.308530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.308498 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kw84\" (UniqueName: \"kubernetes.io/projected/5be78108-87f2-41b7-bee1-0b404be3614b-kube-api-access-6kw84\") pod \"maas-api-revoke-sub-7db40c21-6w6k7\" (UID: \"5be78108-87f2-41b7-bee1-0b404be3614b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7" Oct 30 16:36:41.390701 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.390665 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7" Oct 30 16:36:41.687643 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:41.687614 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7"] Oct 30 16:36:41.687917 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:36:41.687885 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be78108_87f2_41b7_bee1_0b404be3614b.slice/crio-5501f7f71c00a5d1f83e7d51829b1524f53e87c45ba54ca7d0981114ee6bc845 WatchSource:0}: Error finding container 5501f7f71c00a5d1f83e7d51829b1524f53e87c45ba54ca7d0981114ee6bc845: Status 404 returned error can't find the container with id 5501f7f71c00a5d1f83e7d51829b1524f53e87c45ba54ca7d0981114ee6bc845 Oct 30 16:36:42.522176 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:42.522138 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7" event={"ID":"5be78108-87f2-41b7-bee1-0b404be3614b","Type":"ContainerStarted","Data":"e18d37bdd437248fb764b2bd429dc46ee7cebc1225a315dc75de2b246ce88b20"} Oct 30 16:36:42.522671 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:42.522195 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7" event={"ID":"5be78108-87f2-41b7-bee1-0b404be3614b","Type":"ContainerStarted","Data":"5501f7f71c00a5d1f83e7d51829b1524f53e87c45ba54ca7d0981114ee6bc845"} Oct 30 16:36:42.575040 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:42.574963 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7" podStartSLOduration=1.574942359 podStartE2EDuration="1.574942359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:42.567670303 +0000 UTC m=+916.278125046" watchObservedRunningTime="2026-10-30 16:36:42.574942359 +0000 UTC m=+916.285397104" Oct 30 16:36:42.685499 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:42.685462 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:43.530343 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:43.530305 6202 generic.go:356] "Generic (PLEG): container finished" podID="5be78108-87f2-41b7-bee1-0b404be3614b" containerID="e18d37bdd437248fb764b2bd429dc46ee7cebc1225a315dc75de2b246ce88b20" exitCode=0 Oct 30 16:36:43.530810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:43.530388 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7" event={"ID":"5be78108-87f2-41b7-bee1-0b404be3614b","Type":"ContainerDied","Data":"e18d37bdd437248fb764b2bd429dc46ee7cebc1225a315dc75de2b246ce88b20"} Oct 30 16:36:43.724599 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:43.724564 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:43.737046 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:43.737022 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:44.069301 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:36:44.069257 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:36:44.398609 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:44.398570 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:44.550939 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:44.550413 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:36:44.700048 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:44.700017 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7" Oct 30 16:36:44.725668 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:44.725638 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5be78108-87f2-41b7-bee1-0b404be3614b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be78108-87f2-41b7-bee1-0b404be3614b-maas-api-xks-ca\") pod \"5be78108-87f2-41b7-bee1-0b404be3614b\" (UID: \"5be78108-87f2-41b7-bee1-0b404be3614b\") " Oct 30 16:36:44.725856 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:44.725761 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be78108-87f2-41b7-bee1-0b404be3614b-kube-api-access-6kw84\" (UniqueName: \"kubernetes.io/projected/5be78108-87f2-41b7-bee1-0b404be3614b-kube-api-access-6kw84\") pod \"5be78108-87f2-41b7-bee1-0b404be3614b\" (UID: \"5be78108-87f2-41b7-bee1-0b404be3614b\") " Oct 30 16:36:44.725856 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:44.725820 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be78108-87f2-41b7-bee1-0b404be3614b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be78108-87f2-41b7-bee1-0b404be3614b-maas-api-openshift-ca\") pod \"5be78108-87f2-41b7-bee1-0b404be3614b\" (UID: \"5be78108-87f2-41b7-bee1-0b404be3614b\") " Oct 30 16:36:44.726212 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:44.726188 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be78108-87f2-41b7-bee1-0b404be3614b-maas-api-openshift-ca" pod "5be78108-87f2-41b7-bee1-0b404be3614b" (UID: "5be78108-87f2-41b7-bee1-0b404be3614b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:44.727706 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:44.727677 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5be78108-87f2-41b7-bee1-0b404be3614b-maas-api-xks-ca" pod "5be78108-87f2-41b7-bee1-0b404be3614b" (UID: "5be78108-87f2-41b7-bee1-0b404be3614b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:44.727910 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:44.727890 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be78108-87f2-41b7-bee1-0b404be3614b-kube-api-access-6kw84" pod "5be78108-87f2-41b7-bee1-0b404be3614b" (UID: "5be78108-87f2-41b7-bee1-0b404be3614b"). InnerVolumeSpecName "kube-api-access-6kw84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:44.827332 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:44.827296 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be78108-87f2-41b7-bee1-0b404be3614b-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:44.827332 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:44.827325 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kw84\" (UniqueName: \"kubernetes.io/projected/5be78108-87f2-41b7-bee1-0b404be3614b-kube-api-access-6kw84\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:44.827332 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:44.827335 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be78108-87f2-41b7-bee1-0b404be3614b-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:45.545246 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:45.545211 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7" Oct 30 16:36:45.545438 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:45.545246 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7" event={"ID":"5be78108-87f2-41b7-bee1-0b404be3614b","Type":"ContainerDied","Data":"5501f7f71c00a5d1f83e7d51829b1524f53e87c45ba54ca7d0981114ee6bc845"} Oct 30 16:36:45.545438 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:45.545304 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5501f7f71c00a5d1f83e7d51829b1524f53e87c45ba54ca7d0981114ee6bc845" Oct 30 16:36:45.832645 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:45.832614 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:45.945950 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:45.945915 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:46.270632 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:46.270550 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:47.053568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.053526 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl"] Oct 30 16:36:47.054200 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.054181 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5be78108-87f2-41b7-bee1-0b404be3614b" containerName="revoke-keys" Oct 30 16:36:47.054200 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.054200 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be78108-87f2-41b7-bee1-0b404be3614b" containerName="revoke-keys" Oct 30 16:36:47.054343 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.054334 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5be78108-87f2-41b7-bee1-0b404be3614b" containerName="revoke-keys" Oct 30 16:36:47.058044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.058028 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.092312 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.092284 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl"] Oct 30 16:36:47.153614 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.153584 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.153861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.153624 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.153861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.153801 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.153861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.153847 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.154080 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.153981 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.154080 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.154022 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrrz6\" (UniqueName: \"kubernetes.io/projected/8fa1e96c-f52b-4de4-8949-c22c45d6d842-kube-api-access-nrrz6\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.154202 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.154098 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.154202 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.154137 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.154202 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.154176 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.255524 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.255490 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.255524 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.255529 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.255721 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.255575 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.255721 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.255615 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrrz6\" (UniqueName: \"kubernetes.io/projected/8fa1e96c-f52b-4de4-8949-c22c45d6d842-kube-api-access-nrrz6\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.255721 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.255674 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.255721 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.255699 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.255940 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.255744 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.255940 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.255880 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.255940 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.255924 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.256214 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.256183 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.256387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.256369 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.256477 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.256398 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.256477 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.256455 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.256675 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.256658 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.258300 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.258271 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.258594 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.258574 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.278968 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.278948 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.280502 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.280484 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrrz6\" (UniqueName: \"kubernetes.io/projected/8fa1e96c-f52b-4de4-8949-c22c45d6d842-kube-api-access-nrrz6\") pod \"maas-default-gateway-openshift-default-76f4876667-57mzl\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.372462 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.372422 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:47.707520 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.707492 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl"] Oct 30 16:36:47.707882 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:36:47.707862 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa1e96c_f52b_4de4_8949_c22c45d6d842.slice/crio-455c909908141d760fe57b50b39d6b941b8a753bab098186440021f96c3e6e1e WatchSource:0}: Error finding container 455c909908141d760fe57b50b39d6b941b8a753bab098186440021f96c3e6e1e: Status 404 returned error can't find the container with id 455c909908141d760fe57b50b39d6b941b8a753bab098186440021f96c3e6e1e Oct 30 16:36:47.710894 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.710858 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:47.711006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.710938 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:47.711006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.710997 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:47.974435 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:47.974347 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:48.330192 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.330150 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7"] Oct 30 16:36:48.334291 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.334273 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.379710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.379396 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.379710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.379481 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.379710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.379513 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.379710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.379545 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.379710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.379647 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.379710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.379687 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.380034 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.379729 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4wxw\" (UniqueName: \"kubernetes.io/projected/78eb7d90-810f-48fb-aaaf-e54f2806d02a-kube-api-access-b4wxw\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.380034 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.379795 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.380034 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.379866 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.381440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.381412 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7"] Oct 30 16:36:48.481586 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.481548 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.481845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.481596 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.481845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.481624 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4wxw\" (UniqueName: \"kubernetes.io/projected/78eb7d90-810f-48fb-aaaf-e54f2806d02a-kube-api-access-b4wxw\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.481845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.481647 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.481845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.481691 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.481845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.481718 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.481845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.481825 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.482181 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.481854 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.482181 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.481883 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.482293 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.482259 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.482357 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.482307 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.482431 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.482413 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.482542 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.482520 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.482728 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.482709 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.484476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.484450 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.485324 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.485304 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.504897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.504870 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.505398 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.505380 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4wxw\" (UniqueName: \"kubernetes.io/projected/78eb7d90-810f-48fb-aaaf-e54f2806d02a-kube-api-access-b4wxw\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:48.574325 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.574284 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" event={"ID":"8fa1e96c-f52b-4de4-8949-c22c45d6d842","Type":"ContainerStarted","Data":"3e38c7fb3bd2a1ac8ac9b841a1908bacc03ecfd5e8f62b8d06db5776284b080c"} Oct 30 16:36:48.574544 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.574351 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" event={"ID":"8fa1e96c-f52b-4de4-8949-c22c45d6d842","Type":"ContainerStarted","Data":"455c909908141d760fe57b50b39d6b941b8a753bab098186440021f96c3e6e1e"} Oct 30 16:36:48.619684 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.619558 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" podStartSLOduration=1.6195441750000001 podStartE2EDuration="1.619544175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:48.618358777 +0000 UTC m=+922.328813530" watchObservedRunningTime="2026-10-30 16:36:48.619544175 +0000 UTC m=+922.329998916" Oct 30 16:36:48.689142 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:48.689108 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:49.030949 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.030917 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc"] Oct 30 16:36:49.032033 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:36:49.031979 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78eb7d90_810f_48fb_aaaf_e54f2806d02a.slice/crio-dd46a1d1a93592e460db7ff78b3f77e86fc78bf9ae46e89733a9cffaca42e41d WatchSource:0}: Error finding container dd46a1d1a93592e460db7ff78b3f77e86fc78bf9ae46e89733a9cffaca42e41d: Status 404 returned error can't find the container with id dd46a1d1a93592e460db7ff78b3f77e86fc78bf9ae46e89733a9cffaca42e41d Oct 30 16:36:49.033802 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.033751 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:49.033896 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.033830 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:49.033896 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.033858 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:49.036389 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.036375 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7"] Oct 30 16:36:49.036471 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.036461 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.080105 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.080065 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc"] Oct 30 16:36:49.090948 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.090702 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdbzn\" (UniqueName: \"kubernetes.io/projected/c3f813c4-823b-4109-a2e4-6027d5fabf2e-kube-api-access-kdbzn\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.090948 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.090833 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.090948 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.090876 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.090948 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.090901 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.091216 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.091002 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.091216 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.091036 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.091216 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.091059 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.091216 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.091099 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.091216 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.091156 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.192584 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.192553 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbzn\" (UniqueName: \"kubernetes.io/projected/c3f813c4-823b-4109-a2e4-6027d5fabf2e-kube-api-access-kdbzn\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.192712 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.192617 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.192712 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.192642 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.192712 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.192658 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.192712 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.192704 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.192860 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.192725 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.192860 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.192741 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.192860 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.192780 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.192860 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.192817 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.193141 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.193128 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.194466 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.193752 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.194466 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.194013 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.194466 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.194178 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.194466 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.194230 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.196102 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.196077 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.196841 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.196820 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.216069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.216047 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdbzn\" (UniqueName: \"kubernetes.io/projected/c3f813c4-823b-4109-a2e4-6027d5fabf2e-kube-api-access-kdbzn\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.217251 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.217226 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.380796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.375334 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:49.380796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.375464 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-57mzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:36:49.380796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.375505 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" podUID="8fa1e96c-f52b-4de4-8949-c22c45d6d842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 16:36:49.420031 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.419981 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:49.614948 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.614872 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" event={"ID":"78eb7d90-810f-48fb-aaaf-e54f2806d02a","Type":"ContainerStarted","Data":"c5fe7f2711a9a958a8ac8eea0ee3fa528100114e6a79c5c74d1bcbc8ff663503"} Oct 30 16:36:49.615227 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.615206 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" event={"ID":"78eb7d90-810f-48fb-aaaf-e54f2806d02a","Type":"ContainerStarted","Data":"dd46a1d1a93592e460db7ff78b3f77e86fc78bf9ae46e89733a9cffaca42e41d"} Oct 30 16:36:49.673961 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.673671 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" podStartSLOduration=1.6736536960000001 podStartE2EDuration="1.673653696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:49.66225176 +0000 UTC m=+923.372706518" watchObservedRunningTime="2026-10-30 16:36:49.673653696 +0000 UTC m=+923.384108844" Oct 30 16:36:49.689960 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.689922 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:49.786504 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.786461 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc"] Oct 30 16:36:49.788267 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:36:49.788243 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f813c4_823b_4109_a2e4_6027d5fabf2e.slice/crio-c588e7eee015f8505ffbf53bdf1910947a5e92f08925efe7a4230607bfc8606e WatchSource:0}: Error finding container c588e7eee015f8505ffbf53bdf1910947a5e92f08925efe7a4230607bfc8606e: Status 404 returned error can't find the container with id c588e7eee015f8505ffbf53bdf1910947a5e92f08925efe7a4230607bfc8606e Oct 30 16:36:49.791584 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.791544 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:49.791657 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.791628 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:49.791690 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.791673 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:49.838311 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:49.838282 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:50.388794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:50.386324 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-57mzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:36:50.388794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:50.386382 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" podUID="8fa1e96c-f52b-4de4-8949-c22c45d6d842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 16:36:50.627247 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:50.626793 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" event={"ID":"c3f813c4-823b-4109-a2e4-6027d5fabf2e","Type":"ContainerStarted","Data":"7a41858953788d1abeb4a2f05bd7d09017e4d57bca1a7bdc67ddf57e85f4fd5c"} Oct 30 16:36:50.627247 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:50.626846 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" event={"ID":"c3f813c4-823b-4109-a2e4-6027d5fabf2e","Type":"ContainerStarted","Data":"c588e7eee015f8505ffbf53bdf1910947a5e92f08925efe7a4230607bfc8606e"} Oct 30 16:36:50.695284 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:50.695182 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:36:50.695284 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:50.695248 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" podUID="78eb7d90-810f-48fb-aaaf-e54f2806d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 16:36:51.379219 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.379177 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-57mzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:36:51.379528 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.379500 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" podUID="8fa1e96c-f52b-4de4-8949-c22c45d6d842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 16:36:51.421749 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.421718 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:51.422444 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.422363 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:51.422585 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.422564 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" podUID="c3f813c4-823b-4109-a2e4-6027d5fabf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 16:36:51.517616 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.517551 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" podStartSLOduration=2.5175315510000003 podStartE2EDuration="2.517531551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:50.681974934 +0000 UTC m=+924.392429689" watchObservedRunningTime="2026-10-30 16:36:51.517531551 +0000 UTC m=+925.227986296" Oct 30 16:36:51.526292 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.526261 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85"] Oct 30 16:36:51.531803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.531779 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85" Oct 30 16:36:51.547838 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.546757 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r77jf\" (UniqueName: \"kubernetes.io/projected/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-kube-api-access-r77jf\") pod \"maas-api-revoke-sub-8dccf310-w9f85\" (UID: \"88ef6c68-ddb8-4e03-afbf-d8a908183cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85" Oct 30 16:36:51.547838 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.546856 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dccf310-w9f85\" (UID: \"88ef6c68-ddb8-4e03-afbf-d8a908183cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85" Oct 30 16:36:51.547838 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.546915 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dccf310-w9f85\" (UID: \"88ef6c68-ddb8-4e03-afbf-d8a908183cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85" Oct 30 16:36:51.555794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.550965 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:36:51.555794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.551058 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:36:51.556257 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.556138 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:36:51.556257 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.556230 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:36:51.667865 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.660491 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r77jf\" (UniqueName: \"kubernetes.io/projected/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-kube-api-access-r77jf\") pod \"maas-api-revoke-sub-8dccf310-w9f85\" (UID: \"88ef6c68-ddb8-4e03-afbf-d8a908183cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85" Oct 30 16:36:51.692891 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.692855 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dccf310-w9f85\" (UID: \"88ef6c68-ddb8-4e03-afbf-d8a908183cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85" Oct 30 16:36:51.693205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.693181 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dccf310-w9f85\" (UID: \"88ef6c68-ddb8-4e03-afbf-d8a908183cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85" Oct 30 16:36:51.704802 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.694588 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:36:51.704802 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.694638 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" podUID="78eb7d90-810f-48fb-aaaf-e54f2806d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 16:36:51.704802 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.695683 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dccf310-w9f85\" (UID: \"88ef6c68-ddb8-4e03-afbf-d8a908183cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85" Oct 30 16:36:51.719890 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.719806 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85"] Oct 30 16:36:51.841016 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.840944 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r77jf\" (UniqueName: \"kubernetes.io/projected/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-kube-api-access-r77jf\") pod \"maas-api-revoke-sub-8dccf310-w9f85\" (UID: \"88ef6c68-ddb8-4e03-afbf-d8a908183cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85" Oct 30 16:36:51.869873 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:51.865083 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dccf310-w9f85\" (UID: \"88ef6c68-ddb8-4e03-afbf-d8a908183cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85" Oct 30 16:36:52.087304 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:52.086970 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85" Oct 30 16:36:52.272855 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:52.268030 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:52.375860 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:52.375379 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-57mzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:36:52.375860 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:52.375446 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" podUID="8fa1e96c-f52b-4de4-8949-c22c45d6d842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 16:36:52.428081 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:52.428045 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:52.468720 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:52.468686 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" podUID="c3f813c4-823b-4109-a2e4-6027d5fabf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 16:36:52.760822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:52.722434 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:36:52.948788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:52.936461 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" podUID="78eb7d90-810f-48fb-aaaf-e54f2806d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 16:36:53.198894 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:53.198308 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85"] Oct 30 16:36:53.415794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:53.393886 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-57mzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:36:53.415794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:53.393958 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" podUID="8fa1e96c-f52b-4de4-8949-c22c45d6d842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 16:36:53.458037 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:53.457902 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:53.458619 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:53.458587 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" podUID="c3f813c4-823b-4109-a2e4-6027d5fabf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 16:36:53.582542 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:53.582492 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:53.604395 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:53.582755 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:53.700466 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:53.700434 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:36:53.700805 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:53.700742 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" podUID="78eb7d90-810f-48fb-aaaf-e54f2806d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 16:36:53.712183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:53.711840 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 16:36:53.723612 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:53.723107 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:53.909915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:53.909847 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:54.022241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:54.022145 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85" event={"ID":"88ef6c68-ddb8-4e03-afbf-d8a908183cc2","Type":"ContainerStarted","Data":"ae6394a14d9e73813d9287c78d9c7c77ad320d72d56476620809911190694a98"} Oct 30 16:36:54.022450 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:54.022430 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85" event={"ID":"88ef6c68-ddb8-4e03-afbf-d8a908183cc2","Type":"ContainerStarted","Data":"28ec1218b9befd7ca6c960f5ceebd3f02f806e88588d8c77cf2d50ddba2bf9d3"} Oct 30 16:36:54.103798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:54.094324 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85" podStartSLOduration=3.094303817 podStartE2EDuration="3.094303817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:54.094083271 +0000 UTC m=+927.804538018" watchObservedRunningTime="2026-10-30 16:36:54.094303817 +0000 UTC m=+927.804758561" Oct 30 16:36:54.212803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:54.212735 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:54.377902 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:54.377866 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-57mzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:36:54.378149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:54.378120 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" podUID="8fa1e96c-f52b-4de4-8949-c22c45d6d842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 16:36:54.434821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:54.424356 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:54.434821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:54.424411 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" podUID="c3f813c4-823b-4109-a2e4-6027d5fabf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 16:36:54.693846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:54.693220 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:36:54.693846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:54.693349 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" podUID="78eb7d90-810f-48fb-aaaf-e54f2806d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 16:36:55.144917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:55.144411 6202 generic.go:356] "Generic (PLEG): container finished" podID="88ef6c68-ddb8-4e03-afbf-d8a908183cc2" containerID="ae6394a14d9e73813d9287c78d9c7c77ad320d72d56476620809911190694a98" exitCode=0 Oct 30 16:36:55.144917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:55.144461 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85" event={"ID":"88ef6c68-ddb8-4e03-afbf-d8a908183cc2","Type":"ContainerDied","Data":"ae6394a14d9e73813d9287c78d9c7c77ad320d72d56476620809911190694a98"} Oct 30 16:36:55.378478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:55.378446 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-57mzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:36:55.378786 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:55.378732 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" podUID="8fa1e96c-f52b-4de4-8949-c22c45d6d842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 16:36:55.424339 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:55.424245 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:55.424578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:55.424543 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" podUID="c3f813c4-823b-4109-a2e4-6027d5fabf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 16:36:55.625914 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:55.625866 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:55.691470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:55.691352 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:36:55.691744 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:55.691696 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" podUID="78eb7d90-810f-48fb-aaaf-e54f2806d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 16:36:55.699914 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:55.699862 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:55.850798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:55.850016 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 16:36:56.423612 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:56.423210 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:56.423612 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:56.423268 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" podUID="c3f813c4-823b-4109-a2e4-6027d5fabf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 16:36:56.487619 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:56.487589 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85" Oct 30 16:36:56.577957 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:56.577907 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-kube-api-access-r77jf\" (UniqueName: \"kubernetes.io/projected/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-kube-api-access-r77jf\") pod \"88ef6c68-ddb8-4e03-afbf-d8a908183cc2\" (UID: \"88ef6c68-ddb8-4e03-afbf-d8a908183cc2\") " Oct 30 16:36:56.578425 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:56.578392 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-maas-api-xks-ca\") pod \"88ef6c68-ddb8-4e03-afbf-d8a908183cc2\" (UID: \"88ef6c68-ddb8-4e03-afbf-d8a908183cc2\") " Oct 30 16:36:56.578798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:56.578755 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-maas-api-openshift-ca\") pod \"88ef6c68-ddb8-4e03-afbf-d8a908183cc2\" (UID: \"88ef6c68-ddb8-4e03-afbf-d8a908183cc2\") " Oct 30 16:36:56.580701 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:56.580679 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-maas-api-openshift-ca" pod "88ef6c68-ddb8-4e03-afbf-d8a908183cc2" (UID: "88ef6c68-ddb8-4e03-afbf-d8a908183cc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:56.583808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:56.583746 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-maas-api-xks-ca" pod "88ef6c68-ddb8-4e03-afbf-d8a908183cc2" (UID: "88ef6c68-ddb8-4e03-afbf-d8a908183cc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:56.584419 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:56.584385 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-kube-api-access-r77jf" pod "88ef6c68-ddb8-4e03-afbf-d8a908183cc2" (UID: "88ef6c68-ddb8-4e03-afbf-d8a908183cc2"). InnerVolumeSpecName "kube-api-access-r77jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:56.684856 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:56.683759 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:56.684856 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:56.683832 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r77jf\" (UniqueName: \"kubernetes.io/projected/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-kube-api-access-r77jf\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:56.684856 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:56.683846 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ef6c68-ddb8-4e03-afbf-d8a908183cc2-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:36:56.711302 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:56.711268 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:57.191524 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:57.191484 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85" event={"ID":"88ef6c68-ddb8-4e03-afbf-d8a908183cc2","Type":"ContainerDied","Data":"28ec1218b9befd7ca6c960f5ceebd3f02f806e88588d8c77cf2d50ddba2bf9d3"} Oct 30 16:36:57.191821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:57.191568 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85" Oct 30 16:36:57.192056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:57.191760 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28ec1218b9befd7ca6c960f5ceebd3f02f806e88588d8c77cf2d50ddba2bf9d3" Oct 30 16:36:57.394205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:57.394136 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-57mzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:36:57.394400 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:57.394247 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" podUID="8fa1e96c-f52b-4de4-8949-c22c45d6d842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:57.442706 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:57.442618 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:57.443982 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:57.443959 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:57.445110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:57.445069 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:36:57.733026 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:57.732933 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:36:57.733026 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:57.733008 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" podUID="78eb7d90-810f-48fb-aaaf-e54f2806d02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:58.111745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:58.111705 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:58.112277 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:58.112253 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:58.113005 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:58.112986 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:36:58.399209 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:58.399115 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:58.399903 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:58.399866 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:58.401478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:58.401458 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:36:58.486168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:58.486139 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:59.229167 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.229140 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk"] Oct 30 16:36:59.229678 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.229665 6202 cpu_manager.go:414] "containerMap: removing container" podUID="88ef6c68-ddb8-4e03-afbf-d8a908183cc2" containerName="revoke-keys" Oct 30 16:36:59.229717 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.229678 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88ef6c68-ddb8-4e03-afbf-d8a908183cc2" containerName="revoke-keys" Oct 30 16:36:59.229797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.229777 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="88ef6c68-ddb8-4e03-afbf-d8a908183cc2" containerName="revoke-keys" Oct 30 16:36:59.233265 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.233251 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.271818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.271761 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-263f00\"/\"openshift-service-ca.crt\"" Oct 30 16:36:59.272036 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.271982 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-263f00\"/\"kube-root-ca.crt\"" Oct 30 16:36:59.272115 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.272063 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-263f00\"/\"default-dockercfg-b54g7\"" Oct 30 16:36:59.272222 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.272163 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-263f00\"/\"distinct-w4-263f00-kserve-self-signed-certs\"" Oct 30 16:36:59.275167 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.275142 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk"] Oct 30 16:36:59.320150 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.320119 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:59.326825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.326805 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-tls-certs\") pod \"distinct-w4-263f00-kserve-78c874f856-nbqfk\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.326933 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.326844 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-home\") pod \"distinct-w4-263f00-kserve-78c874f856-nbqfk\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.326984 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.326936 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx29h\" (UniqueName: \"kubernetes.io/projected/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-kube-api-access-gx29h\") pod \"distinct-w4-263f00-kserve-78c874f856-nbqfk\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.327069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.327053 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-dshm\") pod \"distinct-w4-263f00-kserve-78c874f856-nbqfk\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.327108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.327095 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-model-cache\") pod \"distinct-w4-263f00-kserve-78c874f856-nbqfk\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.327216 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.327200 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-tmp-dir\") pod \"distinct-w4-263f00-kserve-78c874f856-nbqfk\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.428377 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.428341 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-dshm\") pod \"distinct-w4-263f00-kserve-78c874f856-nbqfk\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.428539 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.428401 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-model-cache\") pod \"distinct-w4-263f00-kserve-78c874f856-nbqfk\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.428610 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.428585 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-tmp-dir\") pod \"distinct-w4-263f00-kserve-78c874f856-nbqfk\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.428694 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.428678 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-tls-certs\") pod \"distinct-w4-263f00-kserve-78c874f856-nbqfk\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.428751 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.428694 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-model-cache\") pod \"distinct-w4-263f00-kserve-78c874f856-nbqfk\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.428751 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.428744 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-home\") pod \"distinct-w4-263f00-kserve-78c874f856-nbqfk\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.428917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.428820 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx29h\" (UniqueName: \"kubernetes.io/projected/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-kube-api-access-gx29h\") pod \"distinct-w4-263f00-kserve-78c874f856-nbqfk\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.428978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.428959 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-tmp-dir\") pod \"distinct-w4-263f00-kserve-78c874f856-nbqfk\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.429154 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.429136 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-home\") pod \"distinct-w4-263f00-kserve-78c874f856-nbqfk\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.430844 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.430822 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-dshm\") pod \"distinct-w4-263f00-kserve-78c874f856-nbqfk\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.431083 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.431067 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-tls-certs\") pod \"distinct-w4-263f00-kserve-78c874f856-nbqfk\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.452264 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.452235 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx29h\" (UniqueName: \"kubernetes.io/projected/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-kube-api-access-gx29h\") pod \"distinct-w4-263f00-kserve-78c874f856-nbqfk\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.545517 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.545430 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:36:59.855345 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:36:59.855318 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod940f627d_3d5f_4c5d_9b6b_4ffd5b5502d9.slice/crio-feb44cc899530f952331e1ddd500f73cddc22a678904bfff2eeb6b56e6140035 WatchSource:0}: Error finding container feb44cc899530f952331e1ddd500f73cddc22a678904bfff2eeb6b56e6140035: Status 404 returned error can't find the container with id feb44cc899530f952331e1ddd500f73cddc22a678904bfff2eeb6b56e6140035 Oct 30 16:36:59.855435 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:36:59.855381 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk"] Oct 30 16:37:00.097642 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:00.097605 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:00.221896 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:00.221793 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" event={"ID":"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9","Type":"ContainerStarted","Data":"34ef19b24a07f4654236190370b147b1a28f4fb8904156664a0ebe669ecee82d"} Oct 30 16:37:00.221896 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:00.221840 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" event={"ID":"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9","Type":"ContainerStarted","Data":"feb44cc899530f952331e1ddd500f73cddc22a678904bfff2eeb6b56e6140035"} Oct 30 16:37:00.263820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:00.263752 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podStartSLOduration=1.263739601 podStartE2EDuration="1.263739601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:00.26203285 +0000 UTC m=+933.972487595" watchObservedRunningTime="2026-10-30 16:37:00.263739601 +0000 UTC m=+933.974194344" Oct 30 16:37:00.545606 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:00.545518 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:37:01.762284 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:01.762253 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:03.441457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:03.441425 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:05.064148 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:05.064114 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:06.883865 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:06.883835 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:08.568351 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:08.568320 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:09.546324 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:09.546285 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:37:09.560566 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:09.560538 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:37:10.169322 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:10.169286 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:10.321976 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:10.321945 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:37:11.736741 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:11.736707 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:13.229639 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:13.229602 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:14.068014 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:37:14.067937 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:37:14.204965 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.204932 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq"] Oct 30 16:37:14.209715 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.209687 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.229825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.229801 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq"] Oct 30 16:37:14.237504 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.237478 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0b2348\"/\"distinct-2-w3-0b2348-kserve-self-signed-certs\"" Oct 30 16:37:14.274140 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.274112 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:14.321098 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.321013 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-tmp-dir\") pod \"distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.321098 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.321093 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-home\") pod \"distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.321317 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.321164 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfj9l\" (UniqueName: \"kubernetes.io/projected/afe32415-d7f2-480d-9035-45509a784874-kube-api-access-lfj9l\") pod \"distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.321317 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.321237 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe32415-d7f2-480d-9035-45509a784874-tls-certs\") pod \"distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.321317 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.321267 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-model-cache\") pod \"distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.321474 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.321379 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-dshm\") pod \"distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.422541 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.422495 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfj9l\" (UniqueName: \"kubernetes.io/projected/afe32415-d7f2-480d-9035-45509a784874-kube-api-access-lfj9l\") pod \"distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.423222 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.423195 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe32415-d7f2-480d-9035-45509a784874-tls-certs\") pod \"distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.423823 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.423803 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-model-cache\") pod \"distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.424067 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.424052 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-dshm\") pod \"distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.424877 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.424858 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-tmp-dir\") pod \"distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.425054 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.425039 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-home\") pod \"distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.425500 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.425483 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-home\") pod \"distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.426493 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.424698 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-model-cache\") pod \"distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.426654 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.425849 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-tmp-dir\") pod \"distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.451845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.451811 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfj9l\" (UniqueName: \"kubernetes.io/projected/afe32415-d7f2-480d-9035-45509a784874-kube-api-access-lfj9l\") pod \"distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.458254 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.458227 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe32415-d7f2-480d-9035-45509a784874-tls-certs\") pod \"distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.460643 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.460616 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-dshm\") pod \"distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.528042 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.528006 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:14.825326 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:37:14.825300 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafe32415_d7f2_480d_9035_45509a784874.slice/crio-7058cad8da1f4880cbac4b29ea21c271e44f3d078330adcdc08e4c31cbb8d6f8 WatchSource:0}: Error finding container 7058cad8da1f4880cbac4b29ea21c271e44f3d078330adcdc08e4c31cbb8d6f8: Status 404 returned error can't find the container with id 7058cad8da1f4880cbac4b29ea21c271e44f3d078330adcdc08e4c31cbb8d6f8 Oct 30 16:37:14.825326 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:14.825302 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq"] Oct 30 16:37:15.012857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:15.012824 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:15.347373 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:15.347339 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" event={"ID":"afe32415-d7f2-480d-9035-45509a784874","Type":"ContainerStarted","Data":"d4547bf9d19c6e8d191af019bcc3e3aac0c0ff2280ee7b5a432f5886f396f170"} Oct 30 16:37:15.347898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:15.347382 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" event={"ID":"afe32415-d7f2-480d-9035-45509a784874","Type":"ContainerStarted","Data":"7058cad8da1f4880cbac4b29ea21c271e44f3d078330adcdc08e4c31cbb8d6f8"} Oct 30 16:37:15.391665 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:15.391615 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podStartSLOduration=1.391600941 podStartE2EDuration="1.391600941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:15.389068799 +0000 UTC m=+949.099523542" watchObservedRunningTime="2026-10-30 16:37:15.391600941 +0000 UTC m=+949.102055683" Oct 30 16:37:15.528206 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:15.528174 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:16.619289 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:16.619252 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:18.103982 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:18.103937 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:19.041188 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.041159 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm"] Oct 30 16:37:19.045545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.045528 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.081401 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.081372 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm"] Oct 30 16:37:19.184299 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.184263 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.184748 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.184304 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.184748 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.184480 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.184748 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.184533 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afba1ef-8cb5-456c-8725-10ef761e3524-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.184748 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.184631 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.184748 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.184661 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ktj\" (UniqueName: \"kubernetes.io/projected/3afba1ef-8cb5-456c-8725-10ef761e3524-kube-api-access-s4ktj\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.184748 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.184700 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.184967 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.184839 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.184967 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.184875 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.285594 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.285563 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.285594 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.285597 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.285859 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.285637 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.285859 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.285660 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.285859 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.285737 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.285859 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.285795 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afba1ef-8cb5-456c-8725-10ef761e3524-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.286032 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.285860 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.286032 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.285896 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ktj\" (UniqueName: \"kubernetes.io/projected/3afba1ef-8cb5-456c-8725-10ef761e3524-kube-api-access-s4ktj\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.286032 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.285940 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.286295 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.286267 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.286612 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.286551 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afba1ef-8cb5-456c-8725-10ef761e3524-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.286738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.286717 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.286834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.286757 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.287006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.286982 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.288338 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.288310 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.288894 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.288877 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.310396 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.310337 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ktj\" (UniqueName: \"kubernetes.io/projected/3afba1ef-8cb5-456c-8725-10ef761e3524-kube-api-access-s4ktj\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.310396 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.310355 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.364019 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.363971 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:19.676533 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:37:19.676503 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afba1ef_8cb5_456c_8725_10ef761e3524.slice/crio-fbdd9b46ce590b260fe3f10c7655ad95bf52f80e08bfa3db0aefea5d28225a9e WatchSource:0}: Error finding container fbdd9b46ce590b260fe3f10c7655ad95bf52f80e08bfa3db0aefea5d28225a9e: Status 404 returned error can't find the container with id fbdd9b46ce590b260fe3f10c7655ad95bf52f80e08bfa3db0aefea5d28225a9e Oct 30 16:37:19.677738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.677663 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm"] Oct 30 16:37:19.679686 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.679646 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:37:19.679758 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.679731 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:37:19.679830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.679791 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:37:19.864370 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:19.864330 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:20.388429 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:20.388389 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" event={"ID":"3afba1ef-8cb5-456c-8725-10ef761e3524","Type":"ContainerStarted","Data":"b72b30c3979e5948306ce9748d3d383821063e8e09947f363aee1be8d293c72b"} Oct 30 16:37:20.388855 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:20.388451 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" event={"ID":"3afba1ef-8cb5-456c-8725-10ef761e3524","Type":"ContainerStarted","Data":"fbdd9b46ce590b260fe3f10c7655ad95bf52f80e08bfa3db0aefea5d28225a9e"} Oct 30 16:37:20.469403 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:20.469335 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" podStartSLOduration=1.469318857 podStartE2EDuration="1.469318857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:20.450212713 +0000 UTC m=+954.160667456" watchObservedRunningTime="2026-10-30 16:37:20.469318857 +0000 UTC m=+954.179773600" Oct 30 16:37:21.375792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:21.369608 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:21.375792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:21.369826 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:37:21.375792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:21.369885 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" podUID="3afba1ef-8cb5-456c-8725-10ef761e3524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 16:37:21.568791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:21.568108 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:21.720846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:21.720673 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:22.365788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:22.365183 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:37:22.365788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:22.365259 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" podUID="3afba1ef-8cb5-456c-8725-10ef761e3524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 16:37:23.369793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:23.367134 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:37:23.369793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:23.367212 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" podUID="3afba1ef-8cb5-456c-8725-10ef761e3524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 16:37:23.860791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:23.860221 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:24.366792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:24.365157 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:37:24.366792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:24.365245 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" podUID="3afba1ef-8cb5-456c-8725-10ef761e3524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 16:37:24.528651 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:24.528609 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:24.569601 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:24.569295 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:24.669385 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:24.669268 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:25.369198 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:25.369161 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:37:25.369444 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:25.369409 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" podUID="3afba1ef-8cb5-456c-8725-10ef761e3524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 16:37:25.470162 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:25.470110 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:25.480903 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:25.480872 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:37:27.056195 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:27.056163 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:27.056820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:27.056793 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:27.061116 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:27.061080 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:37:27.098036 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:27.098009 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:27.922299 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:27.922264 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:28.875494 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:28.875457 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:30.344867 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:30.344826 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:31.960030 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:31.959972 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:33.449937 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:33.449907 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:35.073553 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:35.073518 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:36.756791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:36.755972 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:36.913787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:36.913732 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:37.063473 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:37.063382 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:38.786090 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:38.786057 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:40.489222 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:40.489179 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:42.061454 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:42.061417 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:43.228445 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:43.228413 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:43.988080 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:43.988053 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:44.068573 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:37:44.068536 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:37:45.560703 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:45.560662 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:47.436879 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:47.436833 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:48.802247 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:48.802213 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg"] Oct 30 16:37:48.806399 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:48.806374 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:48.837826 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:48.837797 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg"] Oct 30 16:37:48.912545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:48.912510 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:48.912711 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:48.912564 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:48.912711 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:48.912637 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:48.912806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:48.912721 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcd7t\" (UniqueName: \"kubernetes.io/projected/ec24f992-d313-4d6d-987d-98c4c8361df8-kube-api-access-bcd7t\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:48.912806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:48.912787 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:48.912874 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:48.912865 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:48.912925 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:48.912912 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:48.912987 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:48.912977 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:48.913094 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:48.913062 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec24f992-d313-4d6d-987d-98c4c8361df8-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:48.975319 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:48.975273 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:49.014718 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.014679 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.015614 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.015575 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.015818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.015674 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcd7t\" (UniqueName: \"kubernetes.io/projected/ec24f992-d313-4d6d-987d-98c4c8361df8-kube-api-access-bcd7t\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.015818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.015729 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.016002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.015825 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.016002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.015872 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.016002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.015998 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.016186 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.016060 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec24f992-d313-4d6d-987d-98c4c8361df8-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.016186 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.016137 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.017237 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.017212 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.017473 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.017453 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.017731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.017707 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.017983 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.017964 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.018560 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.018537 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec24f992-d313-4d6d-987d-98c4c8361df8-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.019621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.019594 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.020068 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.020046 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.052866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.040845 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcd7t\" (UniqueName: \"kubernetes.io/projected/ec24f992-d313-4d6d-987d-98c4c8361df8-kube-api-access-bcd7t\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.052866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.041152 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.120435 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.120409 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:49.438860 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:37:49.438816 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec24f992_d313_4d6d_987d_98c4c8361df8.slice/crio-c697988d07bd9e2d04a3baaa38806c768bed35c78552947c9e3ea5c473efc950 WatchSource:0}: Error finding container c697988d07bd9e2d04a3baaa38806c768bed35c78552947c9e3ea5c473efc950: Status 404 returned error can't find the container with id c697988d07bd9e2d04a3baaa38806c768bed35c78552947c9e3ea5c473efc950 Oct 30 16:37:49.438987 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.438963 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg"] Oct 30 16:37:49.441799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.441744 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:37:49.441898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.441847 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:37:49.441898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.441890 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:37:49.636651 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.636616 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" event={"ID":"ec24f992-d313-4d6d-987d-98c4c8361df8","Type":"ContainerStarted","Data":"32bafbb010b706a8dc04ec9b5b727d70a5946780147d60e6acbd9924c36cde66"} Oct 30 16:37:49.636855 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.636671 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" event={"ID":"ec24f992-d313-4d6d-987d-98c4c8361df8","Type":"ContainerStarted","Data":"c697988d07bd9e2d04a3baaa38806c768bed35c78552947c9e3ea5c473efc950"} Oct 30 16:37:49.683930 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.683878 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" podStartSLOduration=1.683860694 podStartE2EDuration="1.683860694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:49.680524947 +0000 UTC m=+983.390979692" watchObservedRunningTime="2026-10-30 16:37:49.683860694 +0000 UTC m=+983.394315437" Oct 30 16:37:49.746241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.746170 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf"] Oct 30 16:37:49.751652 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.751630 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf" Oct 30 16:37:49.767757 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.767713 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:37:49.768104 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.768085 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:37:49.776157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.771496 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:37:49.776157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.771669 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:37:49.776157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.772814 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf"] Oct 30 16:37:49.827219 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.827190 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm8dg\" (UniqueName: \"kubernetes.io/projected/006b1304-8f0c-4891-a71d-2447e6f77298-kube-api-access-jm8dg\") pod \"maas-api-revoke-sub-d3655964-nqhmf\" (UID: \"006b1304-8f0c-4891-a71d-2447e6f77298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf" Oct 30 16:37:49.827572 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.827314 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/006b1304-8f0c-4891-a71d-2447e6f77298-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3655964-nqhmf\" (UID: \"006b1304-8f0c-4891-a71d-2447e6f77298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf" Oct 30 16:37:49.827572 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.827420 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/006b1304-8f0c-4891-a71d-2447e6f77298-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3655964-nqhmf\" (UID: \"006b1304-8f0c-4891-a71d-2447e6f77298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf" Oct 30 16:37:49.929222 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.929186 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm8dg\" (UniqueName: \"kubernetes.io/projected/006b1304-8f0c-4891-a71d-2447e6f77298-kube-api-access-jm8dg\") pod \"maas-api-revoke-sub-d3655964-nqhmf\" (UID: \"006b1304-8f0c-4891-a71d-2447e6f77298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf" Oct 30 16:37:49.929429 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.929298 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/006b1304-8f0c-4891-a71d-2447e6f77298-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3655964-nqhmf\" (UID: \"006b1304-8f0c-4891-a71d-2447e6f77298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf" Oct 30 16:37:49.929429 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.929358 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/006b1304-8f0c-4891-a71d-2447e6f77298-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3655964-nqhmf\" (UID: \"006b1304-8f0c-4891-a71d-2447e6f77298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf" Oct 30 16:37:49.930177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.930150 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/006b1304-8f0c-4891-a71d-2447e6f77298-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3655964-nqhmf\" (UID: \"006b1304-8f0c-4891-a71d-2447e6f77298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf" Oct 30 16:37:49.931950 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.931924 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/006b1304-8f0c-4891-a71d-2447e6f77298-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3655964-nqhmf\" (UID: \"006b1304-8f0c-4891-a71d-2447e6f77298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf" Oct 30 16:37:49.952780 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:49.952745 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm8dg\" (UniqueName: \"kubernetes.io/projected/006b1304-8f0c-4891-a71d-2447e6f77298-kube-api-access-jm8dg\") pod \"maas-api-revoke-sub-d3655964-nqhmf\" (UID: \"006b1304-8f0c-4891-a71d-2447e6f77298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf" Oct 30 16:37:50.066236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:50.066154 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf" Oct 30 16:37:50.120784 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:50.120741 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:50.123160 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:50.123134 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:37:50.123290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:50.123203 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" podUID="ec24f992-d313-4d6d-987d-98c4c8361df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 16:37:50.378071 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:50.378043 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf"] Oct 30 16:37:50.521391 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:50.521358 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:50.676987 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:50.676860 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf" event={"ID":"006b1304-8f0c-4891-a71d-2447e6f77298","Type":"ContainerStarted","Data":"f7347820d95284b10875b3db0968eeb7c53b5baaaf8cbfdca453e15eb9ad1cf2"} Oct 30 16:37:51.120993 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:51.120952 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:37:51.121381 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:51.121016 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" podUID="ec24f992-d313-4d6d-987d-98c4c8361df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 16:37:51.686211 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:51.686174 6202 generic.go:356] "Generic (PLEG): container finished" podID="006b1304-8f0c-4891-a71d-2447e6f77298" containerID="3efd4ceb5d9a5308129739f3bb69d0c6b7db66579599957de15bb5ee5d4fcefb" exitCode=0 Oct 30 16:37:51.686367 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:51.686214 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf" event={"ID":"006b1304-8f0c-4891-a71d-2447e6f77298","Type":"ContainerDied","Data":"3efd4ceb5d9a5308129739f3bb69d0c6b7db66579599957de15bb5ee5d4fcefb"} Oct 30 16:37:52.130622 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:52.129271 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:37:52.130622 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:52.129347 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" podUID="ec24f992-d313-4d6d-987d-98c4c8361df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 16:37:52.304845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:52.304138 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:53.130788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:53.126238 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:37:53.130788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:53.126294 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" podUID="ec24f992-d313-4d6d-987d-98c4c8361df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 16:37:53.139470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:53.139163 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf" Oct 30 16:37:53.234907 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:53.234851 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/006b1304-8f0c-4891-a71d-2447e6f77298-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/006b1304-8f0c-4891-a71d-2447e6f77298-maas-api-openshift-ca\") pod \"006b1304-8f0c-4891-a71d-2447e6f77298\" (UID: \"006b1304-8f0c-4891-a71d-2447e6f77298\") " Oct 30 16:37:53.235224 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:53.235200 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006b1304-8f0c-4891-a71d-2447e6f77298-kube-api-access-jm8dg\" (UniqueName: \"kubernetes.io/projected/006b1304-8f0c-4891-a71d-2447e6f77298-kube-api-access-jm8dg\") pod \"006b1304-8f0c-4891-a71d-2447e6f77298\" (UID: \"006b1304-8f0c-4891-a71d-2447e6f77298\") " Oct 30 16:37:53.235353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:53.235335 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/006b1304-8f0c-4891-a71d-2447e6f77298-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/006b1304-8f0c-4891-a71d-2447e6f77298-maas-api-xks-ca\") pod \"006b1304-8f0c-4891-a71d-2447e6f77298\" (UID: \"006b1304-8f0c-4891-a71d-2447e6f77298\") " Oct 30 16:37:53.235640 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:53.235613 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006b1304-8f0c-4891-a71d-2447e6f77298-maas-api-openshift-ca" pod "006b1304-8f0c-4891-a71d-2447e6f77298" (UID: "006b1304-8f0c-4891-a71d-2447e6f77298"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:37:53.236006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:53.235963 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/006b1304-8f0c-4891-a71d-2447e6f77298-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:37:53.314792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:53.303915 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/006b1304-8f0c-4891-a71d-2447e6f77298-maas-api-xks-ca" pod "006b1304-8f0c-4891-a71d-2447e6f77298" (UID: "006b1304-8f0c-4891-a71d-2447e6f77298"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:37:53.314792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:53.304074 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006b1304-8f0c-4891-a71d-2447e6f77298-kube-api-access-jm8dg" pod "006b1304-8f0c-4891-a71d-2447e6f77298" (UID: "006b1304-8f0c-4891-a71d-2447e6f77298"). InnerVolumeSpecName "kube-api-access-jm8dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:37:53.415792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:53.397103 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm8dg\" (UniqueName: \"kubernetes.io/projected/006b1304-8f0c-4891-a71d-2447e6f77298-kube-api-access-jm8dg\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:37:53.416206 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:53.416178 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/006b1304-8f0c-4891-a71d-2447e6f77298-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:37:53.711286 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:53.711231 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf" event={"ID":"006b1304-8f0c-4891-a71d-2447e6f77298","Type":"ContainerDied","Data":"f7347820d95284b10875b3db0968eeb7c53b5baaaf8cbfdca453e15eb9ad1cf2"} Oct 30 16:37:53.711591 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:53.711556 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7347820d95284b10875b3db0968eeb7c53b5baaaf8cbfdca453e15eb9ad1cf2" Oct 30 16:37:53.711811 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:53.711795 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf" Oct 30 16:37:54.120799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:54.120735 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:37:54.121012 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:54.120828 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" podUID="ec24f992-d313-4d6d-987d-98c4c8361df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 16:37:54.226167 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:54.226124 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:55.121266 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:55.121231 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:37:55.121429 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:55.121284 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" podUID="ec24f992-d313-4d6d-987d-98c4c8361df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 16:37:55.868624 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:55.868586 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:56.121753 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:56.121659 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:37:56.121753 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:56.121724 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" podUID="ec24f992-d313-4d6d-987d-98c4c8361df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 16:37:57.124807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:57.124760 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:57.125230 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:57.125121 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:57.125811 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:57.125761 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:37:57.680273 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:57.680233 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:58.132605 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.132571 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6"] Oct 30 16:37:58.133421 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.133399 6202 cpu_manager.go:414] "containerMap: removing container" podUID="006b1304-8f0c-4891-a71d-2447e6f77298" containerName="revoke-keys" Oct 30 16:37:58.133510 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.133424 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006b1304-8f0c-4891-a71d-2447e6f77298" containerName="revoke-keys" Oct 30 16:37:58.133576 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.133564 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="006b1304-8f0c-4891-a71d-2447e6f77298" containerName="revoke-keys" Oct 30 16:37:58.137331 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.137314 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6" Oct 30 16:37:58.158295 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.158260 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6"] Oct 30 16:37:58.165235 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.165214 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:37:58.165393 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.165260 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:37:58.165393 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.165276 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:37:58.165503 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.165281 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:37:58.209343 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.209314 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxdqh\" (UniqueName: \"kubernetes.io/projected/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-kube-api-access-pxdqh\") pod \"maas-api-revoke-sub-3ce183d3-sjqm6\" (UID: \"1943464d-3e38-4d10-ad22-6e4a3a4a47eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6" Oct 30 16:37:58.209446 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.209372 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ce183d3-sjqm6\" (UID: \"1943464d-3e38-4d10-ad22-6e4a3a4a47eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6" Oct 30 16:37:58.209508 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.209489 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ce183d3-sjqm6\" (UID: \"1943464d-3e38-4d10-ad22-6e4a3a4a47eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6" Oct 30 16:37:58.310547 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.310511 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ce183d3-sjqm6\" (UID: \"1943464d-3e38-4d10-ad22-6e4a3a4a47eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6" Oct 30 16:37:58.310722 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.310626 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxdqh\" (UniqueName: \"kubernetes.io/projected/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-kube-api-access-pxdqh\") pod \"maas-api-revoke-sub-3ce183d3-sjqm6\" (UID: \"1943464d-3e38-4d10-ad22-6e4a3a4a47eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6" Oct 30 16:37:58.310722 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.310690 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ce183d3-sjqm6\" (UID: \"1943464d-3e38-4d10-ad22-6e4a3a4a47eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6" Oct 30 16:37:58.311318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.311292 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ce183d3-sjqm6\" (UID: \"1943464d-3e38-4d10-ad22-6e4a3a4a47eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6" Oct 30 16:37:58.313080 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.313055 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ce183d3-sjqm6\" (UID: \"1943464d-3e38-4d10-ad22-6e4a3a4a47eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6" Oct 30 16:37:58.334509 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.334488 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxdqh\" (UniqueName: \"kubernetes.io/projected/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-kube-api-access-pxdqh\") pod \"maas-api-revoke-sub-3ce183d3-sjqm6\" (UID: \"1943464d-3e38-4d10-ad22-6e4a3a4a47eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6" Oct 30 16:37:58.453748 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.453678 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6" Oct 30 16:37:58.752524 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:37:58.752489 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1943464d_3e38_4d10_ad22_6e4a3a4a47eb.slice/crio-71eaf95546a9f63d32a6b60e7d6aa2ebef477d1f8c5a4ab20c656218bb81fb3c WatchSource:0}: Error finding container 71eaf95546a9f63d32a6b60e7d6aa2ebef477d1f8c5a4ab20c656218bb81fb3c: Status 404 returned error can't find the container with id 71eaf95546a9f63d32a6b60e7d6aa2ebef477d1f8c5a4ab20c656218bb81fb3c Oct 30 16:37:58.753017 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:58.752976 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6"] Oct 30 16:37:59.293223 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:59.293191 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:59.768847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:59.768753 6202 generic.go:356] "Generic (PLEG): container finished" podID="1943464d-3e38-4d10-ad22-6e4a3a4a47eb" containerID="75b922eab2b94e3b57256d964bbdec6216f617c45a4cb9b3e97e7b06c3178472" exitCode=0 Oct 30 16:37:59.769010 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:59.768848 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6" event={"ID":"1943464d-3e38-4d10-ad22-6e4a3a4a47eb","Type":"ContainerDied","Data":"75b922eab2b94e3b57256d964bbdec6216f617c45a4cb9b3e97e7b06c3178472"} Oct 30 16:37:59.769010 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:37:59.768923 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6" event={"ID":"1943464d-3e38-4d10-ad22-6e4a3a4a47eb","Type":"ContainerStarted","Data":"71eaf95546a9f63d32a6b60e7d6aa2ebef477d1f8c5a4ab20c656218bb81fb3c"} Oct 30 16:38:00.928825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:00.928757 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:00.932968 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:00.932948 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6" Oct 30 16:38:01.044433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:01.044330 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-maas-api-xks-ca\") pod \"1943464d-3e38-4d10-ad22-6e4a3a4a47eb\" (UID: \"1943464d-3e38-4d10-ad22-6e4a3a4a47eb\") " Oct 30 16:38:01.044704 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:01.044676 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-kube-api-access-pxdqh\" (UniqueName: \"kubernetes.io/projected/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-kube-api-access-pxdqh\") pod \"1943464d-3e38-4d10-ad22-6e4a3a4a47eb\" (UID: \"1943464d-3e38-4d10-ad22-6e4a3a4a47eb\") " Oct 30 16:38:01.044906 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:01.044789 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-maas-api-openshift-ca\") pod \"1943464d-3e38-4d10-ad22-6e4a3a4a47eb\" (UID: \"1943464d-3e38-4d10-ad22-6e4a3a4a47eb\") " Oct 30 16:38:01.045500 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:01.045129 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-maas-api-openshift-ca" pod "1943464d-3e38-4d10-ad22-6e4a3a4a47eb" (UID: "1943464d-3e38-4d10-ad22-6e4a3a4a47eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:01.045500 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:01.045353 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:01.048377 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:01.048345 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-maas-api-xks-ca" pod "1943464d-3e38-4d10-ad22-6e4a3a4a47eb" (UID: "1943464d-3e38-4d10-ad22-6e4a3a4a47eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:01.049963 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:01.049935 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-kube-api-access-pxdqh" pod "1943464d-3e38-4d10-ad22-6e4a3a4a47eb" (UID: "1943464d-3e38-4d10-ad22-6e4a3a4a47eb"). InnerVolumeSpecName "kube-api-access-pxdqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:01.147341 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:01.147296 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxdqh\" (UniqueName: \"kubernetes.io/projected/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-kube-api-access-pxdqh\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:01.147341 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:01.147336 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1943464d-3e38-4d10-ad22-6e4a3a4a47eb-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:01.783294 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:01.783263 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6" Oct 30 16:38:01.783294 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:01.783275 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6" event={"ID":"1943464d-3e38-4d10-ad22-6e4a3a4a47eb","Type":"ContainerDied","Data":"71eaf95546a9f63d32a6b60e7d6aa2ebef477d1f8c5a4ab20c656218bb81fb3c"} Oct 30 16:38:01.783536 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:01.783319 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71eaf95546a9f63d32a6b60e7d6aa2ebef477d1f8c5a4ab20c656218bb81fb3c" Oct 30 16:38:02.665469 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:02.665428 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:03.299327 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:03.299269 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:04.241198 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.241160 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm"] Oct 30 16:38:04.242161 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.242138 6202 cpu_manager.go:414] "containerMap: removing container" podUID="1943464d-3e38-4d10-ad22-6e4a3a4a47eb" containerName="revoke-keys" Oct 30 16:38:04.242161 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.242162 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1943464d-3e38-4d10-ad22-6e4a3a4a47eb" containerName="revoke-keys" Oct 30 16:38:04.242329 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.242307 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="1943464d-3e38-4d10-ad22-6e4a3a4a47eb" containerName="revoke-keys" Oct 30 16:38:04.246834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.246804 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.309178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.309140 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:04.312310 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.312277 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk"] Oct 30 16:38:04.318052 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.318029 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm"] Oct 30 16:38:04.318340 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.318322 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.319602 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.319584 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv"] Oct 30 16:38:04.326272 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.326253 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.330934 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.330246 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk"] Oct 30 16:38:04.373330 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.373302 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv"] Oct 30 16:38:04.376708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.376671 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q"] Oct 30 16:38:04.382143 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.382116 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.387813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.387099 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10786690-fb43-4289-826d-1bbe86793662-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.387813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.387211 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10786690-fb43-4289-826d-1bbe86793662-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.387813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.387240 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.387813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.387278 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.387813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.387301 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.387813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.387337 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.387813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.387383 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z2bq\" (UniqueName: \"kubernetes.io/projected/10786690-fb43-4289-826d-1bbe86793662-kube-api-access-4z2bq\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.387813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.387432 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.387813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.387458 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.387813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.387491 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.387813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.387609 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpldp\" (UniqueName: \"kubernetes.io/projected/860b7b50-ea1d-4832-80e1-f842e1e9f74c-kube-api-access-tpldp\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.387813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.387639 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.387813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.387679 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.387813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.387735 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10786690-fb43-4289-826d-1bbe86793662-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.387813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.387800 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.388745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.387827 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.388745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.388252 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.388745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.388372 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.399635 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.399612 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q"] Oct 30 16:38:04.489473 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.489442 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnkf7\" (UniqueName: \"kubernetes.io/projected/71652f10-3fa9-4e81-8394-f631af440cae-kube-api-access-qnkf7\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.489473 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.489478 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4gr2\" (UniqueName: \"kubernetes.io/projected/ded079d5-65af-48a8-82d2-922eb053d8c2-kube-api-access-h4gr2\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.489725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.489501 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.489725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.489596 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z2bq\" (UniqueName: \"kubernetes.io/projected/10786690-fb43-4289-826d-1bbe86793662-kube-api-access-4z2bq\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.489725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.489635 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.489725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.489674 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.490009 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.489849 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.490009 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.489887 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.490009 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.489926 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.490009 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.489959 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.490208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.490114 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpldp\" (UniqueName: \"kubernetes.io/projected/860b7b50-ea1d-4832-80e1-f842e1e9f74c-kube-api-access-tpldp\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.490208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.490143 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded079d5-65af-48a8-82d2-922eb053d8c2-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.490208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.490174 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.490208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.490199 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.490414 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.490239 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.490414 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.490238 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.490414 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.490258 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.490414 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.490244 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.490598 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.490457 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10786690-fb43-4289-826d-1bbe86793662-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.490598 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.490479 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.490598 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.490496 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.490751 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.490645 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.490751 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.490653 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71652f10-3fa9-4e81-8394-f631af440cae-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.490877 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.490781 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.490937 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.490915 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.490984 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.490965 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.491093 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.491067 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.491167 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.491094 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.491316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.491263 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.491386 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.491367 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.491433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.491406 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.491507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.491491 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.491556 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.491530 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.491592 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.491565 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10786690-fb43-4289-826d-1bbe86793662-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.491592 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.491581 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71652f10-3fa9-4e81-8394-f631af440cae-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.491691 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.491598 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71652f10-3fa9-4e81-8394-f631af440cae-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.491691 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.491668 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10786690-fb43-4289-826d-1bbe86793662-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.491836 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.491710 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.491836 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.491758 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.491836 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.491802 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.491836 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.491830 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.492046 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.491855 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.492118 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.492093 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.492287 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.492267 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.492725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.492387 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.492725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.492528 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10786690-fb43-4289-826d-1bbe86793662-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.492900 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.492814 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.493333 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.493315 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.493463 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.493443 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.494046 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.494022 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10786690-fb43-4289-826d-1bbe86793662-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.521466 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.521444 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z2bq\" (UniqueName: \"kubernetes.io/projected/10786690-fb43-4289-826d-1bbe86793662-kube-api-access-4z2bq\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.522640 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.522617 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10786690-fb43-4289-826d-1bbe86793662-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.524243 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.524211 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.525579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.525557 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpldp\" (UniqueName: \"kubernetes.io/projected/860b7b50-ea1d-4832-80e1-f842e1e9f74c-kube-api-access-tpldp\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.593337 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.593309 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.593496 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.593417 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded079d5-65af-48a8-82d2-922eb053d8c2-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.593496 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.593464 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.593608 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.593500 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71652f10-3fa9-4e81-8394-f631af440cae-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.593608 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.593545 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.593608 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.593569 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.593786 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.593702 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.593786 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.593702 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.593898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.593759 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.593898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.593787 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.593898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.593840 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.593898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.593869 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.594117 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.593907 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.594117 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.593949 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71652f10-3fa9-4e81-8394-f631af440cae-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.594117 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.593970 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71652f10-3fa9-4e81-8394-f631af440cae-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.594117 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.593972 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.594117 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.593992 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.594117 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.594032 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.594117 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.594080 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnkf7\" (UniqueName: \"kubernetes.io/projected/71652f10-3fa9-4e81-8394-f631af440cae-kube-api-access-qnkf7\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.594117 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.594110 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4gr2\" (UniqueName: \"kubernetes.io/projected/ded079d5-65af-48a8-82d2-922eb053d8c2-kube-api-access-h4gr2\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.594547 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.594167 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.594547 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.594220 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded079d5-65af-48a8-82d2-922eb053d8c2-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.594547 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.594268 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.594547 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.594362 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.594547 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.594442 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.594547 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.594506 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.594833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.594799 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71652f10-3fa9-4e81-8394-f631af440cae-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.594926 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.594841 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.596301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.596282 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.596781 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.596749 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.596851 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.596836 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.597095 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.597065 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71652f10-3fa9-4e81-8394-f631af440cae-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.617695 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.617678 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:04.631386 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.631312 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71652f10-3fa9-4e81-8394-f631af440cae-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.631386 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.631317 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4gr2\" (UniqueName: \"kubernetes.io/projected/ded079d5-65af-48a8-82d2-922eb053d8c2-kube-api-access-h4gr2\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.634081 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.634063 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.641131 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.641107 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnkf7\" (UniqueName: \"kubernetes.io/projected/71652f10-3fa9-4e81-8394-f631af440cae-kube-api-access-qnkf7\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.663219 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.663021 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:04.665219 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.665183 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:04.700389 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.700354 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:04.942291 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:38:04.942253 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860b7b50_ea1d_4832_80e1_f842e1e9f74c.slice/crio-ed655a4bf18c3e7d8edfdac26cd79b02804ccad23fd4d9f0404270412d8f4f8e WatchSource:0}: Error finding container ed655a4bf18c3e7d8edfdac26cd79b02804ccad23fd4d9f0404270412d8f4f8e: Status 404 returned error can't find the container with id ed655a4bf18c3e7d8edfdac26cd79b02804ccad23fd4d9f0404270412d8f4f8e Oct 30 16:38:04.943166 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.943142 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm"] Oct 30 16:38:04.945238 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.945201 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:04.945317 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.945278 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:04.945352 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:04.945318 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:05.001674 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:38:05.001620 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10786690_fb43_4289_826d_1bbe86793662.slice/crio-21c2df5fbd22b9203a72ca2c04099ecd0651e1e56f12c359c957ece4735aca9a WatchSource:0}: Error finding container 21c2df5fbd22b9203a72ca2c04099ecd0651e1e56f12c359c957ece4735aca9a: Status 404 returned error can't find the container with id 21c2df5fbd22b9203a72ca2c04099ecd0651e1e56f12c359c957ece4735aca9a Oct 30 16:38:05.003695 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.003669 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk"] Oct 30 16:38:05.004319 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.004281 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:05.004403 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.004365 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:05.004451 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.004412 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:05.019386 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:38:05.019362 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71652f10_3fa9_4e81_8394_f631af440cae.slice/crio-759a2fcd4ea42dd6a717d9543ba60681bf71d723a2dab7c1af3e18d1d6de0a31 WatchSource:0}: Error finding container 759a2fcd4ea42dd6a717d9543ba60681bf71d723a2dab7c1af3e18d1d6de0a31: Status 404 returned error can't find the container with id 759a2fcd4ea42dd6a717d9543ba60681bf71d723a2dab7c1af3e18d1d6de0a31 Oct 30 16:38:05.020326 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.020305 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv"] Oct 30 16:38:05.021726 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.021673 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:05.021981 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.021747 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:05.021981 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.021819 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:05.049463 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.049436 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q"] Oct 30 16:38:05.055252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.055212 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:05.055356 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.055302 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:05.055356 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.055349 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:05.816898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.816797 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" event={"ID":"10786690-fb43-4289-826d-1bbe86793662","Type":"ContainerStarted","Data":"f3955af3bb8f925971b694a0658de920514c8aacfca451863deb645d971950d6"} Oct 30 16:38:05.816898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.816895 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" event={"ID":"10786690-fb43-4289-826d-1bbe86793662","Type":"ContainerStarted","Data":"21c2df5fbd22b9203a72ca2c04099ecd0651e1e56f12c359c957ece4735aca9a"} Oct 30 16:38:05.818904 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.818877 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" event={"ID":"71652f10-3fa9-4e81-8394-f631af440cae","Type":"ContainerStarted","Data":"09db65d1e0f1ae274e6bc0de74b7edaa1eb747bd326164da56372ee7ac4b7f72"} Oct 30 16:38:05.819026 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.818921 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" event={"ID":"71652f10-3fa9-4e81-8394-f631af440cae","Type":"ContainerStarted","Data":"759a2fcd4ea42dd6a717d9543ba60681bf71d723a2dab7c1af3e18d1d6de0a31"} Oct 30 16:38:05.820821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.820794 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" event={"ID":"860b7b50-ea1d-4832-80e1-f842e1e9f74c","Type":"ContainerStarted","Data":"57243f94287d5b04ab6205bc73b886b182f6e71cf968b8720f0ad7a12f87eb85"} Oct 30 16:38:05.820929 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.820837 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" event={"ID":"860b7b50-ea1d-4832-80e1-f842e1e9f74c","Type":"ContainerStarted","Data":"ed655a4bf18c3e7d8edfdac26cd79b02804ccad23fd4d9f0404270412d8f4f8e"} Oct 30 16:38:05.822556 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.822534 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" event={"ID":"ded079d5-65af-48a8-82d2-922eb053d8c2","Type":"ContainerStarted","Data":"18f855d140aebe67422e376d53027b5c80553d1f19eeb00b4e4d2372ff7e78bf"} Oct 30 16:38:05.822627 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.822572 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" event={"ID":"ded079d5-65af-48a8-82d2-922eb053d8c2","Type":"ContainerStarted","Data":"04484285fc7dac305b53fed9c6e004774916e8a258a25b938e5a4496d95287ba"} Oct 30 16:38:06.005792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:05.999439 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" podStartSLOduration=1.999419401 podStartE2EDuration="1.999419401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:05.912104085 +0000 UTC m=+999.622558828" watchObservedRunningTime="2026-10-30 16:38:05.999419401 +0000 UTC m=+999.709874147" Oct 30 16:38:06.060173 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:06.060107 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" podStartSLOduration=2.060090127 podStartE2EDuration="2.060090127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:06.058309809 +0000 UTC m=+999.768764553" watchObservedRunningTime="2026-10-30 16:38:06.060090127 +0000 UTC m=+999.770544872" Oct 30 16:38:06.060794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:06.060728 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" podStartSLOduration=2.060719403 podStartE2EDuration="2.060719403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:06.001064311 +0000 UTC m=+999.711519057" watchObservedRunningTime="2026-10-30 16:38:06.060719403 +0000 UTC m=+999.771174127" Oct 30 16:38:06.206809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:06.204690 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:06.620792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:06.620071 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:38:06.620792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:06.620154 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" podUID="860b7b50-ea1d-4832-80e1-f842e1e9f74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 16:38:06.620792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:06.620431 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:06.665793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:06.665661 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:06.669789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:06.667228 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:06.669789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:06.667517 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:38:06.669789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:06.667627 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:38:06.669789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:06.667672 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" podUID="10786690-fb43-4289-826d-1bbe86793662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 16:38:06.669789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:06.667746 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" podUID="71652f10-3fa9-4e81-8394-f631af440cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 16:38:06.706793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:06.702495 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:06.725832 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:06.725276 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:38:06.725832 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:06.725334 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" podUID="ded079d5-65af-48a8-82d2-922eb053d8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 16:38:07.456888 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:07.456844 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 16:38:07.578980 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:07.577860 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:07.620707 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:07.620235 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:38:07.620707 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:07.620302 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" podUID="860b7b50-ea1d-4832-80e1-f842e1e9f74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 16:38:07.645617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:07.644840 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:07.666058 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:07.666025 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:38:07.666574 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:07.666549 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" podUID="10786690-fb43-4289-826d-1bbe86793662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 16:38:07.666708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:07.666496 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:38:07.666828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:07.666810 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" podUID="71652f10-3fa9-4e81-8394-f631af440cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 16:38:07.698882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:07.698841 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:07.709161 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:07.709072 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:38:07.709353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:07.709331 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" podUID="ded079d5-65af-48a8-82d2-922eb053d8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 16:38:07.869921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:07.869876 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:07.877824 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:07.870268 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:08.009852 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.000904 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:08.093795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.089669 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:08.326788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.324861 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:38:08.457792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.457724 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 16:38:08.473495 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.473463 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:38:08.521105 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.521029 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" podStartSLOduration=4.5210092060000004 podStartE2EDuration="4.521009206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:06.125559857 +0000 UTC m=+999.836014600" watchObservedRunningTime="2026-10-30 16:38:08.521009206 +0000 UTC m=+1002.231463952" Oct 30 16:38:08.712843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.703861 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:08.712843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.712096 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 16:38:08.712843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.712483 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:38:08.845796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.715401 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 16:38:08.845796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.715734 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:38:08.999646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.901913 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:38:08.999646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.901988 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" podUID="71652f10-3fa9-4e81-8394-f631af440cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 16:38:08.999646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.902075 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:38:08.999646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.902094 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" podUID="10786690-fb43-4289-826d-1bbe86793662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 16:38:08.999646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.902152 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:38:08.999646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.902170 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" podUID="860b7b50-ea1d-4832-80e1-f842e1e9f74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 16:38:08.999646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.910983 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded" Oct 30 16:38:08.999646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.911222 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:38:08.999646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:08.950891 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 16:38:09.013795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.003875 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:38:09.013795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.008044 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf"] Oct 30 16:38:09.235830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.196954 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:38:09.235830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.197032 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" podUID="ded079d5-65af-48a8-82d2-922eb053d8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 16:38:09.371791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.345195 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf" Oct 30 16:38:09.457791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.452135 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:38:09.457791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.452320 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:38:09.480793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.474375 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:38:09.480793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.474443 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 16:38:09.503871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.483857 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:09.503871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.502663 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf"] Oct 30 16:38:09.503871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.502787 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:38:09.588819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.569755 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:38:09.620093 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.588570 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4def6444-97fsf\" (UID: \"a0c4264c-1b11-40d8-a62b-ca9928aef5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf" Oct 30 16:38:09.620622 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.620593 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2h4l\" (UniqueName: \"kubernetes.io/projected/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-kube-api-access-x2h4l\") pod \"maas-api-revoke-sub-4def6444-97fsf\" (UID: \"a0c4264c-1b11-40d8-a62b-ca9928aef5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf" Oct 30 16:38:09.620858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.620836 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4def6444-97fsf\" (UID: \"a0c4264c-1b11-40d8-a62b-ca9928aef5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf" Oct 30 16:38:09.755841 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.730880 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2h4l\" (UniqueName: \"kubernetes.io/projected/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-kube-api-access-x2h4l\") pod \"maas-api-revoke-sub-4def6444-97fsf\" (UID: \"a0c4264c-1b11-40d8-a62b-ca9928aef5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf" Oct 30 16:38:09.841151 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.841109 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4def6444-97fsf\" (UID: \"a0c4264c-1b11-40d8-a62b-ca9928aef5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf" Oct 30 16:38:09.933790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.734208 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 16:38:09.933790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.734241 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:09.933790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.869172 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:38:09.933790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.738526 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:38:09.933790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.870058 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" podUID="10786690-fb43-4289-826d-1bbe86793662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 16:38:09.933790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.734172 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:38:09.933790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.813163 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:09.933790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.831637 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2h4l\" (UniqueName: \"kubernetes.io/projected/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-kube-api-access-x2h4l\") pod \"maas-api-revoke-sub-4def6444-97fsf\" (UID: \"a0c4264c-1b11-40d8-a62b-ca9928aef5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf" Oct 30 16:38:09.933790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.738578 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:38:09.933790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.889391 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" podUID="71652f10-3fa9-4e81-8394-f631af440cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 16:38:09.933790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.738619 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:38:09.933790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.889451 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" podUID="860b7b50-ea1d-4832-80e1-f842e1e9f74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 16:38:09.933790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.756807 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:38:09.933790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.889493 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" podUID="ded079d5-65af-48a8-82d2-922eb053d8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 16:38:09.933790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.919394 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:09.953932 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.953888 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:09.965029 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.964989 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4def6444-97fsf\" (UID: \"a0c4264c-1b11-40d8-a62b-ca9928aef5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf" Oct 30 16:38:09.990793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:09.971789 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4def6444-97fsf\" (UID: \"a0c4264c-1b11-40d8-a62b-ca9928aef5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf" Oct 30 16:38:10.132359 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.132321 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4def6444-97fsf\" (UID: \"a0c4264c-1b11-40d8-a62b-ca9928aef5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf" Oct 30 16:38:10.142020 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.141987 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf" Oct 30 16:38:10.514852 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.484176 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:10.514852 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.488837 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:10.514852 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.488866 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:10.582795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.581893 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:10.699318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.699281 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:38:10.706797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.699567 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" podUID="10786690-fb43-4289-826d-1bbe86793662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 16:38:10.707146 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.707119 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:38:10.707311 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.707284 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" podUID="71652f10-3fa9-4e81-8394-f631af440cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 16:38:10.740791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.721842 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2"] Oct 30 16:38:10.762921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.728881 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:38:10.762921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.762809 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" podUID="860b7b50-ea1d-4832-80e1-f842e1e9f74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 16:38:10.762921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.729217 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:38:10.762921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.762881 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" podUID="ded079d5-65af-48a8-82d2-922eb053d8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 16:38:10.809145 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.809040 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2"] Oct 30 16:38:10.814919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.809434 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2" Oct 30 16:38:10.927788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.881907 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:38:10.927788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.882416 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:10.927788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.882672 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 16:38:10.931791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.929745 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 16:38:10.931791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.930138 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:38:10.942665 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.942628 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:10.969399 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.969360 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 16:38:10.974827 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.974797 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f683a67-3e8a-4d72-99a8-6a3bae293802-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-806c8e77-tfxt2\" (UID: \"8f683a67-3e8a-4d72-99a8-6a3bae293802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2" Oct 30 16:38:10.975062 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.975046 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jbvd\" (UniqueName: \"kubernetes.io/projected/8f683a67-3e8a-4d72-99a8-6a3bae293802-kube-api-access-2jbvd\") pod \"maas-api-revoke-sub-806c8e77-tfxt2\" (UID: \"8f683a67-3e8a-4d72-99a8-6a3bae293802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2" Oct 30 16:38:10.975425 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:10.975404 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f683a67-3e8a-4d72-99a8-6a3bae293802-maas-api-xks-ca\") pod \"maas-api-revoke-sub-806c8e77-tfxt2\" (UID: \"8f683a67-3e8a-4d72-99a8-6a3bae293802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2" Oct 30 16:38:11.085799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.080140 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f683a67-3e8a-4d72-99a8-6a3bae293802-maas-api-xks-ca\") pod \"maas-api-revoke-sub-806c8e77-tfxt2\" (UID: \"8f683a67-3e8a-4d72-99a8-6a3bae293802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2" Oct 30 16:38:11.118706 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.118674 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f683a67-3e8a-4d72-99a8-6a3bae293802-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-806c8e77-tfxt2\" (UID: \"8f683a67-3e8a-4d72-99a8-6a3bae293802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2" Oct 30 16:38:11.186793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.119724 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f683a67-3e8a-4d72-99a8-6a3bae293802-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-806c8e77-tfxt2\" (UID: \"8f683a67-3e8a-4d72-99a8-6a3bae293802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2" Oct 30 16:38:11.186793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.185017 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jbvd\" (UniqueName: \"kubernetes.io/projected/8f683a67-3e8a-4d72-99a8-6a3bae293802-kube-api-access-2jbvd\") pod \"maas-api-revoke-sub-806c8e77-tfxt2\" (UID: \"8f683a67-3e8a-4d72-99a8-6a3bae293802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2" Oct 30 16:38:11.202796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.193854 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:11.328900 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.324357 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jbvd\" (UniqueName: \"kubernetes.io/projected/8f683a67-3e8a-4d72-99a8-6a3bae293802-kube-api-access-2jbvd\") pod \"maas-api-revoke-sub-806c8e77-tfxt2\" (UID: \"8f683a67-3e8a-4d72-99a8-6a3bae293802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2" Oct 30 16:38:11.330214 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.330185 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f683a67-3e8a-4d72-99a8-6a3bae293802-maas-api-xks-ca\") pod \"maas-api-revoke-sub-806c8e77-tfxt2\" (UID: \"8f683a67-3e8a-4d72-99a8-6a3bae293802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2" Oct 30 16:38:11.525848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.524883 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:11.576619 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.560413 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:38:11.731820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.706371 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2" Oct 30 16:38:11.731820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.706871 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:38:11.731820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.706925 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" podUID="860b7b50-ea1d-4832-80e1-f842e1e9f74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 16:38:11.731820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.708740 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:38:11.731820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.708806 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" podUID="10786690-fb43-4289-826d-1bbe86793662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 16:38:11.731820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.708875 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:38:11.731820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.708899 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" podUID="71652f10-3fa9-4e81-8394-f631af440cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 16:38:11.731820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.709130 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:38:11.731820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.709154 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" podUID="ded079d5-65af-48a8-82d2-922eb053d8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 16:38:11.731820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.713376 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf"] Oct 30 16:38:11.776500 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:38:11.776420 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c4264c_1b11_40d8_a62b_ca9928aef5d4.slice/crio-7a0539517663a454ebe69b1aa84c20aeef905b1d2fe453e9eb5545e618d67618 WatchSource:0}: Error finding container 7a0539517663a454ebe69b1aa84c20aeef905b1d2fe453e9eb5545e618d67618: Status 404 returned error can't find the container with id 7a0539517663a454ebe69b1aa84c20aeef905b1d2fe453e9eb5545e618d67618 Oct 30 16:38:11.891699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.882210 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:11.891699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.882873 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:11.891699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.883118 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 16:38:11.966794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.955671 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded" Oct 30 16:38:11.966794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.955791 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded" Oct 30 16:38:11.966794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.956549 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:11.984875 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:11.974567 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:12.212174 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:12.209955 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:38:12.318797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:12.318623 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:12.440849 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:12.435176 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:12.612792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:12.596326 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 16:38:12.612792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:12.596625 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:38:12.623797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:12.622102 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:38:12.623797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:12.622154 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" podUID="860b7b50-ea1d-4832-80e1-f842e1e9f74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 16:38:12.705680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:12.705452 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:38:12.716919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:12.706026 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" podUID="10786690-fb43-4289-826d-1bbe86793662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 16:38:12.749280 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:12.748804 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:38:12.749553 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:12.749511 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" podUID="71652f10-3fa9-4e81-8394-f631af440cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 16:38:12.883789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:12.882970 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:12.883789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:12.883418 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:12.926958 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:12.926917 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:38:12.927262 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:12.927232 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" podUID="ded079d5-65af-48a8-82d2-922eb053d8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 16:38:12.943619 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:12.943573 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 16:38:13.002686 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:12.974134 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 16:38:13.085109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:13.085048 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 16:38:13.468435 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:13.418386 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:38:13.651168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:13.651129 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:38:13.657802 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:13.651733 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" podUID="860b7b50-ea1d-4832-80e1-f842e1e9f74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 16:38:13.692225 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:13.692181 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:13.712154 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:13.693851 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:38:13.712394 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:13.712363 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" podUID="10786690-fb43-4289-826d-1bbe86793662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 16:38:13.712528 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:13.697024 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:38:13.712638 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:13.712621 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" podUID="71652f10-3fa9-4e81-8394-f631af440cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 16:38:13.712730 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:13.701098 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:38:13.737795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:13.712830 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" podUID="ded079d5-65af-48a8-82d2-922eb053d8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 16:38:13.738062 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:13.729651 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf" event={"ID":"a0c4264c-1b11-40d8-a62b-ca9928aef5d4","Type":"ContainerStarted","Data":"7a0539517663a454ebe69b1aa84c20aeef905b1d2fe453e9eb5545e618d67618"} Oct 30 16:38:13.894109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:13.892906 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:13.894109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:13.893803 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:38:13.990787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:13.989894 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:14.038991 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.038925 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:14.039619 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.039574 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:14.192854 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.173849 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:14.295085 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.275164 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:38:14.317792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.311916 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2"] Oct 30 16:38:14.517235 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.517146 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:14.655801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.637823 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:38:14.655801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.637970 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" podUID="860b7b50-ea1d-4832-80e1-f842e1e9f74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 16:38:14.665720 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.665236 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:38:14.665720 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.665289 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" podUID="10786690-fb43-4289-826d-1bbe86793662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 16:38:14.666577 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.666020 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:38:14.666577 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.666058 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" podUID="71652f10-3fa9-4e81-8394-f631af440cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 16:38:14.720812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.712847 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:14.741788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.724479 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:38:14.741788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.724542 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" podUID="ded079d5-65af-48a8-82d2-922eb053d8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 16:38:14.765465 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.762056 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:38:14.768484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.767123 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:14.768484 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:38:14.767212 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:38:14.771881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.771005 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:38:14.780846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.780661 6202 generic.go:356] "Generic (PLEG): container finished" podID="a0c4264c-1b11-40d8-a62b-ca9928aef5d4" containerID="2f1c52f03b06e8ebb5cb716abaee394e77d0d01afcc0cc567effd0b9703fc9c1" exitCode=0 Oct 30 16:38:14.780846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.780745 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf" event={"ID":"a0c4264c-1b11-40d8-a62b-ca9928aef5d4","Type":"ContainerDied","Data":"2f1c52f03b06e8ebb5cb716abaee394e77d0d01afcc0cc567effd0b9703fc9c1"} Oct 30 16:38:14.790512 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.790490 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:38:14.809982 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.803025 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:38:14.825700 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.817268 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:38:14.840988 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:14.834906 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2" event={"ID":"8f683a67-3e8a-4d72-99a8-6a3bae293802","Type":"ContainerStarted","Data":"1a668a4f2e8eb15a35986a6ead4bb4354d058a5180a3365322f6b1728b11396b"} Oct 30 16:38:15.618790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:15.618430 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:38:15.618790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:15.618508 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" podUID="860b7b50-ea1d-4832-80e1-f842e1e9f74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 16:38:15.668789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:15.668169 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:38:15.668789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:15.668246 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" podUID="71652f10-3fa9-4e81-8394-f631af440cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 16:38:15.895297 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:15.893839 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2" event={"ID":"8f683a67-3e8a-4d72-99a8-6a3bae293802","Type":"ContainerStarted","Data":"5e9c2713b13fbb17eefe44f8fb1e4571ad3ff4e9fd9f61a3a0d1437c9f134901"} Oct 30 16:38:16.255854 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.247211 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:16.581568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.581532 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:16.585028 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.585006 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:16.640957 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.640931 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:38:16.650273 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.650216 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2" podStartSLOduration=6.650198088 podStartE2EDuration="6.650198088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:15.963118394 +0000 UTC m=+1009.673573141" watchObservedRunningTime="2026-10-30 16:38:16.650198088 +0000 UTC m=+1010.360652832" Oct 30 16:38:16.701653 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.701616 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:16.702130 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.701684 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" podUID="ded079d5-65af-48a8-82d2-922eb053d8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:16.734226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.734179 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:16.735628 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.735606 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:16.751473 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.751442 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:38:16.765754 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.765724 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf" Oct 30 16:38:16.854252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.854168 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-maas-api-xks-ca\") pod \"a0c4264c-1b11-40d8-a62b-ca9928aef5d4\" (UID: \"a0c4264c-1b11-40d8-a62b-ca9928aef5d4\") " Oct 30 16:38:16.854252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.854234 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-kube-api-access-x2h4l\" (UniqueName: \"kubernetes.io/projected/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-kube-api-access-x2h4l\") pod \"a0c4264c-1b11-40d8-a62b-ca9928aef5d4\" (UID: \"a0c4264c-1b11-40d8-a62b-ca9928aef5d4\") " Oct 30 16:38:16.854522 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.854288 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-maas-api-openshift-ca\") pod \"a0c4264c-1b11-40d8-a62b-ca9928aef5d4\" (UID: \"a0c4264c-1b11-40d8-a62b-ca9928aef5d4\") " Oct 30 16:38:16.854715 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.854681 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-maas-api-openshift-ca" pod "a0c4264c-1b11-40d8-a62b-ca9928aef5d4" (UID: "a0c4264c-1b11-40d8-a62b-ca9928aef5d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:16.855279 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.855258 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:16.873373 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.868682 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-maas-api-xks-ca" pod "a0c4264c-1b11-40d8-a62b-ca9928aef5d4" (UID: "a0c4264c-1b11-40d8-a62b-ca9928aef5d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:16.882423 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.882008 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-kube-api-access-x2h4l" pod "a0c4264c-1b11-40d8-a62b-ca9928aef5d4" (UID: "a0c4264c-1b11-40d8-a62b-ca9928aef5d4"). InnerVolumeSpecName "kube-api-access-x2h4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:16.946389 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.946346 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf" event={"ID":"a0c4264c-1b11-40d8-a62b-ca9928aef5d4","Type":"ContainerDied","Data":"7a0539517663a454ebe69b1aa84c20aeef905b1d2fe453e9eb5545e618d67618"} Oct 30 16:38:16.946571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.946417 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a0539517663a454ebe69b1aa84c20aeef905b1d2fe453e9eb5545e618d67618" Oct 30 16:38:16.946571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.946485 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf" Oct 30 16:38:16.976843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.958746 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:16.977043 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.976857 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2h4l\" (UniqueName: \"kubernetes.io/projected/a0c4264c-1b11-40d8-a62b-ca9928aef5d4-kube-api-access-x2h4l\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:16.983750 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.983712 6202 generic.go:356] "Generic (PLEG): container finished" podID="8f683a67-3e8a-4d72-99a8-6a3bae293802" containerID="5e9c2713b13fbb17eefe44f8fb1e4571ad3ff4e9fd9f61a3a0d1437c9f134901" exitCode=0 Oct 30 16:38:16.984327 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:16.984305 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2" event={"ID":"8f683a67-3e8a-4d72-99a8-6a3bae293802","Type":"ContainerDied","Data":"5e9c2713b13fbb17eefe44f8fb1e4571ad3ff4e9fd9f61a3a0d1437c9f134901"} Oct 30 16:38:17.712795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:17.709937 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:17.712795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:17.710068 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" podUID="71652f10-3fa9-4e81-8394-f631af440cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:17.756793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:17.750207 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:17.756793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:17.750269 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" podUID="ded079d5-65af-48a8-82d2-922eb053d8c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:17.898790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:17.898625 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:18.257822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:18.257466 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2" Oct 30 16:38:18.258862 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:18.258726 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:18.259881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:18.259818 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:18.264786 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:18.263716 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:38:18.413224 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:18.413175 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f683a67-3e8a-4d72-99a8-6a3bae293802-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f683a67-3e8a-4d72-99a8-6a3bae293802-maas-api-xks-ca\") pod \"8f683a67-3e8a-4d72-99a8-6a3bae293802\" (UID: \"8f683a67-3e8a-4d72-99a8-6a3bae293802\") " Oct 30 16:38:18.413431 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:18.413264 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f683a67-3e8a-4d72-99a8-6a3bae293802-kube-api-access-2jbvd\" (UniqueName: \"kubernetes.io/projected/8f683a67-3e8a-4d72-99a8-6a3bae293802-kube-api-access-2jbvd\") pod \"8f683a67-3e8a-4d72-99a8-6a3bae293802\" (UID: \"8f683a67-3e8a-4d72-99a8-6a3bae293802\") " Oct 30 16:38:18.413431 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:18.413422 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f683a67-3e8a-4d72-99a8-6a3bae293802-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f683a67-3e8a-4d72-99a8-6a3bae293802-maas-api-openshift-ca\") pod \"8f683a67-3e8a-4d72-99a8-6a3bae293802\" (UID: \"8f683a67-3e8a-4d72-99a8-6a3bae293802\") " Oct 30 16:38:18.414544 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:18.414082 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f683a67-3e8a-4d72-99a8-6a3bae293802-maas-api-openshift-ca" pod "8f683a67-3e8a-4d72-99a8-6a3bae293802" (UID: "8f683a67-3e8a-4d72-99a8-6a3bae293802"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:18.423108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:18.423077 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f683a67-3e8a-4d72-99a8-6a3bae293802-maas-api-xks-ca" pod "8f683a67-3e8a-4d72-99a8-6a3bae293802" (UID: "8f683a67-3e8a-4d72-99a8-6a3bae293802"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:18.423892 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:18.423872 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f683a67-3e8a-4d72-99a8-6a3bae293802-kube-api-access-2jbvd" pod "8f683a67-3e8a-4d72-99a8-6a3bae293802" (UID: "8f683a67-3e8a-4d72-99a8-6a3bae293802"). InnerVolumeSpecName "kube-api-access-2jbvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:18.493517 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:18.493470 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:18.494934 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:18.494426 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:18.515421 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:18.515394 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f683a67-3e8a-4d72-99a8-6a3bae293802-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:18.515421 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:18.515422 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jbvd\" (UniqueName: \"kubernetes.io/projected/8f683a67-3e8a-4d72-99a8-6a3bae293802-kube-api-access-2jbvd\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:18.515581 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:18.515436 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f683a67-3e8a-4d72-99a8-6a3bae293802-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:18.543626 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:18.543600 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:38:19.019676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:19.019646 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2" Oct 30 16:38:19.020077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:19.019675 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2" event={"ID":"8f683a67-3e8a-4d72-99a8-6a3bae293802","Type":"ContainerDied","Data":"1a668a4f2e8eb15a35986a6ead4bb4354d058a5180a3365322f6b1728b11396b"} Oct 30 16:38:19.020077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:19.019753 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a668a4f2e8eb15a35986a6ead4bb4354d058a5180a3365322f6b1728b11396b" Oct 30 16:38:19.673945 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:19.673908 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:21.147348 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.147310 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq"] Oct 30 16:38:21.148313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.148288 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a0c4264c-1b11-40d8-a62b-ca9928aef5d4" containerName="revoke-keys" Oct 30 16:38:21.148417 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.148314 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c4264c-1b11-40d8-a62b-ca9928aef5d4" containerName="revoke-keys" Oct 30 16:38:21.148417 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.148335 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8f683a67-3e8a-4d72-99a8-6a3bae293802" containerName="revoke-keys" Oct 30 16:38:21.148417 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.148343 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f683a67-3e8a-4d72-99a8-6a3bae293802" containerName="revoke-keys" Oct 30 16:38:21.148569 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.148534 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c4264c-1b11-40d8-a62b-ca9928aef5d4" containerName="revoke-keys" Oct 30 16:38:21.148569 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.148549 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f683a67-3e8a-4d72-99a8-6a3bae293802" containerName="revoke-keys" Oct 30 16:38:21.153229 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.153203 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq" Oct 30 16:38:21.172128 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.172103 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq"] Oct 30 16:38:21.189810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.178997 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:38:21.190308 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.190289 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:38:21.190402 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.190337 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:38:21.190402 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.190355 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:38:21.258904 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.258856 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7eee821-a910-467e-9a41-16ed03cfc82e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23ff8ff3-p5kdq\" (UID: \"d7eee821-a910-467e-9a41-16ed03cfc82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq" Oct 30 16:38:21.259113 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.259088 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7eee821-a910-467e-9a41-16ed03cfc82e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23ff8ff3-p5kdq\" (UID: \"d7eee821-a910-467e-9a41-16ed03cfc82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq" Oct 30 16:38:21.259230 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.259211 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59kt4\" (UniqueName: \"kubernetes.io/projected/d7eee821-a910-467e-9a41-16ed03cfc82e-kube-api-access-59kt4\") pod \"maas-api-revoke-sub-23ff8ff3-p5kdq\" (UID: \"d7eee821-a910-467e-9a41-16ed03cfc82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq" Oct 30 16:38:21.360195 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.360158 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7eee821-a910-467e-9a41-16ed03cfc82e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23ff8ff3-p5kdq\" (UID: \"d7eee821-a910-467e-9a41-16ed03cfc82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq" Oct 30 16:38:21.360418 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.360282 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59kt4\" (UniqueName: \"kubernetes.io/projected/d7eee821-a910-467e-9a41-16ed03cfc82e-kube-api-access-59kt4\") pod \"maas-api-revoke-sub-23ff8ff3-p5kdq\" (UID: \"d7eee821-a910-467e-9a41-16ed03cfc82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq" Oct 30 16:38:21.360418 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.360344 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7eee821-a910-467e-9a41-16ed03cfc82e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23ff8ff3-p5kdq\" (UID: \"d7eee821-a910-467e-9a41-16ed03cfc82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq" Oct 30 16:38:21.361157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.361131 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7eee821-a910-467e-9a41-16ed03cfc82e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23ff8ff3-p5kdq\" (UID: \"d7eee821-a910-467e-9a41-16ed03cfc82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq" Oct 30 16:38:21.362813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.362782 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7eee821-a910-467e-9a41-16ed03cfc82e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23ff8ff3-p5kdq\" (UID: \"d7eee821-a910-467e-9a41-16ed03cfc82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq" Oct 30 16:38:21.383609 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.383579 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59kt4\" (UniqueName: \"kubernetes.io/projected/d7eee821-a910-467e-9a41-16ed03cfc82e-kube-api-access-59kt4\") pod \"maas-api-revoke-sub-23ff8ff3-p5kdq\" (UID: \"d7eee821-a910-467e-9a41-16ed03cfc82e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq" Oct 30 16:38:21.423493 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.423404 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:21.472709 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.472676 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq" Oct 30 16:38:21.800730 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:38:21.800702 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7eee821_a910_467e_9a41_16ed03cfc82e.slice/crio-2a55fe59e51f4b2d5fa0fc5cebd6a0bb712b0524c1fc29785b62ddfb852f1302 WatchSource:0}: Error finding container 2a55fe59e51f4b2d5fa0fc5cebd6a0bb712b0524c1fc29785b62ddfb852f1302: Status 404 returned error can't find the container with id 2a55fe59e51f4b2d5fa0fc5cebd6a0bb712b0524c1fc29785b62ddfb852f1302 Oct 30 16:38:21.800893 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:21.800805 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq"] Oct 30 16:38:22.045946 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:22.045914 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq" event={"ID":"d7eee821-a910-467e-9a41-16ed03cfc82e","Type":"ContainerStarted","Data":"cc0853ee64225f3e2b2490e75d6f311e6fcb1b6bb52b17bfa3ecfeae4b1b5db1"} Oct 30 16:38:22.046105 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:22.045973 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq" event={"ID":"d7eee821-a910-467e-9a41-16ed03cfc82e","Type":"ContainerStarted","Data":"2a55fe59e51f4b2d5fa0fc5cebd6a0bb712b0524c1fc29785b62ddfb852f1302"} Oct 30 16:38:22.102566 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:22.102514 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq" podStartSLOduration=1.102501034 podStartE2EDuration="1.102501034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:22.099404645 +0000 UTC m=+1015.809859388" watchObservedRunningTime="2026-10-30 16:38:22.102501034 +0000 UTC m=+1015.812955833" Oct 30 16:38:23.055610 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:23.055509 6202 generic.go:356] "Generic (PLEG): container finished" podID="d7eee821-a910-467e-9a41-16ed03cfc82e" containerID="cc0853ee64225f3e2b2490e75d6f311e6fcb1b6bb52b17bfa3ecfeae4b1b5db1" exitCode=0 Oct 30 16:38:23.056047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:23.055600 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq" event={"ID":"d7eee821-a910-467e-9a41-16ed03cfc82e","Type":"ContainerDied","Data":"cc0853ee64225f3e2b2490e75d6f311e6fcb1b6bb52b17bfa3ecfeae4b1b5db1"} Oct 30 16:38:23.179482 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:23.179445 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:24.727788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:24.727435 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:24.828031 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:24.823542 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 16:38:24.877820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:24.874107 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:24.945793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:24.917910 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 16:38:24.945793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:24.918270 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:38:24.958876 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:24.947631 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:24.983878 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:24.982877 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.252847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.245195 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:38:25.317831 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.303997 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq" Oct 30 16:38:25.412793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.401044 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7eee821-a910-467e-9a41-16ed03cfc82e-kube-api-access-59kt4\" (UniqueName: \"kubernetes.io/projected/d7eee821-a910-467e-9a41-16ed03cfc82e-kube-api-access-59kt4\") pod \"d7eee821-a910-467e-9a41-16ed03cfc82e\" (UID: \"d7eee821-a910-467e-9a41-16ed03cfc82e\") " Oct 30 16:38:25.457436 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.457395 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7eee821-a910-467e-9a41-16ed03cfc82e-maas-api-openshift-ca" pod "d7eee821-a910-467e-9a41-16ed03cfc82e" (UID: "d7eee821-a910-467e-9a41-16ed03cfc82e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:25.496302 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.496258 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.498480 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.498455 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7eee821-a910-467e-9a41-16ed03cfc82e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7eee821-a910-467e-9a41-16ed03cfc82e-maas-api-openshift-ca\") pod \"d7eee821-a910-467e-9a41-16ed03cfc82e\" (UID: \"d7eee821-a910-467e-9a41-16ed03cfc82e\") " Oct 30 16:38:25.498736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.498718 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7eee821-a910-467e-9a41-16ed03cfc82e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7eee821-a910-467e-9a41-16ed03cfc82e-maas-api-xks-ca\") pod \"d7eee821-a910-467e-9a41-16ed03cfc82e\" (UID: \"d7eee821-a910-467e-9a41-16ed03cfc82e\") " Oct 30 16:38:25.504582 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.504522 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7eee821-a910-467e-9a41-16ed03cfc82e-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:25.536600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.536567 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7eee821-a910-467e-9a41-16ed03cfc82e-kube-api-access-59kt4" pod "d7eee821-a910-467e-9a41-16ed03cfc82e" (UID: "d7eee821-a910-467e-9a41-16ed03cfc82e"). InnerVolumeSpecName "kube-api-access-59kt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:25.544184 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.543857 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.570292 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.570251 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7eee821-a910-467e-9a41-16ed03cfc82e-maas-api-xks-ca" pod "d7eee821-a910-467e-9a41-16ed03cfc82e" (UID: "d7eee821-a910-467e-9a41-16ed03cfc82e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:25.636136 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.636092 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59kt4\" (UniqueName: \"kubernetes.io/projected/d7eee821-a910-467e-9a41-16ed03cfc82e-kube-api-access-59kt4\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:25.636361 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.636342 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7eee821-a910-467e-9a41-16ed03cfc82e-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:25.832795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.830865 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.832795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.832747 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:38:25.885810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.882994 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.885810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.883952 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:38:25.918117 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.918075 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.920202 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.919715 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:38:25.920202 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.918391 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.924786 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.920470 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:38:25.973795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.948821 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded" Oct 30 16:38:26.000793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.975317 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:38:26.000793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:25.992198 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.028138 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:26.003437 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:38:26.175506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:26.173336 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:26.259841 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:26.259800 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.268857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:26.268825 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:38:26.636794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:26.630252 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.960911 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:26.930718 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 16:38:26.990794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:26.964869 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.990794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:26.965003 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:38:27.055794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:27.019926 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.055794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:27.020075 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 16:38:27.191404 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:27.191360 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:27.191702 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:27.191670 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" podUID="05d8ddc3-7f1c-4c56-a906-3722b011ab90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.431796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:27.386853 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:27.456795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:27.432078 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" podUID="35198b3d-3a56-45c7-8d36-d4de1780a938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.496361 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:27.457145 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 16:38:27.734929 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:27.734817 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:38:28.207024 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:38:28.206989 6202 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.805s" Oct 30 16:38:28.210918 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:28.210891 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:38:28.294456 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:28.294417 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq" event={"ID":"d7eee821-a910-467e-9a41-16ed03cfc82e","Type":"ContainerDied","Data":"2a55fe59e51f4b2d5fa0fc5cebd6a0bb712b0524c1fc29785b62ddfb852f1302"} Oct 30 16:38:28.301085 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:28.301058 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a55fe59e51f4b2d5fa0fc5cebd6a0bb712b0524c1fc29785b62ddfb852f1302" Oct 30 16:38:28.301336 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:28.294735 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq" Oct 30 16:38:28.743923 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:28.743890 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:28.783678 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:28.771916 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:38:28.864307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:28.784619 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:38:28.864307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:28.822932 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:28.864307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:28.825795 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:28.864307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:28.825828 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded" Oct 30 16:38:28.864307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:28.825850 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:38:28.864307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:28.827883 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:28.864307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:28.856869 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:28.915543 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:28.915138 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:28.915543 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:28.915193 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" podUID="92906137-4350-4921-813c-cdcaf92a2143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:28.915543 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:28.915271 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:29.049443 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.049325 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:38:29.050806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.050784 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:38:29.052635 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.052619 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:38:29.069839 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.069811 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv"] Oct 30 16:38:29.082013 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.081983 6202 cpu_manager.go:414] "containerMap: removing container" podUID="d7eee821-a910-467e-9a41-16ed03cfc82e" containerName="revoke-keys" Oct 30 16:38:29.082197 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.082184 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7eee821-a910-467e-9a41-16ed03cfc82e" containerName="revoke-keys" Oct 30 16:38:29.082515 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.082497 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7eee821-a910-467e-9a41-16ed03cfc82e" containerName="revoke-keys" Oct 30 16:38:29.117920 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:38:29.117881 6202 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7eee821_a910_467e_9a41_16ed03cfc82e.slice\": RecentStats: unable to find data in memory cache]" Oct 30 16:38:29.140445 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.140413 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv"] Oct 30 16:38:29.151015 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.150986 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:38:29.151355 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.151329 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:38:29.152029 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.152007 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:38:29.152153 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.152109 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:38:29.153804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.140753 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv" Oct 30 16:38:29.176638 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.176355 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:38:29.177126 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.177093 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:38:29.177625 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.177608 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:38:29.177902 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.177866 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:38:29.221645 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.221599 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e083e19b-60cc-4d85-b312-ac6d93ea66ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fd04269-xzrnv\" (UID: \"e083e19b-60cc-4d85-b312-ac6d93ea66ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv" Oct 30 16:38:29.222200 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.221687 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e083e19b-60cc-4d85-b312-ac6d93ea66ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fd04269-xzrnv\" (UID: \"e083e19b-60cc-4d85-b312-ac6d93ea66ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv" Oct 30 16:38:29.222200 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.221814 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4ln4\" (UniqueName: \"kubernetes.io/projected/e083e19b-60cc-4d85-b312-ac6d93ea66ec-kube-api-access-b4ln4\") pod \"maas-api-revoke-sub-0fd04269-xzrnv\" (UID: \"e083e19b-60cc-4d85-b312-ac6d93ea66ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv" Oct 30 16:38:29.323031 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.323004 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e083e19b-60cc-4d85-b312-ac6d93ea66ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fd04269-xzrnv\" (UID: \"e083e19b-60cc-4d85-b312-ac6d93ea66ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv" Oct 30 16:38:29.323189 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.323064 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e083e19b-60cc-4d85-b312-ac6d93ea66ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fd04269-xzrnv\" (UID: \"e083e19b-60cc-4d85-b312-ac6d93ea66ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv" Oct 30 16:38:29.323189 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.323168 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4ln4\" (UniqueName: \"kubernetes.io/projected/e083e19b-60cc-4d85-b312-ac6d93ea66ec-kube-api-access-b4ln4\") pod \"maas-api-revoke-sub-0fd04269-xzrnv\" (UID: \"e083e19b-60cc-4d85-b312-ac6d93ea66ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv" Oct 30 16:38:29.323834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.323811 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e083e19b-60cc-4d85-b312-ac6d93ea66ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fd04269-xzrnv\" (UID: \"e083e19b-60cc-4d85-b312-ac6d93ea66ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv" Oct 30 16:38:29.325557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.325537 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e083e19b-60cc-4d85-b312-ac6d93ea66ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fd04269-xzrnv\" (UID: \"e083e19b-60cc-4d85-b312-ac6d93ea66ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv" Oct 30 16:38:29.346980 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.346948 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4ln4\" (UniqueName: \"kubernetes.io/projected/e083e19b-60cc-4d85-b312-ac6d93ea66ec-kube-api-access-b4ln4\") pod \"maas-api-revoke-sub-0fd04269-xzrnv\" (UID: \"e083e19b-60cc-4d85-b312-ac6d93ea66ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv" Oct 30 16:38:29.487001 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.486966 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv" Oct 30 16:38:29.803227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:38:29.803197 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode083e19b_60cc_4d85_b312_ac6d93ea66ec.slice/crio-5b39a913e3e1e88b398acdad8051b350617334e3a4795c1732b9035a9ac8aeb3 WatchSource:0}: Error finding container 5b39a913e3e1e88b398acdad8051b350617334e3a4795c1732b9035a9ac8aeb3: Status 404 returned error can't find the container with id 5b39a913e3e1e88b398acdad8051b350617334e3a4795c1732b9035a9ac8aeb3 Oct 30 16:38:29.804667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:29.804639 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv"] Oct 30 16:38:30.311615 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:30.311527 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv" event={"ID":"e083e19b-60cc-4d85-b312-ac6d93ea66ec","Type":"ContainerStarted","Data":"c47f770bd16becbd313d7d36bac6f2c5c0d2588c21c890c2fb2fcf593b2ea069"} Oct 30 16:38:30.311615 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:30.311589 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv" event={"ID":"e083e19b-60cc-4d85-b312-ac6d93ea66ec","Type":"ContainerStarted","Data":"5b39a913e3e1e88b398acdad8051b350617334e3a4795c1732b9035a9ac8aeb3"} Oct 30 16:38:30.352725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:30.352679 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv" podStartSLOduration=4.35266723 podStartE2EDuration="4.35266723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:30.351018049 +0000 UTC m=+1024.061472793" watchObservedRunningTime="2026-10-30 16:38:30.35266723 +0000 UTC m=+1024.063122001" Oct 30 16:38:30.400539 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:30.400486 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:31.320920 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:31.320885 6202 generic.go:356] "Generic (PLEG): container finished" podID="e083e19b-60cc-4d85-b312-ac6d93ea66ec" containerID="c47f770bd16becbd313d7d36bac6f2c5c0d2588c21c890c2fb2fcf593b2ea069" exitCode=0 Oct 30 16:38:31.321316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:31.320966 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv" event={"ID":"e083e19b-60cc-4d85-b312-ac6d93ea66ec","Type":"ContainerDied","Data":"c47f770bd16becbd313d7d36bac6f2c5c0d2588c21c890c2fb2fcf593b2ea069"} Oct 30 16:38:31.971875 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:31.971837 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:32.496726 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:32.496700 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv" Oct 30 16:38:32.568547 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:32.568515 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e083e19b-60cc-4d85-b312-ac6d93ea66ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e083e19b-60cc-4d85-b312-ac6d93ea66ec-maas-api-xks-ca\") pod \"e083e19b-60cc-4d85-b312-ac6d93ea66ec\" (UID: \"e083e19b-60cc-4d85-b312-ac6d93ea66ec\") " Oct 30 16:38:32.568750 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:32.568682 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e083e19b-60cc-4d85-b312-ac6d93ea66ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e083e19b-60cc-4d85-b312-ac6d93ea66ec-maas-api-openshift-ca\") pod \"e083e19b-60cc-4d85-b312-ac6d93ea66ec\" (UID: \"e083e19b-60cc-4d85-b312-ac6d93ea66ec\") " Oct 30 16:38:32.568927 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:32.568848 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e083e19b-60cc-4d85-b312-ac6d93ea66ec-kube-api-access-b4ln4\" (UniqueName: \"kubernetes.io/projected/e083e19b-60cc-4d85-b312-ac6d93ea66ec-kube-api-access-b4ln4\") pod \"e083e19b-60cc-4d85-b312-ac6d93ea66ec\" (UID: \"e083e19b-60cc-4d85-b312-ac6d93ea66ec\") " Oct 30 16:38:32.569091 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:32.569065 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e083e19b-60cc-4d85-b312-ac6d93ea66ec-maas-api-openshift-ca" pod "e083e19b-60cc-4d85-b312-ac6d93ea66ec" (UID: "e083e19b-60cc-4d85-b312-ac6d93ea66ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:32.569696 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:32.569670 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e083e19b-60cc-4d85-b312-ac6d93ea66ec-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:32.570893 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:32.570848 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e083e19b-60cc-4d85-b312-ac6d93ea66ec-maas-api-xks-ca" pod "e083e19b-60cc-4d85-b312-ac6d93ea66ec" (UID: "e083e19b-60cc-4d85-b312-ac6d93ea66ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:32.571172 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:32.571142 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e083e19b-60cc-4d85-b312-ac6d93ea66ec-kube-api-access-b4ln4" pod "e083e19b-60cc-4d85-b312-ac6d93ea66ec" (UID: "e083e19b-60cc-4d85-b312-ac6d93ea66ec"). InnerVolumeSpecName "kube-api-access-b4ln4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:32.671093 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:32.671056 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e083e19b-60cc-4d85-b312-ac6d93ea66ec-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:32.671093 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:32.671083 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4ln4\" (UniqueName: \"kubernetes.io/projected/e083e19b-60cc-4d85-b312-ac6d93ea66ec-kube-api-access-b4ln4\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:33.336069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.336038 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv" Oct 30 16:38:33.336243 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.336077 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv" event={"ID":"e083e19b-60cc-4d85-b312-ac6d93ea66ec","Type":"ContainerDied","Data":"5b39a913e3e1e88b398acdad8051b350617334e3a4795c1732b9035a9ac8aeb3"} Oct 30 16:38:33.336243 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.336122 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b39a913e3e1e88b398acdad8051b350617334e3a4795c1732b9035a9ac8aeb3" Oct 30 16:38:33.459024 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.458991 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb"] Oct 30 16:38:33.459663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.459649 6202 cpu_manager.go:414] "containerMap: removing container" podUID="e083e19b-60cc-4d85-b312-ac6d93ea66ec" containerName="revoke-keys" Oct 30 16:38:33.459706 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.459665 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e083e19b-60cc-4d85-b312-ac6d93ea66ec" containerName="revoke-keys" Oct 30 16:38:33.459757 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.459748 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="e083e19b-60cc-4d85-b312-ac6d93ea66ec" containerName="revoke-keys" Oct 30 16:38:33.463439 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.463419 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb" Oct 30 16:38:33.479579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.479287 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:38:33.479579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.479339 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:38:33.479880 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.479623 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:38:33.485013 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.484984 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb"] Oct 30 16:38:33.490584 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.490556 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:38:33.538223 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.538183 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:33.585531 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.585414 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7wnh\" (UniqueName: \"kubernetes.io/projected/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-kube-api-access-t7wnh\") pod \"maas-api-revoke-sub-0d9821a5-6n7sb\" (UID: \"c28d5268-f22a-4970-84d6-0bcafaa5ee7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb" Oct 30 16:38:33.585531 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.585538 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d9821a5-6n7sb\" (UID: \"c28d5268-f22a-4970-84d6-0bcafaa5ee7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb" Oct 30 16:38:33.585873 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.585741 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d9821a5-6n7sb\" (UID: \"c28d5268-f22a-4970-84d6-0bcafaa5ee7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb" Oct 30 16:38:33.687570 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.687463 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d9821a5-6n7sb\" (UID: \"c28d5268-f22a-4970-84d6-0bcafaa5ee7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb" Oct 30 16:38:33.687749 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.687626 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d9821a5-6n7sb\" (UID: \"c28d5268-f22a-4970-84d6-0bcafaa5ee7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb" Oct 30 16:38:33.687853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.687800 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7wnh\" (UniqueName: \"kubernetes.io/projected/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-kube-api-access-t7wnh\") pod \"maas-api-revoke-sub-0d9821a5-6n7sb\" (UID: \"c28d5268-f22a-4970-84d6-0bcafaa5ee7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb" Oct 30 16:38:33.688456 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.688423 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d9821a5-6n7sb\" (UID: \"c28d5268-f22a-4970-84d6-0bcafaa5ee7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb" Oct 30 16:38:33.690289 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.690258 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d9821a5-6n7sb\" (UID: \"c28d5268-f22a-4970-84d6-0bcafaa5ee7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb" Oct 30 16:38:33.711517 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.711485 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7wnh\" (UniqueName: \"kubernetes.io/projected/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-kube-api-access-t7wnh\") pod \"maas-api-revoke-sub-0d9821a5-6n7sb\" (UID: \"c28d5268-f22a-4970-84d6-0bcafaa5ee7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb" Oct 30 16:38:33.780693 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:33.780654 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb" Oct 30 16:38:34.086789 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:38:34.086720 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28d5268_f22a_4970_84d6_0bcafaa5ee7f.slice/crio-0d5ee422aebc24a4134c33815ff0f56207001eef84cb7f6553c8f1b5a13504ae WatchSource:0}: Error finding container 0d5ee422aebc24a4134c33815ff0f56207001eef84cb7f6553c8f1b5a13504ae: Status 404 returned error can't find the container with id 0d5ee422aebc24a4134c33815ff0f56207001eef84cb7f6553c8f1b5a13504ae Oct 30 16:38:34.087242 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:34.087219 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb"] Oct 30 16:38:34.347270 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:34.347240 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb" event={"ID":"c28d5268-f22a-4970-84d6-0bcafaa5ee7f","Type":"ContainerStarted","Data":"0d5ee422aebc24a4134c33815ff0f56207001eef84cb7f6553c8f1b5a13504ae"} Oct 30 16:38:35.152034 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:35.151979 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:35.356848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:35.356804 6202 generic.go:356] "Generic (PLEG): container finished" podID="c28d5268-f22a-4970-84d6-0bcafaa5ee7f" containerID="22a47864ccfc2e4360dfce577b65bad93f8a4d1bba6c8c8081958503826487a5" exitCode=0 Oct 30 16:38:35.357046 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:35.356906 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb" event={"ID":"c28d5268-f22a-4970-84d6-0bcafaa5ee7f","Type":"ContainerDied","Data":"22a47864ccfc2e4360dfce577b65bad93f8a4d1bba6c8c8081958503826487a5"} Oct 30 16:38:36.920791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:36.919928 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:37.241793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:37.231064 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:37.865961 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:37.865917 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:37.961420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:37.936219 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:37.961420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:37.938380 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 16:38:37.979290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:37.977910 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:37.994985 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:37.994883 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.003812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.003100 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.032906 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.032869 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 16:38:38.036018 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.033049 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.120842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.110053 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb" Oct 30 16:38:38.328795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.302626 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-kube-api-access-t7wnh\" (UniqueName: \"kubernetes.io/projected/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-kube-api-access-t7wnh\") pod \"c28d5268-f22a-4970-84d6-0bcafaa5ee7f\" (UID: \"c28d5268-f22a-4970-84d6-0bcafaa5ee7f\") " Oct 30 16:38:38.345802 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.329205 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-maas-api-xks-ca\") pod \"c28d5268-f22a-4970-84d6-0bcafaa5ee7f\" (UID: \"c28d5268-f22a-4970-84d6-0bcafaa5ee7f\") " Oct 30 16:38:38.370638 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.369934 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-maas-api-openshift-ca\") pod \"c28d5268-f22a-4970-84d6-0bcafaa5ee7f\" (UID: \"c28d5268-f22a-4970-84d6-0bcafaa5ee7f\") " Oct 30 16:38:38.405856 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.379042 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-maas-api-openshift-ca" pod "c28d5268-f22a-4970-84d6-0bcafaa5ee7f" (UID: "c28d5268-f22a-4970-84d6-0bcafaa5ee7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:38.488856 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.406209 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-kube-api-access-t7wnh" pod "c28d5268-f22a-4970-84d6-0bcafaa5ee7f" (UID: "c28d5268-f22a-4970-84d6-0bcafaa5ee7f"). InnerVolumeSpecName "kube-api-access-t7wnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:38.598561 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.515587 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:38.598859 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.429624 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-maas-api-xks-ca" pod "c28d5268-f22a-4970-84d6-0bcafaa5ee7f" (UID: "c28d5268-f22a-4970-84d6-0bcafaa5ee7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:38.740854 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.738674 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:38.740854 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.738717 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7wnh\" (UniqueName: \"kubernetes.io/projected/c28d5268-f22a-4970-84d6-0bcafaa5ee7f-kube-api-access-t7wnh\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:38.963171 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.958698 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.963171 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.961191 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 16:38:38.978792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.975826 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:38:38.978792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.975923 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:38:38.978792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.977069 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 16:38:39.006323 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.985268 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.006323 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.986564 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:38:39.006323 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:38.986645 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:38:39.030907 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:39.017575 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.267885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:39.059239 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 16:38:39.280462 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:39.280427 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:38:39.282227 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:39.282205 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:38:39.300914 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:39.059478 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 16:38:39.300914 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:39.291786 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:38:39.301559 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:39.095888 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:38:39.686394 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:39.682300 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb" event={"ID":"c28d5268-f22a-4970-84d6-0bcafaa5ee7f","Type":"ContainerDied","Data":"0d5ee422aebc24a4134c33815ff0f56207001eef84cb7f6553c8f1b5a13504ae"} Oct 30 16:38:39.686394 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:39.682370 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d5ee422aebc24a4134c33815ff0f56207001eef84cb7f6553c8f1b5a13504ae" Oct 30 16:38:39.686394 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:39.682486 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb" Oct 30 16:38:40.006851 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:39.999014 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.056793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:40.047169 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 16:38:40.110844 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:40.102860 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.316756 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:40.314820 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.316756 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:40.315124 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.316756 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:40.315339 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.316756 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:40.315854 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.316756 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:40.316162 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.328808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:40.328673 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:38:40.328808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:40.328739 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:38:40.367614 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:40.366116 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:40.377994 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:40.377968 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:38:40.379609 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:40.379591 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:38:40.601585 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:40.601533 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded" Oct 30 16:38:40.739760 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:40.739730 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:38:40.766378 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:40.766350 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:38:40.769084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:40.769060 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:38:40.774086 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:40.774064 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:38:40.775558 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:40.775538 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:38:41.953260 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:41.953226 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:43.578263 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:43.578227 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:44.071273 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:38:44.071239 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:38:45.220093 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:45.220057 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:46.783180 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:46.783144 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:48.479510 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:48.479476 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:50.500871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:50.500835 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:52.023826 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:52.023756 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:53.586623 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:53.586584 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:54.670398 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.670362 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx"] Oct 30 16:38:54.671020 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.671005 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c28d5268-f22a-4970-84d6-0bcafaa5ee7f" containerName="revoke-keys" Oct 30 16:38:54.671020 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.671019 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c28d5268-f22a-4970-84d6-0bcafaa5ee7f" containerName="revoke-keys" Oct 30 16:38:54.671195 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.671149 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c28d5268-f22a-4970-84d6-0bcafaa5ee7f" containerName="revoke-keys" Oct 30 16:38:54.675177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.675156 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx" Oct 30 16:38:54.695872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.695851 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx"] Oct 30 16:38:54.702100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.702074 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:38:54.702218 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.702165 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:38:54.702283 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.702268 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:38:54.702327 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.702317 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:38:54.838085 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.838045 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b43d22-8371-40a4-aa48-3baf6e3f96b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-090532a8-dlzfx\" (UID: \"34b43d22-8371-40a4-aa48-3baf6e3f96b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx" Oct 30 16:38:54.838271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.838241 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b43d22-8371-40a4-aa48-3baf6e3f96b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-090532a8-dlzfx\" (UID: \"34b43d22-8371-40a4-aa48-3baf6e3f96b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx" Oct 30 16:38:54.838324 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.838272 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqck8\" (UniqueName: \"kubernetes.io/projected/34b43d22-8371-40a4-aa48-3baf6e3f96b2-kube-api-access-vqck8\") pod \"maas-api-revoke-sub-090532a8-dlzfx\" (UID: \"34b43d22-8371-40a4-aa48-3baf6e3f96b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx" Oct 30 16:38:54.939186 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.939102 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b43d22-8371-40a4-aa48-3baf6e3f96b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-090532a8-dlzfx\" (UID: \"34b43d22-8371-40a4-aa48-3baf6e3f96b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx" Oct 30 16:38:54.939337 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.939226 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b43d22-8371-40a4-aa48-3baf6e3f96b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-090532a8-dlzfx\" (UID: \"34b43d22-8371-40a4-aa48-3baf6e3f96b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx" Oct 30 16:38:54.939337 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.939250 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqck8\" (UniqueName: \"kubernetes.io/projected/34b43d22-8371-40a4-aa48-3baf6e3f96b2-kube-api-access-vqck8\") pod \"maas-api-revoke-sub-090532a8-dlzfx\" (UID: \"34b43d22-8371-40a4-aa48-3baf6e3f96b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx" Oct 30 16:38:54.940087 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.940009 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b43d22-8371-40a4-aa48-3baf6e3f96b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-090532a8-dlzfx\" (UID: \"34b43d22-8371-40a4-aa48-3baf6e3f96b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx" Oct 30 16:38:54.941897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.941870 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b43d22-8371-40a4-aa48-3baf6e3f96b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-090532a8-dlzfx\" (UID: \"34b43d22-8371-40a4-aa48-3baf6e3f96b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx" Oct 30 16:38:54.962992 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.962961 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqck8\" (UniqueName: \"kubernetes.io/projected/34b43d22-8371-40a4-aa48-3baf6e3f96b2-kube-api-access-vqck8\") pod \"maas-api-revoke-sub-090532a8-dlzfx\" (UID: \"34b43d22-8371-40a4-aa48-3baf6e3f96b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx" Oct 30 16:38:54.991871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:54.991840 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx" Oct 30 16:38:55.213956 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:55.213871 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:55.308302 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:55.308270 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx"] Oct 30 16:38:55.311568 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:38:55.311539 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b43d22_8371_40a4_aa48_3baf6e3f96b2.slice/crio-308de9574065ebf9750adb9a4d8924242b601941c61c2d2aeb9ed9f72bf9f7f2 WatchSource:0}: Error finding container 308de9574065ebf9750adb9a4d8924242b601941c61c2d2aeb9ed9f72bf9f7f2: Status 404 returned error can't find the container with id 308de9574065ebf9750adb9a4d8924242b601941c61c2d2aeb9ed9f72bf9f7f2 Oct 30 16:38:55.827632 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:55.827595 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx" event={"ID":"34b43d22-8371-40a4-aa48-3baf6e3f96b2","Type":"ContainerStarted","Data":"ae1ecd0319c472802f9581b1cd1aaac5effe3ccd6e94a68da1e2a35bf79f91e1"} Oct 30 16:38:55.828060 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:55.827652 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx" event={"ID":"34b43d22-8371-40a4-aa48-3baf6e3f96b2","Type":"ContainerStarted","Data":"308de9574065ebf9750adb9a4d8924242b601941c61c2d2aeb9ed9f72bf9f7f2"} Oct 30 16:38:55.871111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:55.871052 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx" podStartSLOduration=1.871037013 podStartE2EDuration="1.871037013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:55.868232852 +0000 UTC m=+1049.578687598" watchObservedRunningTime="2026-10-30 16:38:55.871037013 +0000 UTC m=+1049.581491759" Oct 30 16:38:56.752031 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:56.751986 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:56.840981 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:56.839891 6202 generic.go:356] "Generic (PLEG): container finished" podID="34b43d22-8371-40a4-aa48-3baf6e3f96b2" containerID="ae1ecd0319c472802f9581b1cd1aaac5effe3ccd6e94a68da1e2a35bf79f91e1" exitCode=0 Oct 30 16:38:56.840981 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:56.839963 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx" event={"ID":"34b43d22-8371-40a4-aa48-3baf6e3f96b2","Type":"ContainerDied","Data":"ae1ecd0319c472802f9581b1cd1aaac5effe3ccd6e94a68da1e2a35bf79f91e1"} Oct 30 16:38:58.019217 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:58.019192 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx" Oct 30 16:38:58.187863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:58.187756 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b43d22-8371-40a4-aa48-3baf6e3f96b2-kube-api-access-vqck8\" (UniqueName: \"kubernetes.io/projected/34b43d22-8371-40a4-aa48-3baf6e3f96b2-kube-api-access-vqck8\") pod \"34b43d22-8371-40a4-aa48-3baf6e3f96b2\" (UID: \"34b43d22-8371-40a4-aa48-3baf6e3f96b2\") " Oct 30 16:38:58.188042 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:58.187883 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b43d22-8371-40a4-aa48-3baf6e3f96b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b43d22-8371-40a4-aa48-3baf6e3f96b2-maas-api-openshift-ca\") pod \"34b43d22-8371-40a4-aa48-3baf6e3f96b2\" (UID: \"34b43d22-8371-40a4-aa48-3baf6e3f96b2\") " Oct 30 16:38:58.188042 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:58.187988 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34b43d22-8371-40a4-aa48-3baf6e3f96b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b43d22-8371-40a4-aa48-3baf6e3f96b2-maas-api-xks-ca\") pod \"34b43d22-8371-40a4-aa48-3baf6e3f96b2\" (UID: \"34b43d22-8371-40a4-aa48-3baf6e3f96b2\") " Oct 30 16:38:58.188280 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:58.188255 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b43d22-8371-40a4-aa48-3baf6e3f96b2-maas-api-openshift-ca" pod "34b43d22-8371-40a4-aa48-3baf6e3f96b2" (UID: "34b43d22-8371-40a4-aa48-3baf6e3f96b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:58.188550 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:58.188527 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b43d22-8371-40a4-aa48-3baf6e3f96b2-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:58.190004 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:58.189977 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b43d22-8371-40a4-aa48-3baf6e3f96b2-kube-api-access-vqck8" pod "34b43d22-8371-40a4-aa48-3baf6e3f96b2" (UID: "34b43d22-8371-40a4-aa48-3baf6e3f96b2"). InnerVolumeSpecName "kube-api-access-vqck8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:58.190099 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:58.190048 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b43d22-8371-40a4-aa48-3baf6e3f96b2-maas-api-xks-ca" pod "34b43d22-8371-40a4-aa48-3baf6e3f96b2" (UID: "34b43d22-8371-40a4-aa48-3baf6e3f96b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:58.289650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:58.289619 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b43d22-8371-40a4-aa48-3baf6e3f96b2-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:58.289650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:58.289646 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqck8\" (UniqueName: \"kubernetes.io/projected/34b43d22-8371-40a4-aa48-3baf6e3f96b2-kube-api-access-vqck8\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:38:58.473583 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:58.473504 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:58.918178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:58.918130 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx" event={"ID":"34b43d22-8371-40a4-aa48-3baf6e3f96b2","Type":"ContainerDied","Data":"308de9574065ebf9750adb9a4d8924242b601941c61c2d2aeb9ed9f72bf9f7f2"} Oct 30 16:38:58.918462 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:58.918439 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="308de9574065ebf9750adb9a4d8924242b601941c61c2d2aeb9ed9f72bf9f7f2" Oct 30 16:38:58.918663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:38:58.918646 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx" Oct 30 16:39:00.050916 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:00.050864 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:00.111655 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:00.111610 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:00.317078 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:00.316967 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:00.329640 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:00.328848 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 16:39:00.329999 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:00.329931 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:00.387898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:00.387855 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 16:39:00.559823 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:00.559330 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded" Oct 30 16:39:00.777432 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:00.776154 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:00.782588 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:00.781849 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.053968 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.053859 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.055351 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.055324 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:39:01.112204 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.112161 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.146396 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.146349 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:39:01.230799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.229550 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.346477 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.346428 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:39:01.406289 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.350609 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 16:39:01.429564 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.405663 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.429859 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.406746 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 16:39:01.528896 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.528866 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:39:01.529166 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.529153 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:39:01.529300 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.529289 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:39:01.529411 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.529402 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:39:01.549791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.545746 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:01.783956 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.783834 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.821176 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.792939 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.857198 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.857170 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:39:01.943801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.935928 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:39:01.947828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:01.945285 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:02.097419 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:02.067918 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 16:39:02.184799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:02.164919 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:02.413847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:02.412880 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Oct 30 16:39:02.532315 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:02.516825 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:02.532315 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:02.515034 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/health\": context deadline exceeded" Oct 30 16:39:02.581078 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:02.581038 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:02.581648 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:02.581616 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:39:02.721287 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:02.629790 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:02.721287 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:02.643166 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:39:02.721287 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:02.717472 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:39:02.721287 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:02.717510 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:39:02.723173 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:02.723147 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:39:02.739876 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:02.739850 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:39:02.740152 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:02.740139 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:39:02.742174 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:02.742154 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:39:02.804161 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:02.804131 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:39:03.488341 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.488310 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc"] Oct 30 16:39:03.488964 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.488951 6202 cpu_manager.go:414] "containerMap: removing container" podUID="34b43d22-8371-40a4-aa48-3baf6e3f96b2" containerName="revoke-keys" Oct 30 16:39:03.489019 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.488965 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b43d22-8371-40a4-aa48-3baf6e3f96b2" containerName="revoke-keys" Oct 30 16:39:03.489061 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.489056 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b43d22-8371-40a4-aa48-3baf6e3f96b2" containerName="revoke-keys" Oct 30 16:39:03.493152 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.493127 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc" Oct 30 16:39:03.508331 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.508289 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:39:03.508331 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.508292 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:39:03.508331 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.508311 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:39:03.511558 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.511537 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:39:03.511999 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.511976 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc"] Oct 30 16:39:03.568070 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.568037 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbad445a-ffd5-427b-be3c-59225c098c21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d35981fb-cpcdc\" (UID: \"cbad445a-ffd5-427b-be3c-59225c098c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc" Oct 30 16:39:03.568235 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.568211 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbad445a-ffd5-427b-be3c-59225c098c21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d35981fb-cpcdc\" (UID: \"cbad445a-ffd5-427b-be3c-59225c098c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc" Oct 30 16:39:03.568304 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.568286 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75zmr\" (UniqueName: \"kubernetes.io/projected/cbad445a-ffd5-427b-be3c-59225c098c21-kube-api-access-75zmr\") pod \"maas-api-revoke-sub-d35981fb-cpcdc\" (UID: \"cbad445a-ffd5-427b-be3c-59225c098c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc" Oct 30 16:39:03.669496 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.669458 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbad445a-ffd5-427b-be3c-59225c098c21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d35981fb-cpcdc\" (UID: \"cbad445a-ffd5-427b-be3c-59225c098c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc" Oct 30 16:39:03.669662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.669595 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbad445a-ffd5-427b-be3c-59225c098c21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d35981fb-cpcdc\" (UID: \"cbad445a-ffd5-427b-be3c-59225c098c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc" Oct 30 16:39:03.669662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.669655 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75zmr\" (UniqueName: \"kubernetes.io/projected/cbad445a-ffd5-427b-be3c-59225c098c21-kube-api-access-75zmr\") pod \"maas-api-revoke-sub-d35981fb-cpcdc\" (UID: \"cbad445a-ffd5-427b-be3c-59225c098c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc" Oct 30 16:39:03.670190 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.670168 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbad445a-ffd5-427b-be3c-59225c098c21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d35981fb-cpcdc\" (UID: \"cbad445a-ffd5-427b-be3c-59225c098c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc" Oct 30 16:39:03.671885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.671869 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbad445a-ffd5-427b-be3c-59225c098c21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d35981fb-cpcdc\" (UID: \"cbad445a-ffd5-427b-be3c-59225c098c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc" Oct 30 16:39:03.690561 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.690528 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:03.693034 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.693015 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75zmr\" (UniqueName: \"kubernetes.io/projected/cbad445a-ffd5-427b-be3c-59225c098c21-kube-api-access-75zmr\") pod \"maas-api-revoke-sub-d35981fb-cpcdc\" (UID: \"cbad445a-ffd5-427b-be3c-59225c098c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc" Oct 30 16:39:03.808161 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:03.808071 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc" Oct 30 16:39:04.123570 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:39:04.123542 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbad445a_ffd5_427b_be3c_59225c098c21.slice/crio-781ba8a47c96af971b8a436393e5c0d229c6ad48b131372af0d4dcb801682552 WatchSource:0}: Error finding container 781ba8a47c96af971b8a436393e5c0d229c6ad48b131372af0d4dcb801682552: Status 404 returned error can't find the container with id 781ba8a47c96af971b8a436393e5c0d229c6ad48b131372af0d4dcb801682552 Oct 30 16:39:04.124323 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:04.124284 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc"] Oct 30 16:39:04.307572 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:04.307534 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc" event={"ID":"cbad445a-ffd5-427b-be3c-59225c098c21","Type":"ContainerStarted","Data":"781ba8a47c96af971b8a436393e5c0d229c6ad48b131372af0d4dcb801682552"} Oct 30 16:39:05.165882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:05.165846 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:05.315653 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:05.315567 6202 generic.go:356] "Generic (PLEG): container finished" podID="cbad445a-ffd5-427b-be3c-59225c098c21" containerID="d6d6e913625c75d5bdb2847a0d84125c01e421fb0f716e213378b7205fe95bd7" exitCode=0 Oct 30 16:39:05.315839 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:05.315651 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc" event={"ID":"cbad445a-ffd5-427b-be3c-59225c098c21","Type":"ContainerDied","Data":"d6d6e913625c75d5bdb2847a0d84125c01e421fb0f716e213378b7205fe95bd7"} Oct 30 16:39:05.520759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:05.520726 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:05.905297 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:05.905259 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:06.474709 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:06.474682 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc" Oct 30 16:39:06.511651 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:06.511624 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbad445a-ffd5-427b-be3c-59225c098c21-kube-api-access-75zmr\" (UniqueName: \"kubernetes.io/projected/cbad445a-ffd5-427b-be3c-59225c098c21-kube-api-access-75zmr\") pod \"cbad445a-ffd5-427b-be3c-59225c098c21\" (UID: \"cbad445a-ffd5-427b-be3c-59225c098c21\") " Oct 30 16:39:06.511799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:06.511682 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbad445a-ffd5-427b-be3c-59225c098c21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbad445a-ffd5-427b-be3c-59225c098c21-maas-api-openshift-ca\") pod \"cbad445a-ffd5-427b-be3c-59225c098c21\" (UID: \"cbad445a-ffd5-427b-be3c-59225c098c21\") " Oct 30 16:39:06.511799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:06.511789 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbad445a-ffd5-427b-be3c-59225c098c21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbad445a-ffd5-427b-be3c-59225c098c21-maas-api-xks-ca\") pod \"cbad445a-ffd5-427b-be3c-59225c098c21\" (UID: \"cbad445a-ffd5-427b-be3c-59225c098c21\") " Oct 30 16:39:06.512053 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:06.512030 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbad445a-ffd5-427b-be3c-59225c098c21-maas-api-openshift-ca" pod "cbad445a-ffd5-427b-be3c-59225c098c21" (UID: "cbad445a-ffd5-427b-be3c-59225c098c21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:06.512263 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:06.512244 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbad445a-ffd5-427b-be3c-59225c098c21-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:06.513834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:06.513810 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbad445a-ffd5-427b-be3c-59225c098c21-kube-api-access-75zmr" pod "cbad445a-ffd5-427b-be3c-59225c098c21" (UID: "cbad445a-ffd5-427b-be3c-59225c098c21"). InnerVolumeSpecName "kube-api-access-75zmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:06.513977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:06.513958 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbad445a-ffd5-427b-be3c-59225c098c21-maas-api-xks-ca" pod "cbad445a-ffd5-427b-be3c-59225c098c21" (UID: "cbad445a-ffd5-427b-be3c-59225c098c21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:06.613178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:06.613100 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbad445a-ffd5-427b-be3c-59225c098c21-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:06.613178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:06.613132 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75zmr\" (UniqueName: \"kubernetes.io/projected/cbad445a-ffd5-427b-be3c-59225c098c21-kube-api-access-75zmr\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:07.191103 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:07.191065 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:07.332563 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:07.332523 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc" Oct 30 16:39:07.332563 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:07.332535 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc" event={"ID":"cbad445a-ffd5-427b-be3c-59225c098c21","Type":"ContainerDied","Data":"781ba8a47c96af971b8a436393e5c0d229c6ad48b131372af0d4dcb801682552"} Oct 30 16:39:07.332791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:07.332630 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="781ba8a47c96af971b8a436393e5c0d229c6ad48b131372af0d4dcb801682552" Oct 30 16:39:08.785083 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:08.785043 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:10.420668 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:10.420637 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:12.069860 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:12.069827 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:13.592831 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:13.592061 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 16:39:13.654790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:13.654510 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:39:13.720081 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:13.718877 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 16:39:13.736650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:13.735887 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:13.810808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:13.807868 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:14.121996 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:14.121946 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:14.196943 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:14.196889 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:14.239285 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:39:14.239242 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:39:14.456850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:14.451889 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:14.456850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:14.451979 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" podUID="c3f813c4-823b-4109-a2e4-6027d5fabf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:14.616793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:14.592864 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:14.616793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:14.593686 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:39:14.649988 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:14.649945 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:14.909181 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:14.736856 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:14.980889 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:14.748910 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:14.992794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:14.750913 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 16:39:15.000798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:14.786424 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:15.001532 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:14.893352 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 16:39:15.001968 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:14.988089 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:39:15.002187 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:15.002167 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:39:15.002352 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:15.002336 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:39:15.002488 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:15.002477 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:39:15.094066 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:15.094028 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:15.129323 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:15.127963 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:15.129323 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:15.128384 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:39:15.563718 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:15.563675 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded" Oct 30 16:39:15.617734 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:15.604556 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:15.928443 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:15.927653 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:15.928443 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:15.928009 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:39:15.978791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:15.978677 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:39:15.985566 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:15.983926 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:39:15.985566 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:15.985066 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:16.004787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.002930 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:16.004787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.003939 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 16:39:16.058499 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.058468 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:39:16.074227 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.074056 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:39:16.077613 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.077587 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:39:16.083034 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.083007 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:39:16.098227 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.094425 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:39:16.099608 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.099578 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp"] Oct 30 16:39:16.100567 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.100543 6202 cpu_manager.go:414] "containerMap: removing container" podUID="cbad445a-ffd5-427b-be3c-59225c098c21" containerName="revoke-keys" Oct 30 16:39:16.100659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.100569 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbad445a-ffd5-427b-be3c-59225c098c21" containerName="revoke-keys" Oct 30 16:39:16.100809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.100792 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbad445a-ffd5-427b-be3c-59225c098c21" containerName="revoke-keys" Oct 30 16:39:16.106450 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.106421 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp"] Oct 30 16:39:16.106562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.106538 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp" Oct 30 16:39:16.122509 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.122480 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:39:16.122667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.122633 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:39:16.122867 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.122854 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:39:16.122938 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.122926 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:39:16.280614 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.280518 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d394a5a-344d-428e-b024-0597f6ce1bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e68bae0-gl6gp\" (UID: \"9d394a5a-344d-428e-b024-0597f6ce1bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp" Oct 30 16:39:16.280806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.280634 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d394a5a-344d-428e-b024-0597f6ce1bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e68bae0-gl6gp\" (UID: \"9d394a5a-344d-428e-b024-0597f6ce1bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp" Oct 30 16:39:16.280806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.280688 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65s7l\" (UniqueName: \"kubernetes.io/projected/9d394a5a-344d-428e-b024-0597f6ce1bcb-kube-api-access-65s7l\") pod \"maas-api-revoke-sub-4e68bae0-gl6gp\" (UID: \"9d394a5a-344d-428e-b024-0597f6ce1bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp" Oct 30 16:39:16.382386 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.382344 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d394a5a-344d-428e-b024-0597f6ce1bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e68bae0-gl6gp\" (UID: \"9d394a5a-344d-428e-b024-0597f6ce1bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp" Oct 30 16:39:16.382386 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.382406 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d394a5a-344d-428e-b024-0597f6ce1bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e68bae0-gl6gp\" (UID: \"9d394a5a-344d-428e-b024-0597f6ce1bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp" Oct 30 16:39:16.382663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.382452 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65s7l\" (UniqueName: \"kubernetes.io/projected/9d394a5a-344d-428e-b024-0597f6ce1bcb-kube-api-access-65s7l\") pod \"maas-api-revoke-sub-4e68bae0-gl6gp\" (UID: \"9d394a5a-344d-428e-b024-0597f6ce1bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp" Oct 30 16:39:16.383198 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.383168 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d394a5a-344d-428e-b024-0597f6ce1bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e68bae0-gl6gp\" (UID: \"9d394a5a-344d-428e-b024-0597f6ce1bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp" Oct 30 16:39:16.385272 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.385247 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d394a5a-344d-428e-b024-0597f6ce1bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e68bae0-gl6gp\" (UID: \"9d394a5a-344d-428e-b024-0597f6ce1bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp" Oct 30 16:39:16.408274 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.408244 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65s7l\" (UniqueName: \"kubernetes.io/projected/9d394a5a-344d-428e-b024-0597f6ce1bcb-kube-api-access-65s7l\") pod \"maas-api-revoke-sub-4e68bae0-gl6gp\" (UID: \"9d394a5a-344d-428e-b024-0597f6ce1bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp" Oct 30 16:39:16.426998 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.426969 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp" Oct 30 16:39:16.614742 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.614704 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:16.745913 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:39:16.745882 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d394a5a_344d_428e_b024_0597f6ce1bcb.slice/crio-3505475ef68470db2874a9c79174425e8e233857fe74823c63d68021f06158a4 WatchSource:0}: Error finding container 3505475ef68470db2874a9c79174425e8e233857fe74823c63d68021f06158a4: Status 404 returned error can't find the container with id 3505475ef68470db2874a9c79174425e8e233857fe74823c63d68021f06158a4 Oct 30 16:39:16.746268 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.746101 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp"] Oct 30 16:39:16.863359 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:16.863317 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp" event={"ID":"9d394a5a-344d-428e-b024-0597f6ce1bcb","Type":"ContainerStarted","Data":"3505475ef68470db2874a9c79174425e8e233857fe74823c63d68021f06158a4"} Oct 30 16:39:17.874057 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:17.874006 6202 generic.go:356] "Generic (PLEG): container finished" podID="9d394a5a-344d-428e-b024-0597f6ce1bcb" containerID="1a8801ba81a8d89d034a606ec5162b92e61ce23d3b11076c93bb589f85721c18" exitCode=0 Oct 30 16:39:17.874502 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:17.874095 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp" event={"ID":"9d394a5a-344d-428e-b024-0597f6ce1bcb","Type":"ContainerDied","Data":"1a8801ba81a8d89d034a606ec5162b92e61ce23d3b11076c93bb589f85721c18"} Oct 30 16:39:18.261958 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:18.261908 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:19.048304 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:19.048278 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp" Oct 30 16:39:19.116387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:19.116331 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d394a5a-344d-428e-b024-0597f6ce1bcb-kube-api-access-65s7l\" (UniqueName: \"kubernetes.io/projected/9d394a5a-344d-428e-b024-0597f6ce1bcb-kube-api-access-65s7l\") pod \"9d394a5a-344d-428e-b024-0597f6ce1bcb\" (UID: \"9d394a5a-344d-428e-b024-0597f6ce1bcb\") " Oct 30 16:39:19.116562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:19.116425 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d394a5a-344d-428e-b024-0597f6ce1bcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d394a5a-344d-428e-b024-0597f6ce1bcb-maas-api-openshift-ca\") pod \"9d394a5a-344d-428e-b024-0597f6ce1bcb\" (UID: \"9d394a5a-344d-428e-b024-0597f6ce1bcb\") " Oct 30 16:39:19.116651 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:19.116630 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d394a5a-344d-428e-b024-0597f6ce1bcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d394a5a-344d-428e-b024-0597f6ce1bcb-maas-api-xks-ca\") pod \"9d394a5a-344d-428e-b024-0597f6ce1bcb\" (UID: \"9d394a5a-344d-428e-b024-0597f6ce1bcb\") " Oct 30 16:39:19.117013 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:19.116974 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d394a5a-344d-428e-b024-0597f6ce1bcb-maas-api-openshift-ca" pod "9d394a5a-344d-428e-b024-0597f6ce1bcb" (UID: "9d394a5a-344d-428e-b024-0597f6ce1bcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:19.117988 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:19.117963 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d394a5a-344d-428e-b024-0597f6ce1bcb-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:19.119155 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:19.119124 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d394a5a-344d-428e-b024-0597f6ce1bcb-maas-api-xks-ca" pod "9d394a5a-344d-428e-b024-0597f6ce1bcb" (UID: "9d394a5a-344d-428e-b024-0597f6ce1bcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:19.119704 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:19.119680 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d394a5a-344d-428e-b024-0597f6ce1bcb-kube-api-access-65s7l" pod "9d394a5a-344d-428e-b024-0597f6ce1bcb" (UID: "9d394a5a-344d-428e-b024-0597f6ce1bcb"). InnerVolumeSpecName "kube-api-access-65s7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:19.219508 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:19.219416 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65s7l\" (UniqueName: \"kubernetes.io/projected/9d394a5a-344d-428e-b024-0597f6ce1bcb-kube-api-access-65s7l\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:19.219508 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:19.219450 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d394a5a-344d-428e-b024-0597f6ce1bcb-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:19.771388 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:19.771348 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:19.896350 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:19.895761 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp" Oct 30 16:39:19.896350 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:19.895863 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp" event={"ID":"9d394a5a-344d-428e-b024-0597f6ce1bcb","Type":"ContainerDied","Data":"3505475ef68470db2874a9c79174425e8e233857fe74823c63d68021f06158a4"} Oct 30 16:39:19.896350 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:19.895914 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3505475ef68470db2874a9c79174425e8e233857fe74823c63d68021f06158a4" Oct 30 16:39:21.260243 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:21.260204 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:23.015620 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:23.015585 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:24.565158 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:24.565120 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:25.796118 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:25.796084 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6"] Oct 30 16:39:25.796933 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:25.796912 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9d394a5a-344d-428e-b024-0597f6ce1bcb" containerName="revoke-keys" Oct 30 16:39:25.797044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:25.796933 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d394a5a-344d-428e-b024-0597f6ce1bcb" containerName="revoke-keys" Oct 30 16:39:25.797122 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:25.797108 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d394a5a-344d-428e-b024-0597f6ce1bcb" containerName="revoke-keys" Oct 30 16:39:25.801040 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:25.801018 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6" Oct 30 16:39:25.820504 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:25.820479 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6"] Oct 30 16:39:25.827940 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:25.827884 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:39:25.827940 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:25.827897 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:39:25.828146 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:25.827954 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:39:25.828277 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:25.828251 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:39:25.908813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:25.908753 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdkfl\" (UniqueName: \"kubernetes.io/projected/5114a2a6-9142-4b3e-a820-fb75b364e628-kube-api-access-sdkfl\") pod \"maas-api-revoke-sub-d67e98bf-cq4v6\" (UID: \"5114a2a6-9142-4b3e-a820-fb75b364e628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6" Oct 30 16:39:25.909048 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:25.909020 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5114a2a6-9142-4b3e-a820-fb75b364e628-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d67e98bf-cq4v6\" (UID: \"5114a2a6-9142-4b3e-a820-fb75b364e628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6" Oct 30 16:39:25.909172 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:25.909060 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5114a2a6-9142-4b3e-a820-fb75b364e628-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d67e98bf-cq4v6\" (UID: \"5114a2a6-9142-4b3e-a820-fb75b364e628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6" Oct 30 16:39:26.010324 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.010297 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5114a2a6-9142-4b3e-a820-fb75b364e628-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d67e98bf-cq4v6\" (UID: \"5114a2a6-9142-4b3e-a820-fb75b364e628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6" Oct 30 16:39:26.010324 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.010327 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5114a2a6-9142-4b3e-a820-fb75b364e628-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d67e98bf-cq4v6\" (UID: \"5114a2a6-9142-4b3e-a820-fb75b364e628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6" Oct 30 16:39:26.010537 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.010404 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdkfl\" (UniqueName: \"kubernetes.io/projected/5114a2a6-9142-4b3e-a820-fb75b364e628-kube-api-access-sdkfl\") pod \"maas-api-revoke-sub-d67e98bf-cq4v6\" (UID: \"5114a2a6-9142-4b3e-a820-fb75b364e628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6" Oct 30 16:39:26.010971 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.010949 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5114a2a6-9142-4b3e-a820-fb75b364e628-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d67e98bf-cq4v6\" (UID: \"5114a2a6-9142-4b3e-a820-fb75b364e628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6" Oct 30 16:39:26.013102 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.013079 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5114a2a6-9142-4b3e-a820-fb75b364e628-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d67e98bf-cq4v6\" (UID: \"5114a2a6-9142-4b3e-a820-fb75b364e628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6" Oct 30 16:39:26.034578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.034556 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdkfl\" (UniqueName: \"kubernetes.io/projected/5114a2a6-9142-4b3e-a820-fb75b364e628-kube-api-access-sdkfl\") pod \"maas-api-revoke-sub-d67e98bf-cq4v6\" (UID: \"5114a2a6-9142-4b3e-a820-fb75b364e628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6" Oct 30 16:39:26.115390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.115361 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6" Oct 30 16:39:26.213161 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.213133 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q"] Oct 30 16:39:26.218614 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.218595 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q" Oct 30 16:39:26.255873 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.255835 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q"] Oct 30 16:39:26.314254 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.314216 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0cb704d-rsp2q\" (UID: \"aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q" Oct 30 16:39:26.314426 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.314351 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0cb704d-rsp2q\" (UID: \"aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q" Oct 30 16:39:26.314518 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.314502 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjbp4\" (UniqueName: \"kubernetes.io/projected/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-kube-api-access-xjbp4\") pod \"maas-api-revoke-sub-f0cb704d-rsp2q\" (UID: \"aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q" Oct 30 16:39:26.419410 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.419318 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:26.419864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.419829 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjbp4\" (UniqueName: \"kubernetes.io/projected/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-kube-api-access-xjbp4\") pod \"maas-api-revoke-sub-f0cb704d-rsp2q\" (UID: \"aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q" Oct 30 16:39:26.420025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.419970 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0cb704d-rsp2q\" (UID: \"aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q" Oct 30 16:39:26.420083 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.420066 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0cb704d-rsp2q\" (UID: \"aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q" Oct 30 16:39:26.423475 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.423448 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0cb704d-rsp2q\" (UID: \"aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q" Oct 30 16:39:26.425121 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.425096 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0cb704d-rsp2q\" (UID: \"aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q" Oct 30 16:39:26.426752 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:39:26.426732 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5114a2a6_9142_4b3e_a820_fb75b364e628.slice/crio-a53adafbb9425e59616e1470e61685a57c7ad452d7b6b38843348f4b710c0827 WatchSource:0}: Error finding container a53adafbb9425e59616e1470e61685a57c7ad452d7b6b38843348f4b710c0827: Status 404 returned error can't find the container with id a53adafbb9425e59616e1470e61685a57c7ad452d7b6b38843348f4b710c0827 Oct 30 16:39:26.428404 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.428382 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6"] Oct 30 16:39:26.450387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.446353 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjbp4\" (UniqueName: \"kubernetes.io/projected/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-kube-api-access-xjbp4\") pod \"maas-api-revoke-sub-f0cb704d-rsp2q\" (UID: \"aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q" Oct 30 16:39:26.535025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.534998 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q" Oct 30 16:39:26.839115 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:39:26.838121 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac7cc92_21b3_46c1_bbf6_6f4fa4c6c7f5.slice/crio-1f53a284b08edf344605c5816f87c96665f3f8b79f091990aaed1e63efeaeacd WatchSource:0}: Error finding container 1f53a284b08edf344605c5816f87c96665f3f8b79f091990aaed1e63efeaeacd: Status 404 returned error can't find the container with id 1f53a284b08edf344605c5816f87c96665f3f8b79f091990aaed1e63efeaeacd Oct 30 16:39:26.839651 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.839423 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q"] Oct 30 16:39:26.952307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.952275 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q" event={"ID":"aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5","Type":"ContainerStarted","Data":"1f53a284b08edf344605c5816f87c96665f3f8b79f091990aaed1e63efeaeacd"} Oct 30 16:39:26.954309 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.954277 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6" event={"ID":"5114a2a6-9142-4b3e-a820-fb75b364e628","Type":"ContainerStarted","Data":"64e7e2866f0c54f0995b52354c849bf393cecad225854b2b3e0f3f42c79cd38f"} Oct 30 16:39:26.954464 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:26.954333 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6" event={"ID":"5114a2a6-9142-4b3e-a820-fb75b364e628","Type":"ContainerStarted","Data":"a53adafbb9425e59616e1470e61685a57c7ad452d7b6b38843348f4b710c0827"} Oct 30 16:39:27.006733 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:27.006637 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6" podStartSLOduration=2.006619156 podStartE2EDuration="2.006619156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:27.003372889 +0000 UTC m=+1080.713827632" watchObservedRunningTime="2026-10-30 16:39:27.006619156 +0000 UTC m=+1080.717073903" Oct 30 16:39:27.964557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:27.964519 6202 generic.go:356] "Generic (PLEG): container finished" podID="aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5" containerID="fa45eb852be00c4aa611a8694f030238d953c424c49a21e386bb0c502c0b11f5" exitCode=0 Oct 30 16:39:27.965056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:27.964610 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q" event={"ID":"aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5","Type":"ContainerDied","Data":"fa45eb852be00c4aa611a8694f030238d953c424c49a21e386bb0c502c0b11f5"} Oct 30 16:39:27.966492 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:27.966459 6202 generic.go:356] "Generic (PLEG): container finished" podID="5114a2a6-9142-4b3e-a820-fb75b364e628" containerID="64e7e2866f0c54f0995b52354c849bf393cecad225854b2b3e0f3f42c79cd38f" exitCode=0 Oct 30 16:39:27.966615 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:27.966506 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6" event={"ID":"5114a2a6-9142-4b3e-a820-fb75b364e628","Type":"ContainerDied","Data":"64e7e2866f0c54f0995b52354c849bf393cecad225854b2b3e0f3f42c79cd38f"} Oct 30 16:39:28.059488 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:28.059447 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:29.191735 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.191705 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q" Oct 30 16:39:29.195568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.195548 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6" Oct 30 16:39:29.265420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.265337 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-kube-api-access-xjbp4\" (UniqueName: \"kubernetes.io/projected/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-kube-api-access-xjbp4\") pod \"aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5\" (UID: \"aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5\") " Oct 30 16:39:29.265582 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.265414 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5114a2a6-9142-4b3e-a820-fb75b364e628-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5114a2a6-9142-4b3e-a820-fb75b364e628-maas-api-openshift-ca\") pod \"5114a2a6-9142-4b3e-a820-fb75b364e628\" (UID: \"5114a2a6-9142-4b3e-a820-fb75b364e628\") " Oct 30 16:39:29.265582 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.265524 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5114a2a6-9142-4b3e-a820-fb75b364e628-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5114a2a6-9142-4b3e-a820-fb75b364e628-maas-api-xks-ca\") pod \"5114a2a6-9142-4b3e-a820-fb75b364e628\" (UID: \"5114a2a6-9142-4b3e-a820-fb75b364e628\") " Oct 30 16:39:29.265707 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.265624 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-maas-api-xks-ca\") pod \"aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5\" (UID: \"aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5\") " Oct 30 16:39:29.265783 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.265725 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-maas-api-openshift-ca\") pod \"aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5\" (UID: \"aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5\") " Oct 30 16:39:29.265847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.265810 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5114a2a6-9142-4b3e-a820-fb75b364e628-kube-api-access-sdkfl\" (UniqueName: \"kubernetes.io/projected/5114a2a6-9142-4b3e-a820-fb75b364e628-kube-api-access-sdkfl\") pod \"5114a2a6-9142-4b3e-a820-fb75b364e628\" (UID: \"5114a2a6-9142-4b3e-a820-fb75b364e628\") " Oct 30 16:39:29.268131 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.268099 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-maas-api-openshift-ca" pod "aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5" (UID: "aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:29.268260 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.268130 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5114a2a6-9142-4b3e-a820-fb75b364e628-maas-api-openshift-ca" pod "5114a2a6-9142-4b3e-a820-fb75b364e628" (UID: "5114a2a6-9142-4b3e-a820-fb75b364e628"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:29.269075 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.269036 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5114a2a6-9142-4b3e-a820-fb75b364e628-maas-api-xks-ca" pod "5114a2a6-9142-4b3e-a820-fb75b364e628" (UID: "5114a2a6-9142-4b3e-a820-fb75b364e628"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:29.269174 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.269149 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5114a2a6-9142-4b3e-a820-fb75b364e628-kube-api-access-sdkfl" pod "5114a2a6-9142-4b3e-a820-fb75b364e628" (UID: "5114a2a6-9142-4b3e-a820-fb75b364e628"). InnerVolumeSpecName "kube-api-access-sdkfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:29.271267 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.271229 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-maas-api-xks-ca" pod "aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5" (UID: "aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:29.278227 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.276924 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-kube-api-access-xjbp4" pod "aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5" (UID: "aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5"). InnerVolumeSpecName "kube-api-access-xjbp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:29.367471 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.367438 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdkfl\" (UniqueName: \"kubernetes.io/projected/5114a2a6-9142-4b3e-a820-fb75b364e628-kube-api-access-sdkfl\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.367471 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.367465 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjbp4\" (UniqueName: \"kubernetes.io/projected/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-kube-api-access-xjbp4\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.367471 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.367475 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5114a2a6-9142-4b3e-a820-fb75b364e628-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.367696 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.367484 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5114a2a6-9142-4b3e-a820-fb75b364e628-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.367696 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.367492 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.367696 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.367500 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.599316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.599279 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:29.984587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.984399 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q" Oct 30 16:39:29.984587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.984435 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q" event={"ID":"aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5","Type":"ContainerDied","Data":"1f53a284b08edf344605c5816f87c96665f3f8b79f091990aaed1e63efeaeacd"} Oct 30 16:39:29.984587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.984494 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f53a284b08edf344605c5816f87c96665f3f8b79f091990aaed1e63efeaeacd" Oct 30 16:39:29.986369 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.986247 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6" Oct 30 16:39:29.986369 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.986294 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6" event={"ID":"5114a2a6-9142-4b3e-a820-fb75b364e628","Type":"ContainerDied","Data":"a53adafbb9425e59616e1470e61685a57c7ad452d7b6b38843348f4b710c0827"} Oct 30 16:39:29.986369 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:29.986347 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a53adafbb9425e59616e1470e61685a57c7ad452d7b6b38843348f4b710c0827" Oct 30 16:39:31.569953 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.569923 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67"] Oct 30 16:39:31.570643 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.570626 6202 cpu_manager.go:414] "containerMap: removing container" podUID="aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5" containerName="revoke-keys" Oct 30 16:39:31.570643 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.570645 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5" containerName="revoke-keys" Oct 30 16:39:31.570818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.570667 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5114a2a6-9142-4b3e-a820-fb75b364e628" containerName="revoke-keys" Oct 30 16:39:31.570818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.570674 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5114a2a6-9142-4b3e-a820-fb75b364e628" containerName="revoke-keys" Oct 30 16:39:31.570818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.570795 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5" containerName="revoke-keys" Oct 30 16:39:31.570818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.570809 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5114a2a6-9142-4b3e-a820-fb75b364e628" containerName="revoke-keys" Oct 30 16:39:31.575028 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.575008 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67" Oct 30 16:39:31.594916 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.594892 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67"] Oct 30 16:39:31.607799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.601950 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:39:31.607799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.601973 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:39:31.607799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.601949 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:39:31.607799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.602083 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:39:31.627662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.627636 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a72768-8289-4a5d-8e6d-599ccfec9502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03450b79-xtp67\" (UID: \"72a72768-8289-4a5d-8e6d-599ccfec9502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67" Oct 30 16:39:31.627859 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.627697 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a72768-8289-4a5d-8e6d-599ccfec9502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03450b79-xtp67\" (UID: \"72a72768-8289-4a5d-8e6d-599ccfec9502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67" Oct 30 16:39:31.627986 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.627958 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2dbs\" (UniqueName: \"kubernetes.io/projected/72a72768-8289-4a5d-8e6d-599ccfec9502-kube-api-access-h2dbs\") pod \"maas-api-revoke-sub-03450b79-xtp67\" (UID: \"72a72768-8289-4a5d-8e6d-599ccfec9502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67" Oct 30 16:39:31.728749 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.728713 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a72768-8289-4a5d-8e6d-599ccfec9502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03450b79-xtp67\" (UID: \"72a72768-8289-4a5d-8e6d-599ccfec9502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67" Oct 30 16:39:31.728953 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.728832 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a72768-8289-4a5d-8e6d-599ccfec9502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03450b79-xtp67\" (UID: \"72a72768-8289-4a5d-8e6d-599ccfec9502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67" Oct 30 16:39:31.729014 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.728965 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dbs\" (UniqueName: \"kubernetes.io/projected/72a72768-8289-4a5d-8e6d-599ccfec9502-kube-api-access-h2dbs\") pod \"maas-api-revoke-sub-03450b79-xtp67\" (UID: \"72a72768-8289-4a5d-8e6d-599ccfec9502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67" Oct 30 16:39:31.729574 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.729556 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a72768-8289-4a5d-8e6d-599ccfec9502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03450b79-xtp67\" (UID: \"72a72768-8289-4a5d-8e6d-599ccfec9502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67" Oct 30 16:39:31.731315 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.731295 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a72768-8289-4a5d-8e6d-599ccfec9502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03450b79-xtp67\" (UID: \"72a72768-8289-4a5d-8e6d-599ccfec9502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67" Oct 30 16:39:31.756499 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.756474 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2dbs\" (UniqueName: \"kubernetes.io/projected/72a72768-8289-4a5d-8e6d-599ccfec9502-kube-api-access-h2dbs\") pod \"maas-api-revoke-sub-03450b79-xtp67\" (UID: \"72a72768-8289-4a5d-8e6d-599ccfec9502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67" Oct 30 16:39:31.823381 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.823353 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:31.892198 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:31.891949 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67" Oct 30 16:39:32.211873 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:39:32.211837 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a72768_8289_4a5d_8e6d_599ccfec9502.slice/crio-473b5fe27181dbedcd54224ab0f5e8cb8ae932162291a96b55d008b63f001b5f WatchSource:0}: Error finding container 473b5fe27181dbedcd54224ab0f5e8cb8ae932162291a96b55d008b63f001b5f: Status 404 returned error can't find the container with id 473b5fe27181dbedcd54224ab0f5e8cb8ae932162291a96b55d008b63f001b5f Oct 30 16:39:32.213033 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:32.213007 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67"] Oct 30 16:39:32.440162 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:32.440129 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv"] Oct 30 16:39:32.446722 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:32.446699 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv" Oct 30 16:39:32.464813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:32.464687 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv"] Oct 30 16:39:32.539620 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:32.539595 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7280c2-6b04-4d04-9edc-81323992d117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e04a6d5-f9qrv\" (UID: \"2e7280c2-6b04-4d04-9edc-81323992d117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv" Oct 30 16:39:32.539889 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:32.539868 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blqgp\" (UniqueName: \"kubernetes.io/projected/2e7280c2-6b04-4d04-9edc-81323992d117-kube-api-access-blqgp\") pod \"maas-api-revoke-sub-7e04a6d5-f9qrv\" (UID: \"2e7280c2-6b04-4d04-9edc-81323992d117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv" Oct 30 16:39:32.539947 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:32.539916 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7280c2-6b04-4d04-9edc-81323992d117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e04a6d5-f9qrv\" (UID: \"2e7280c2-6b04-4d04-9edc-81323992d117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv" Oct 30 16:39:32.641077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:32.641043 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blqgp\" (UniqueName: \"kubernetes.io/projected/2e7280c2-6b04-4d04-9edc-81323992d117-kube-api-access-blqgp\") pod \"maas-api-revoke-sub-7e04a6d5-f9qrv\" (UID: \"2e7280c2-6b04-4d04-9edc-81323992d117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv" Oct 30 16:39:32.641545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:32.641098 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7280c2-6b04-4d04-9edc-81323992d117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e04a6d5-f9qrv\" (UID: \"2e7280c2-6b04-4d04-9edc-81323992d117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv" Oct 30 16:39:32.641545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:32.641151 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7280c2-6b04-4d04-9edc-81323992d117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e04a6d5-f9qrv\" (UID: \"2e7280c2-6b04-4d04-9edc-81323992d117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv" Oct 30 16:39:32.641802 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:32.641781 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7280c2-6b04-4d04-9edc-81323992d117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e04a6d5-f9qrv\" (UID: \"2e7280c2-6b04-4d04-9edc-81323992d117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv" Oct 30 16:39:32.643882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:32.643852 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7280c2-6b04-4d04-9edc-81323992d117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e04a6d5-f9qrv\" (UID: \"2e7280c2-6b04-4d04-9edc-81323992d117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv" Oct 30 16:39:32.673095 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:32.673070 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blqgp\" (UniqueName: \"kubernetes.io/projected/2e7280c2-6b04-4d04-9edc-81323992d117-kube-api-access-blqgp\") pod \"maas-api-revoke-sub-7e04a6d5-f9qrv\" (UID: \"2e7280c2-6b04-4d04-9edc-81323992d117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv" Oct 30 16:39:32.765876 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:32.765794 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv" Oct 30 16:39:33.047283 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:33.047189 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67" event={"ID":"72a72768-8289-4a5d-8e6d-599ccfec9502","Type":"ContainerStarted","Data":"fa16b029fb579b7063124cbe73f4b9579e84dd2fda141d5827996a930d857576"} Oct 30 16:39:33.047283 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:33.047254 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67" event={"ID":"72a72768-8289-4a5d-8e6d-599ccfec9502","Type":"ContainerStarted","Data":"473b5fe27181dbedcd54224ab0f5e8cb8ae932162291a96b55d008b63f001b5f"} Oct 30 16:39:33.073181 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:33.072677 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv"] Oct 30 16:39:33.073181 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:33.073116 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:33.073516 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:39:33.073489 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7280c2_6b04_4d04_9edc_81323992d117.slice/crio-6c7d76e9756729e2affb24b061a343c9e3d0e715c8bb961af6873ffe920ff739 WatchSource:0}: Error finding container 6c7d76e9756729e2affb24b061a343c9e3d0e715c8bb961af6873ffe920ff739: Status 404 returned error can't find the container with id 6c7d76e9756729e2affb24b061a343c9e3d0e715c8bb961af6873ffe920ff739 Oct 30 16:39:33.095088 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:33.095043 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67" podStartSLOduration=2.095028465 podStartE2EDuration="2.095028465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:33.091261642 +0000 UTC m=+1086.801716389" watchObservedRunningTime="2026-10-30 16:39:33.095028465 +0000 UTC m=+1086.805483214" Oct 30 16:39:33.215187 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:33.215160 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:33.590277 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:33.590242 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:34.057805 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:34.057711 6202 generic.go:356] "Generic (PLEG): container finished" podID="72a72768-8289-4a5d-8e6d-599ccfec9502" containerID="fa16b029fb579b7063124cbe73f4b9579e84dd2fda141d5827996a930d857576" exitCode=0 Oct 30 16:39:34.058235 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:34.057812 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67" event={"ID":"72a72768-8289-4a5d-8e6d-599ccfec9502","Type":"ContainerDied","Data":"fa16b029fb579b7063124cbe73f4b9579e84dd2fda141d5827996a930d857576"} Oct 30 16:39:34.059853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:34.059825 6202 generic.go:356] "Generic (PLEG): container finished" podID="2e7280c2-6b04-4d04-9edc-81323992d117" containerID="3bf83cc0167fe2038e47815d8a11d10bb49f470ad791811695ec888c7efc596a" exitCode=0 Oct 30 16:39:34.060000 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:34.059912 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv" event={"ID":"2e7280c2-6b04-4d04-9edc-81323992d117","Type":"ContainerDied","Data":"3bf83cc0167fe2038e47815d8a11d10bb49f470ad791811695ec888c7efc596a"} Oct 30 16:39:34.060000 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:34.059978 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv" event={"ID":"2e7280c2-6b04-4d04-9edc-81323992d117","Type":"ContainerStarted","Data":"6c7d76e9756729e2affb24b061a343c9e3d0e715c8bb961af6873ffe920ff739"} Oct 30 16:39:35.201458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.201417 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:35.256691 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.256662 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv" Oct 30 16:39:35.298692 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.298661 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67" Oct 30 16:39:35.405311 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.405224 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e7280c2-6b04-4d04-9edc-81323992d117-kube-api-access-blqgp\" (UniqueName: \"kubernetes.io/projected/2e7280c2-6b04-4d04-9edc-81323992d117-kube-api-access-blqgp\") pod \"2e7280c2-6b04-4d04-9edc-81323992d117\" (UID: \"2e7280c2-6b04-4d04-9edc-81323992d117\") " Oct 30 16:39:35.405476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.405382 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a72768-8289-4a5d-8e6d-599ccfec9502-kube-api-access-h2dbs\" (UniqueName: \"kubernetes.io/projected/72a72768-8289-4a5d-8e6d-599ccfec9502-kube-api-access-h2dbs\") pod \"72a72768-8289-4a5d-8e6d-599ccfec9502\" (UID: \"72a72768-8289-4a5d-8e6d-599ccfec9502\") " Oct 30 16:39:35.405476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.405449 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a72768-8289-4a5d-8e6d-599ccfec9502-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a72768-8289-4a5d-8e6d-599ccfec9502-maas-api-openshift-ca\") pod \"72a72768-8289-4a5d-8e6d-599ccfec9502\" (UID: \"72a72768-8289-4a5d-8e6d-599ccfec9502\") " Oct 30 16:39:35.405604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.405519 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72a72768-8289-4a5d-8e6d-599ccfec9502-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a72768-8289-4a5d-8e6d-599ccfec9502-maas-api-xks-ca\") pod \"72a72768-8289-4a5d-8e6d-599ccfec9502\" (UID: \"72a72768-8289-4a5d-8e6d-599ccfec9502\") " Oct 30 16:39:35.405604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.405562 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e7280c2-6b04-4d04-9edc-81323992d117-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7280c2-6b04-4d04-9edc-81323992d117-maas-api-openshift-ca\") pod \"2e7280c2-6b04-4d04-9edc-81323992d117\" (UID: \"2e7280c2-6b04-4d04-9edc-81323992d117\") " Oct 30 16:39:35.405706 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.405695 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e7280c2-6b04-4d04-9edc-81323992d117-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7280c2-6b04-4d04-9edc-81323992d117-maas-api-xks-ca\") pod \"2e7280c2-6b04-4d04-9edc-81323992d117\" (UID: \"2e7280c2-6b04-4d04-9edc-81323992d117\") " Oct 30 16:39:35.405947 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.405909 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a72768-8289-4a5d-8e6d-599ccfec9502-maas-api-openshift-ca" pod "72a72768-8289-4a5d-8e6d-599ccfec9502" (UID: "72a72768-8289-4a5d-8e6d-599ccfec9502"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:35.406087 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.405988 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7280c2-6b04-4d04-9edc-81323992d117-maas-api-openshift-ca" pod "2e7280c2-6b04-4d04-9edc-81323992d117" (UID: "2e7280c2-6b04-4d04-9edc-81323992d117"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:35.406478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.406451 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a72768-8289-4a5d-8e6d-599ccfec9502-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:35.406755 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.406484 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7280c2-6b04-4d04-9edc-81323992d117-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:35.407637 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.407615 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7280c2-6b04-4d04-9edc-81323992d117-kube-api-access-blqgp" pod "2e7280c2-6b04-4d04-9edc-81323992d117" (UID: "2e7280c2-6b04-4d04-9edc-81323992d117"). InnerVolumeSpecName "kube-api-access-blqgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:35.407755 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.407715 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a72768-8289-4a5d-8e6d-599ccfec9502-maas-api-xks-ca" pod "72a72768-8289-4a5d-8e6d-599ccfec9502" (UID: "72a72768-8289-4a5d-8e6d-599ccfec9502"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:35.408062 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.408038 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a72768-8289-4a5d-8e6d-599ccfec9502-kube-api-access-h2dbs" pod "72a72768-8289-4a5d-8e6d-599ccfec9502" (UID: "72a72768-8289-4a5d-8e6d-599ccfec9502"). InnerVolumeSpecName "kube-api-access-h2dbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:35.408171 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.408149 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7280c2-6b04-4d04-9edc-81323992d117-maas-api-xks-ca" pod "2e7280c2-6b04-4d04-9edc-81323992d117" (UID: "2e7280c2-6b04-4d04-9edc-81323992d117"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:35.507587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.507553 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a72768-8289-4a5d-8e6d-599ccfec9502-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:35.507587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.507586 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7280c2-6b04-4d04-9edc-81323992d117-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:35.507846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.507600 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blqgp\" (UniqueName: \"kubernetes.io/projected/2e7280c2-6b04-4d04-9edc-81323992d117-kube-api-access-blqgp\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:35.507846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:35.507615 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2dbs\" (UniqueName: \"kubernetes.io/projected/72a72768-8289-4a5d-8e6d-599ccfec9502-kube-api-access-h2dbs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:36.077213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.077183 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67" Oct 30 16:39:36.077213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.077204 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67" event={"ID":"72a72768-8289-4a5d-8e6d-599ccfec9502","Type":"ContainerDied","Data":"473b5fe27181dbedcd54224ab0f5e8cb8ae932162291a96b55d008b63f001b5f"} Oct 30 16:39:36.077442 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.077263 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="473b5fe27181dbedcd54224ab0f5e8cb8ae932162291a96b55d008b63f001b5f" Oct 30 16:39:36.078872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.078844 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv" Oct 30 16:39:36.079007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.078877 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv" event={"ID":"2e7280c2-6b04-4d04-9edc-81323992d117","Type":"ContainerDied","Data":"6c7d76e9756729e2affb24b061a343c9e3d0e715c8bb961af6873ffe920ff739"} Oct 30 16:39:36.079007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.078922 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c7d76e9756729e2affb24b061a343c9e3d0e715c8bb961af6873ffe920ff739" Oct 30 16:39:36.141078 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.141035 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs"] Oct 30 16:39:36.142242 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.142200 6202 cpu_manager.go:414] "containerMap: removing container" podUID="72a72768-8289-4a5d-8e6d-599ccfec9502" containerName="revoke-keys" Oct 30 16:39:36.142242 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.142243 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a72768-8289-4a5d-8e6d-599ccfec9502" containerName="revoke-keys" Oct 30 16:39:36.142242 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.142258 6202 cpu_manager.go:414] "containerMap: removing container" podUID="2e7280c2-6b04-4d04-9edc-81323992d117" containerName="revoke-keys" Oct 30 16:39:36.142242 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.142266 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e7280c2-6b04-4d04-9edc-81323992d117" containerName="revoke-keys" Oct 30 16:39:36.142872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.142483 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a72768-8289-4a5d-8e6d-599ccfec9502" containerName="revoke-keys" Oct 30 16:39:36.142872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.142501 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e7280c2-6b04-4d04-9edc-81323992d117" containerName="revoke-keys" Oct 30 16:39:36.147663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.147615 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs" Oct 30 16:39:36.163133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.163105 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:39:36.163398 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.163376 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:39:36.163398 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.163425 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:39:36.163398 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.163432 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:39:36.168519 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.168494 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs"] Oct 30 16:39:36.318836 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.318797 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b2b662-ec40-4014-879a-c60a860da839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bacd9d6-rrfxs\" (UID: \"f6b2b662-ec40-4014-879a-c60a860da839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs" Oct 30 16:39:36.319240 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.318888 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b2b662-ec40-4014-879a-c60a860da839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bacd9d6-rrfxs\" (UID: \"f6b2b662-ec40-4014-879a-c60a860da839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs" Oct 30 16:39:36.319240 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.319040 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m468b\" (UniqueName: \"kubernetes.io/projected/f6b2b662-ec40-4014-879a-c60a860da839-kube-api-access-m468b\") pod \"maas-api-revoke-sub-9bacd9d6-rrfxs\" (UID: \"f6b2b662-ec40-4014-879a-c60a860da839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs" Oct 30 16:39:36.420600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.420512 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m468b\" (UniqueName: \"kubernetes.io/projected/f6b2b662-ec40-4014-879a-c60a860da839-kube-api-access-m468b\") pod \"maas-api-revoke-sub-9bacd9d6-rrfxs\" (UID: \"f6b2b662-ec40-4014-879a-c60a860da839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs" Oct 30 16:39:36.420760 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.420676 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b2b662-ec40-4014-879a-c60a860da839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bacd9d6-rrfxs\" (UID: \"f6b2b662-ec40-4014-879a-c60a860da839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs" Oct 30 16:39:36.420760 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.420721 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b2b662-ec40-4014-879a-c60a860da839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bacd9d6-rrfxs\" (UID: \"f6b2b662-ec40-4014-879a-c60a860da839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs" Oct 30 16:39:36.421294 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.421275 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b2b662-ec40-4014-879a-c60a860da839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bacd9d6-rrfxs\" (UID: \"f6b2b662-ec40-4014-879a-c60a860da839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs" Oct 30 16:39:36.423151 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.423131 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b2b662-ec40-4014-879a-c60a860da839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bacd9d6-rrfxs\" (UID: \"f6b2b662-ec40-4014-879a-c60a860da839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs" Oct 30 16:39:36.444539 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.444511 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m468b\" (UniqueName: \"kubernetes.io/projected/f6b2b662-ec40-4014-879a-c60a860da839-kube-api-access-m468b\") pod \"maas-api-revoke-sub-9bacd9d6-rrfxs\" (UID: \"f6b2b662-ec40-4014-879a-c60a860da839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs" Oct 30 16:39:36.464430 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.464398 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs" Oct 30 16:39:36.770202 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:39:36.770176 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b2b662_ec40_4014_879a_c60a860da839.slice/crio-ca32e9edea123bd9640bd4c811a5cfd67d1e80144ff90b9942a5c7217542eb63 WatchSource:0}: Error finding container ca32e9edea123bd9640bd4c811a5cfd67d1e80144ff90b9942a5c7217542eb63: Status 404 returned error can't find the container with id ca32e9edea123bd9640bd4c811a5cfd67d1e80144ff90b9942a5c7217542eb63 Oct 30 16:39:36.772950 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.772922 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs"] Oct 30 16:39:36.826459 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:36.826427 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:37.089949 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:37.089909 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs" event={"ID":"f6b2b662-ec40-4014-879a-c60a860da839","Type":"ContainerStarted","Data":"74f81fadf13a80d989ee766e8841cf76e7e27a6b740f41af96c635d050255473"} Oct 30 16:39:37.090121 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:37.089978 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs" event={"ID":"f6b2b662-ec40-4014-879a-c60a860da839","Type":"ContainerStarted","Data":"ca32e9edea123bd9640bd4c811a5cfd67d1e80144ff90b9942a5c7217542eb63"} Oct 30 16:39:37.101080 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:37.101056 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:37.131348 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:37.131306 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs" podStartSLOduration=1.131292818 podStartE2EDuration="1.131292818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:37.129746062 +0000 UTC m=+1090.840200805" watchObservedRunningTime="2026-10-30 16:39:37.131292818 +0000 UTC m=+1090.841747561" Oct 30 16:39:37.216232 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:37.216199 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:38.099499 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:38.099467 6202 generic.go:356] "Generic (PLEG): container finished" podID="f6b2b662-ec40-4014-879a-c60a860da839" containerID="74f81fadf13a80d989ee766e8841cf76e7e27a6b740f41af96c635d050255473" exitCode=0 Oct 30 16:39:38.099927 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:38.099591 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs" event={"ID":"f6b2b662-ec40-4014-879a-c60a860da839","Type":"ContainerDied","Data":"74f81fadf13a80d989ee766e8841cf76e7e27a6b740f41af96c635d050255473"} Oct 30 16:39:38.347408 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:38.347367 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:39.275276 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.275250 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs" Oct 30 16:39:39.359540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.359447 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b2b662-ec40-4014-879a-c60a860da839-kube-api-access-m468b\" (UniqueName: \"kubernetes.io/projected/f6b2b662-ec40-4014-879a-c60a860da839-kube-api-access-m468b\") pod \"f6b2b662-ec40-4014-879a-c60a860da839\" (UID: \"f6b2b662-ec40-4014-879a-c60a860da839\") " Oct 30 16:39:39.359540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.359522 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6b2b662-ec40-4014-879a-c60a860da839-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b2b662-ec40-4014-879a-c60a860da839-maas-api-xks-ca\") pod \"f6b2b662-ec40-4014-879a-c60a860da839\" (UID: \"f6b2b662-ec40-4014-879a-c60a860da839\") " Oct 30 16:39:39.359755 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.359620 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b2b662-ec40-4014-879a-c60a860da839-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b2b662-ec40-4014-879a-c60a860da839-maas-api-openshift-ca\") pod \"f6b2b662-ec40-4014-879a-c60a860da839\" (UID: \"f6b2b662-ec40-4014-879a-c60a860da839\") " Oct 30 16:39:39.360440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.360409 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b2b662-ec40-4014-879a-c60a860da839-maas-api-openshift-ca" pod "f6b2b662-ec40-4014-879a-c60a860da839" (UID: "f6b2b662-ec40-4014-879a-c60a860da839"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:39.362816 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.362786 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6b2b662-ec40-4014-879a-c60a860da839-maas-api-xks-ca" pod "f6b2b662-ec40-4014-879a-c60a860da839" (UID: "f6b2b662-ec40-4014-879a-c60a860da839"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:39.362929 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.362893 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b2b662-ec40-4014-879a-c60a860da839-kube-api-access-m468b" pod "f6b2b662-ec40-4014-879a-c60a860da839" (UID: "f6b2b662-ec40-4014-879a-c60a860da839"). InnerVolumeSpecName "kube-api-access-m468b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:39.420480 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.420455 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6"] Oct 30 16:39:39.421116 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.421102 6202 cpu_manager.go:414] "containerMap: removing container" podUID="f6b2b662-ec40-4014-879a-c60a860da839" containerName="revoke-keys" Oct 30 16:39:39.421169 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.421118 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b2b662-ec40-4014-879a-c60a860da839" containerName="revoke-keys" Oct 30 16:39:39.421232 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.421223 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b2b662-ec40-4014-879a-c60a860da839" containerName="revoke-keys" Oct 30 16:39:39.424940 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.424916 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6" Oct 30 16:39:39.454859 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.454825 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6"] Oct 30 16:39:39.462872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.461946 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73814b8-e973-48ff-891c-f4c1ed64c28a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6d5dd50-rvpn6\" (UID: \"b73814b8-e973-48ff-891c-f4c1ed64c28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6" Oct 30 16:39:39.462872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.462141 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73814b8-e973-48ff-891c-f4c1ed64c28a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6d5dd50-rvpn6\" (UID: \"b73814b8-e973-48ff-891c-f4c1ed64c28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6" Oct 30 16:39:39.462872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.462167 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z7ss\" (UniqueName: \"kubernetes.io/projected/b73814b8-e973-48ff-891c-f4c1ed64c28a-kube-api-access-7z7ss\") pod \"maas-api-revoke-sub-c6d5dd50-rvpn6\" (UID: \"b73814b8-e973-48ff-891c-f4c1ed64c28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6" Oct 30 16:39:39.462872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.462282 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m468b\" (UniqueName: \"kubernetes.io/projected/f6b2b662-ec40-4014-879a-c60a860da839-kube-api-access-m468b\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:39.462872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.462298 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b2b662-ec40-4014-879a-c60a860da839-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:39.462872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.462314 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b2b662-ec40-4014-879a-c60a860da839-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:39.564932 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.564884 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73814b8-e973-48ff-891c-f4c1ed64c28a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6d5dd50-rvpn6\" (UID: \"b73814b8-e973-48ff-891c-f4c1ed64c28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6" Oct 30 16:39:39.564932 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.564935 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z7ss\" (UniqueName: \"kubernetes.io/projected/b73814b8-e973-48ff-891c-f4c1ed64c28a-kube-api-access-7z7ss\") pod \"maas-api-revoke-sub-c6d5dd50-rvpn6\" (UID: \"b73814b8-e973-48ff-891c-f4c1ed64c28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6" Oct 30 16:39:39.565138 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.565126 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73814b8-e973-48ff-891c-f4c1ed64c28a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6d5dd50-rvpn6\" (UID: \"b73814b8-e973-48ff-891c-f4c1ed64c28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6" Oct 30 16:39:39.565924 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.565897 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73814b8-e973-48ff-891c-f4c1ed64c28a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6d5dd50-rvpn6\" (UID: \"b73814b8-e973-48ff-891c-f4c1ed64c28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6" Oct 30 16:39:39.567490 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.567466 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73814b8-e973-48ff-891c-f4c1ed64c28a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6d5dd50-rvpn6\" (UID: \"b73814b8-e973-48ff-891c-f4c1ed64c28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6" Oct 30 16:39:39.588948 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.588921 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z7ss\" (UniqueName: \"kubernetes.io/projected/b73814b8-e973-48ff-891c-f4c1ed64c28a-kube-api-access-7z7ss\") pod \"maas-api-revoke-sub-c6d5dd50-rvpn6\" (UID: \"b73814b8-e973-48ff-891c-f4c1ed64c28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6" Oct 30 16:39:39.738292 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.738218 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6" Oct 30 16:39:39.844798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:39.844729 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:40.043608 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:39:40.043566 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73814b8_e973_48ff_891c_f4c1ed64c28a.slice/crio-0a6038b76a8ef412e4c4e2fb80aae18d9bfb98788c77410b1121f61cefa89d99 WatchSource:0}: Error finding container 0a6038b76a8ef412e4c4e2fb80aae18d9bfb98788c77410b1121f61cefa89d99: Status 404 returned error can't find the container with id 0a6038b76a8ef412e4c4e2fb80aae18d9bfb98788c77410b1121f61cefa89d99 Oct 30 16:39:40.044571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:40.044552 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6"] Oct 30 16:39:40.115128 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:40.115101 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs" Oct 30 16:39:40.115272 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:40.115130 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs" event={"ID":"f6b2b662-ec40-4014-879a-c60a860da839","Type":"ContainerDied","Data":"ca32e9edea123bd9640bd4c811a5cfd67d1e80144ff90b9942a5c7217542eb63"} Oct 30 16:39:40.115272 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:40.115179 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca32e9edea123bd9640bd4c811a5cfd67d1e80144ff90b9942a5c7217542eb63" Oct 30 16:39:40.116679 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:40.116657 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6" event={"ID":"b73814b8-e973-48ff-891c-f4c1ed64c28a","Type":"ContainerStarted","Data":"0a6038b76a8ef412e4c4e2fb80aae18d9bfb98788c77410b1121f61cefa89d99"} Oct 30 16:39:41.130991 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:41.130950 6202 generic.go:356] "Generic (PLEG): container finished" podID="b73814b8-e973-48ff-891c-f4c1ed64c28a" containerID="5940e3b0a48fba163fa737275fd9d3be5095b3c3e17aa90c6d21c8baa104b2bc" exitCode=0 Oct 30 16:39:41.131520 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:41.131012 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6" event={"ID":"b73814b8-e973-48ff-891c-f4c1ed64c28a","Type":"ContainerDied","Data":"5940e3b0a48fba163fa737275fd9d3be5095b3c3e17aa90c6d21c8baa104b2bc"} Oct 30 16:39:41.468794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:41.467621 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:42.730966 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:42.730941 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6" Oct 30 16:39:42.852548 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:42.852515 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73814b8-e973-48ff-891c-f4c1ed64c28a-kube-api-access-7z7ss\" (UniqueName: \"kubernetes.io/projected/b73814b8-e973-48ff-891c-f4c1ed64c28a-kube-api-access-7z7ss\") pod \"b73814b8-e973-48ff-891c-f4c1ed64c28a\" (UID: \"b73814b8-e973-48ff-891c-f4c1ed64c28a\") " Oct 30 16:39:42.852731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:42.852560 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b73814b8-e973-48ff-891c-f4c1ed64c28a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73814b8-e973-48ff-891c-f4c1ed64c28a-maas-api-openshift-ca\") pod \"b73814b8-e973-48ff-891c-f4c1ed64c28a\" (UID: \"b73814b8-e973-48ff-891c-f4c1ed64c28a\") " Oct 30 16:39:42.852731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:42.852648 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b73814b8-e973-48ff-891c-f4c1ed64c28a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73814b8-e973-48ff-891c-f4c1ed64c28a-maas-api-xks-ca\") pod \"b73814b8-e973-48ff-891c-f4c1ed64c28a\" (UID: \"b73814b8-e973-48ff-891c-f4c1ed64c28a\") " Oct 30 16:39:42.852996 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:42.852966 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73814b8-e973-48ff-891c-f4c1ed64c28a-maas-api-openshift-ca" pod "b73814b8-e973-48ff-891c-f4c1ed64c28a" (UID: "b73814b8-e973-48ff-891c-f4c1ed64c28a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:42.853493 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:42.853468 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73814b8-e973-48ff-891c-f4c1ed64c28a-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:42.854902 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:42.854873 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73814b8-e973-48ff-891c-f4c1ed64c28a-kube-api-access-7z7ss" pod "b73814b8-e973-48ff-891c-f4c1ed64c28a" (UID: "b73814b8-e973-48ff-891c-f4c1ed64c28a"). InnerVolumeSpecName "kube-api-access-7z7ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:42.855019 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:42.855000 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b73814b8-e973-48ff-891c-f4c1ed64c28a-maas-api-xks-ca" pod "b73814b8-e973-48ff-891c-f4c1ed64c28a" (UID: "b73814b8-e973-48ff-891c-f4c1ed64c28a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:42.954647 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:42.954613 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73814b8-e973-48ff-891c-f4c1ed64c28a-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:42.954647 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:42.954642 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z7ss\" (UniqueName: \"kubernetes.io/projected/b73814b8-e973-48ff-891c-f4c1ed64c28a-kube-api-access-7z7ss\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:43.180643 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:43.180541 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6" event={"ID":"b73814b8-e973-48ff-891c-f4c1ed64c28a","Type":"ContainerDied","Data":"0a6038b76a8ef412e4c4e2fb80aae18d9bfb98788c77410b1121f61cefa89d99"} Oct 30 16:39:43.180643 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:43.180575 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6" Oct 30 16:39:43.180643 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:43.180628 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a6038b76a8ef412e4c4e2fb80aae18d9bfb98788c77410b1121f61cefa89d99" Oct 30 16:39:43.372338 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:43.372308 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:44.070903 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:39:44.070866 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:39:44.998905 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:44.998877 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:46.579142 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:46.579110 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:48.375433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:48.375399 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:49.730390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.730353 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6"] Oct 30 16:39:49.731204 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.731182 6202 cpu_manager.go:414] "containerMap: removing container" podUID="b73814b8-e973-48ff-891c-f4c1ed64c28a" containerName="revoke-keys" Oct 30 16:39:49.731204 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.731205 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73814b8-e973-48ff-891c-f4c1ed64c28a" containerName="revoke-keys" Oct 30 16:39:49.731405 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.731391 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73814b8-e973-48ff-891c-f4c1ed64c28a" containerName="revoke-keys" Oct 30 16:39:49.735432 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.735412 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6" Oct 30 16:39:49.757653 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.757629 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6"] Oct 30 16:39:49.763670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.763650 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:39:49.763846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.763650 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:39:49.763965 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.763735 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:39:49.764076 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.764057 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:39:49.846157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.846122 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0fe0f7d-55rb6\" (UID: \"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6" Oct 30 16:39:49.846301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.846249 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44mrn\" (UniqueName: \"kubernetes.io/projected/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-kube-api-access-44mrn\") pod \"maas-api-revoke-sub-f0fe0f7d-55rb6\" (UID: \"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6" Oct 30 16:39:49.846435 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.846416 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0fe0f7d-55rb6\" (UID: \"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6" Oct 30 16:39:49.881393 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.881365 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:49.947674 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.947639 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44mrn\" (UniqueName: \"kubernetes.io/projected/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-kube-api-access-44mrn\") pod \"maas-api-revoke-sub-f0fe0f7d-55rb6\" (UID: \"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6" Oct 30 16:39:49.947900 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.947840 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0fe0f7d-55rb6\" (UID: \"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6" Oct 30 16:39:49.948017 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.948001 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0fe0f7d-55rb6\" (UID: \"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6" Oct 30 16:39:49.948682 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.948657 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0fe0f7d-55rb6\" (UID: \"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6" Oct 30 16:39:49.950540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.950519 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0fe0f7d-55rb6\" (UID: \"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6" Oct 30 16:39:49.972929 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:49.972559 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44mrn\" (UniqueName: \"kubernetes.io/projected/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-kube-api-access-44mrn\") pod \"maas-api-revoke-sub-f0fe0f7d-55rb6\" (UID: \"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6" Oct 30 16:39:50.051879 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:50.051756 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6" Oct 30 16:39:50.353309 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:39:50.353276 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87cbeb5b_5973_45ec_a9e4_a5ebd55eb8bd.slice/crio-7f5e0741bfb73b7d18691eb22ba2b4feff8818ad4bd03671e47b832018ce33b1 WatchSource:0}: Error finding container 7f5e0741bfb73b7d18691eb22ba2b4feff8818ad4bd03671e47b832018ce33b1: Status 404 returned error can't find the container with id 7f5e0741bfb73b7d18691eb22ba2b4feff8818ad4bd03671e47b832018ce33b1 Oct 30 16:39:50.355078 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:50.355051 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6"] Oct 30 16:39:51.314792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:51.314240 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6" event={"ID":"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd","Type":"ContainerStarted","Data":"20f1aa5a120ccc346bb2270ef339f89dc04868990c7436403e45b20a9345c71d"} Oct 30 16:39:51.314792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:51.314330 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6" event={"ID":"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd","Type":"ContainerStarted","Data":"7f5e0741bfb73b7d18691eb22ba2b4feff8818ad4bd03671e47b832018ce33b1"} Oct 30 16:39:51.378821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:51.377720 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6" podStartSLOduration=2.377702793 podStartE2EDuration="2.377702793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:51.376993786 +0000 UTC m=+1105.087448531" watchObservedRunningTime="2026-10-30 16:39:51.377702793 +0000 UTC m=+1105.088157539" Oct 30 16:39:51.832898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:51.832845 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:52.356803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:52.356533 6202 generic.go:356] "Generic (PLEG): container finished" podID="87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd" containerID="20f1aa5a120ccc346bb2270ef339f89dc04868990c7436403e45b20a9345c71d" exitCode=0 Oct 30 16:39:52.356803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:52.356679 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6" event={"ID":"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd","Type":"ContainerDied","Data":"20f1aa5a120ccc346bb2270ef339f89dc04868990c7436403e45b20a9345c71d"} Oct 30 16:39:53.001697 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:52.996864 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:39:53.001697 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:52.998026 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:53.017833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:53.014623 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:53.017833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:53.015026 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 16:39:53.067789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:53.066515 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:53.092789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:53.091213 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:39:53.204823 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:53.193861 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:53.608724 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:53.608677 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:54.068793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:54.055899 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:54.103850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:54.086413 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:39:54.103850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:54.088440 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:54.103850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:54.088846 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:39:54.117794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:54.108873 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:54.117794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:54.109296 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:39:54.136791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:54.124397 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:39:54.136791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:54.124830 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:39:54.136791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:54.127240 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:54.136791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:54.127540 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:39:54.377861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:54.363914 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:54.377861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:54.364380 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:39:54.747980 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:54.747864 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:54.888799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:54.875618 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6" Oct 30 16:39:55.102206 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.100604 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6" event={"ID":"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd","Type":"ContainerDied","Data":"7f5e0741bfb73b7d18691eb22ba2b4feff8818ad4bd03671e47b832018ce33b1"} Oct 30 16:39:55.102206 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.100673 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f5e0741bfb73b7d18691eb22ba2b4feff8818ad4bd03671e47b832018ce33b1" Oct 30 16:39:55.112175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.103093 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-kube-api-access-44mrn\" (UniqueName: \"kubernetes.io/projected/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-kube-api-access-44mrn\") pod \"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd\" (UID: \"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd\") " Oct 30 16:39:55.112175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.103141 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-maas-api-xks-ca\") pod \"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd\" (UID: \"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd\") " Oct 30 16:39:55.112175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.103317 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-maas-api-openshift-ca\") pod \"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd\" (UID: \"87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd\") " Oct 30 16:39:55.115738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.114534 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-maas-api-openshift-ca" pod "87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd" (UID: "87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:55.118287 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.116916 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:55.119268 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.119241 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:55.121295 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.121261 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:55.256963 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.125080 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:55.311997 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.128359 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:55.311997 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.203696 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:55.311997 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.209151 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:55.352095 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.352055 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-maas-api-xks-ca" pod "87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd" (UID: "87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:55.368649 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.367988 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:55.369967 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.369916 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-kube-api-access-44mrn" pod "87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd" (UID: "87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd"). InnerVolumeSpecName "kube-api-access-44mrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:55.412928 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.388193 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44mrn\" (UniqueName: \"kubernetes.io/projected/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-kube-api-access-44mrn\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:55.412928 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.388226 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:39:55.568249 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.541646 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded" Oct 30 16:39:55.615928 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.615883 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:55.643560 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.643485 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:39:55.646230 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.646195 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:39:55.947821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:55.946788 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6" Oct 30 16:39:56.090579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.088189 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:39:56.114790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.102146 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:39:56.127111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.117044 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 16:39:56.127111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.120833 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:56.127111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.122163 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:56.127111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.122546 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:39:56.156655 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.149422 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:39:56.192247 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.192196 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:39:56.236011 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.235919 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:39:56.277233 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.277192 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:56.672005 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.671969 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr"] Oct 30 16:39:56.672651 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.672630 6202 cpu_manager.go:414] "containerMap: removing container" podUID="87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd" containerName="revoke-keys" Oct 30 16:39:56.672761 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.672652 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd" containerName="revoke-keys" Oct 30 16:39:56.672889 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.672875 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd" containerName="revoke-keys" Oct 30 16:39:56.676661 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.676641 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr" Oct 30 16:39:56.693359 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.693335 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:39:56.693594 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.693577 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:39:56.698248 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.698229 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr"] Oct 30 16:39:56.706153 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.704616 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:39:56.706153 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.704630 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:39:56.735664 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.735631 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4vtm\" (UniqueName: \"kubernetes.io/projected/49f93fed-b0a8-4c0b-9bb9-16526a966a13-kube-api-access-g4vtm\") pod \"maas-api-revoke-sub-d5dc9043-h2vsr\" (UID: \"49f93fed-b0a8-4c0b-9bb9-16526a966a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr" Oct 30 16:39:56.735859 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.735827 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f93fed-b0a8-4c0b-9bb9-16526a966a13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5dc9043-h2vsr\" (UID: \"49f93fed-b0a8-4c0b-9bb9-16526a966a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr" Oct 30 16:39:56.736022 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.735868 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f93fed-b0a8-4c0b-9bb9-16526a966a13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5dc9043-h2vsr\" (UID: \"49f93fed-b0a8-4c0b-9bb9-16526a966a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr" Oct 30 16:39:56.837224 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.837187 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4vtm\" (UniqueName: \"kubernetes.io/projected/49f93fed-b0a8-4c0b-9bb9-16526a966a13-kube-api-access-g4vtm\") pod \"maas-api-revoke-sub-d5dc9043-h2vsr\" (UID: \"49f93fed-b0a8-4c0b-9bb9-16526a966a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr" Oct 30 16:39:56.837395 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.837371 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f93fed-b0a8-4c0b-9bb9-16526a966a13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5dc9043-h2vsr\" (UID: \"49f93fed-b0a8-4c0b-9bb9-16526a966a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr" Oct 30 16:39:56.837464 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.837407 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f93fed-b0a8-4c0b-9bb9-16526a966a13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5dc9043-h2vsr\" (UID: \"49f93fed-b0a8-4c0b-9bb9-16526a966a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr" Oct 30 16:39:56.838279 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.838245 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f93fed-b0a8-4c0b-9bb9-16526a966a13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5dc9043-h2vsr\" (UID: \"49f93fed-b0a8-4c0b-9bb9-16526a966a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr" Oct 30 16:39:56.840645 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.840616 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f93fed-b0a8-4c0b-9bb9-16526a966a13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5dc9043-h2vsr\" (UID: \"49f93fed-b0a8-4c0b-9bb9-16526a966a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr" Oct 30 16:39:56.861342 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.861302 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4vtm\" (UniqueName: \"kubernetes.io/projected/49f93fed-b0a8-4c0b-9bb9-16526a966a13-kube-api-access-g4vtm\") pod \"maas-api-revoke-sub-d5dc9043-h2vsr\" (UID: \"49f93fed-b0a8-4c0b-9bb9-16526a966a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr" Oct 30 16:39:56.991234 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:56.991153 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr" Oct 30 16:39:57.312211 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:39:57.298369 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f93fed_b0a8_4c0b_9bb9_16526a966a13.slice/crio-a3a15372b6441bb4b8592be1708fa57630b75451f059a79e46ecb754210a9207 WatchSource:0}: Error finding container a3a15372b6441bb4b8592be1708fa57630b75451f059a79e46ecb754210a9207: Status 404 returned error can't find the container with id a3a15372b6441bb4b8592be1708fa57630b75451f059a79e46ecb754210a9207 Oct 30 16:39:57.312211 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:57.298558 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr"] Oct 30 16:39:57.359627 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:57.359585 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:57.963727 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:57.963690 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr" event={"ID":"49f93fed-b0a8-4c0b-9bb9-16526a966a13","Type":"ContainerStarted","Data":"fd71d23467470b9dfc6d1c5d4f1af68fff92bc7f35dd1361e9ad5fc620ab5809"} Oct 30 16:39:57.963922 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:57.963795 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr" event={"ID":"49f93fed-b0a8-4c0b-9bb9-16526a966a13","Type":"ContainerStarted","Data":"a3a15372b6441bb4b8592be1708fa57630b75451f059a79e46ecb754210a9207"} Oct 30 16:39:58.007995 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.007947 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr" podStartSLOduration=2.007928781 podStartE2EDuration="2.007928781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:58.004850952 +0000 UTC m=+1111.715305696" watchObservedRunningTime="2026-10-30 16:39:58.007928781 +0000 UTC m=+1111.718383525" Oct 30 16:39:58.035171 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.035141 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4"] Oct 30 16:39:58.039791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.039755 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4" Oct 30 16:39:58.060251 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.060218 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvdcd\" (UniqueName: \"kubernetes.io/projected/0e89a950-1f55-4aa9-8e35-59472d90d654-kube-api-access-rvdcd\") pod \"maas-api-revoke-sub-00adcb0a-prxz4\" (UID: \"0e89a950-1f55-4aa9-8e35-59472d90d654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4" Oct 30 16:39:58.060421 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.060347 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e89a950-1f55-4aa9-8e35-59472d90d654-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00adcb0a-prxz4\" (UID: \"0e89a950-1f55-4aa9-8e35-59472d90d654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4" Oct 30 16:39:58.060577 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.060555 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e89a950-1f55-4aa9-8e35-59472d90d654-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00adcb0a-prxz4\" (UID: \"0e89a950-1f55-4aa9-8e35-59472d90d654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4" Oct 30 16:39:58.061790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.061601 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4"] Oct 30 16:39:58.161835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.161804 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e89a950-1f55-4aa9-8e35-59472d90d654-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00adcb0a-prxz4\" (UID: \"0e89a950-1f55-4aa9-8e35-59472d90d654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4" Oct 30 16:39:58.161990 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.161922 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvdcd\" (UniqueName: \"kubernetes.io/projected/0e89a950-1f55-4aa9-8e35-59472d90d654-kube-api-access-rvdcd\") pod \"maas-api-revoke-sub-00adcb0a-prxz4\" (UID: \"0e89a950-1f55-4aa9-8e35-59472d90d654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4" Oct 30 16:39:58.161990 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.161980 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e89a950-1f55-4aa9-8e35-59472d90d654-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00adcb0a-prxz4\" (UID: \"0e89a950-1f55-4aa9-8e35-59472d90d654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4" Oct 30 16:39:58.162603 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.162578 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e89a950-1f55-4aa9-8e35-59472d90d654-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00adcb0a-prxz4\" (UID: \"0e89a950-1f55-4aa9-8e35-59472d90d654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4" Oct 30 16:39:58.164423 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.164401 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e89a950-1f55-4aa9-8e35-59472d90d654-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00adcb0a-prxz4\" (UID: \"0e89a950-1f55-4aa9-8e35-59472d90d654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4" Oct 30 16:39:58.185700 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.185671 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvdcd\" (UniqueName: \"kubernetes.io/projected/0e89a950-1f55-4aa9-8e35-59472d90d654-kube-api-access-rvdcd\") pod \"maas-api-revoke-sub-00adcb0a-prxz4\" (UID: \"0e89a950-1f55-4aa9-8e35-59472d90d654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4" Oct 30 16:39:58.190324 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.190300 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:58.354396 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.354372 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4" Oct 30 16:39:58.699089 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:39:58.699053 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e89a950_1f55_4aa9_8e35_59472d90d654.slice/crio-2eec65e9d416c36cfe9f42dc4ef709945075f64f1c7a183539493dce898c14a8 WatchSource:0}: Error finding container 2eec65e9d416c36cfe9f42dc4ef709945075f64f1c7a183539493dce898c14a8: Status 404 returned error can't find the container with id 2eec65e9d416c36cfe9f42dc4ef709945075f64f1c7a183539493dce898c14a8 Oct 30 16:39:58.700438 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.700410 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4"] Oct 30 16:39:58.973863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.973809 6202 generic.go:356] "Generic (PLEG): container finished" podID="49f93fed-b0a8-4c0b-9bb9-16526a966a13" containerID="fd71d23467470b9dfc6d1c5d4f1af68fff92bc7f35dd1361e9ad5fc620ab5809" exitCode=0 Oct 30 16:39:58.974023 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.973861 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr" event={"ID":"49f93fed-b0a8-4c0b-9bb9-16526a966a13","Type":"ContainerDied","Data":"fd71d23467470b9dfc6d1c5d4f1af68fff92bc7f35dd1361e9ad5fc620ab5809"} Oct 30 16:39:58.975262 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:58.975243 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4" event={"ID":"0e89a950-1f55-4aa9-8e35-59472d90d654","Type":"ContainerStarted","Data":"2eec65e9d416c36cfe9f42dc4ef709945075f64f1c7a183539493dce898c14a8"} Oct 30 16:39:59.853334 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:59.853294 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:59.986381 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:59.986293 6202 generic.go:356] "Generic (PLEG): container finished" podID="0e89a950-1f55-4aa9-8e35-59472d90d654" containerID="1aba65d61c5ba2d5420f66d3bea0e3ce9a019f5dd723d24b3ef3c907b9749c4d" exitCode=0 Oct 30 16:39:59.986381 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:39:59.986361 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4" event={"ID":"0e89a950-1f55-4aa9-8e35-59472d90d654","Type":"ContainerDied","Data":"1aba65d61c5ba2d5420f66d3bea0e3ce9a019f5dd723d24b3ef3c907b9749c4d"} Oct 30 16:40:00.142282 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:00.142256 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr" Oct 30 16:40:00.195485 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:00.195459 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f93fed-b0a8-4c0b-9bb9-16526a966a13-kube-api-access-g4vtm\" (UniqueName: \"kubernetes.io/projected/49f93fed-b0a8-4c0b-9bb9-16526a966a13-kube-api-access-g4vtm\") pod \"49f93fed-b0a8-4c0b-9bb9-16526a966a13\" (UID: \"49f93fed-b0a8-4c0b-9bb9-16526a966a13\") " Oct 30 16:40:00.195485 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:00.195490 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f93fed-b0a8-4c0b-9bb9-16526a966a13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f93fed-b0a8-4c0b-9bb9-16526a966a13-maas-api-openshift-ca\") pod \"49f93fed-b0a8-4c0b-9bb9-16526a966a13\" (UID: \"49f93fed-b0a8-4c0b-9bb9-16526a966a13\") " Oct 30 16:40:00.195736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:00.195702 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49f93fed-b0a8-4c0b-9bb9-16526a966a13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f93fed-b0a8-4c0b-9bb9-16526a966a13-maas-api-xks-ca\") pod \"49f93fed-b0a8-4c0b-9bb9-16526a966a13\" (UID: \"49f93fed-b0a8-4c0b-9bb9-16526a966a13\") " Oct 30 16:40:00.196233 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:00.196186 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f93fed-b0a8-4c0b-9bb9-16526a966a13-maas-api-openshift-ca" pod "49f93fed-b0a8-4c0b-9bb9-16526a966a13" (UID: "49f93fed-b0a8-4c0b-9bb9-16526a966a13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:40:00.198501 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:00.198476 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f93fed-b0a8-4c0b-9bb9-16526a966a13-kube-api-access-g4vtm" pod "49f93fed-b0a8-4c0b-9bb9-16526a966a13" (UID: "49f93fed-b0a8-4c0b-9bb9-16526a966a13"). InnerVolumeSpecName "kube-api-access-g4vtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:00.198760 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:00.198729 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f93fed-b0a8-4c0b-9bb9-16526a966a13-maas-api-xks-ca" pod "49f93fed-b0a8-4c0b-9bb9-16526a966a13" (UID: "49f93fed-b0a8-4c0b-9bb9-16526a966a13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:40:00.297592 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:00.297518 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f93fed-b0a8-4c0b-9bb9-16526a966a13-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:00.297592 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:00.297544 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4vtm\" (UniqueName: \"kubernetes.io/projected/49f93fed-b0a8-4c0b-9bb9-16526a966a13-kube-api-access-g4vtm\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:00.297592 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:00.297553 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f93fed-b0a8-4c0b-9bb9-16526a966a13-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:00.995155 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:00.995118 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr" Oct 30 16:40:00.995155 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:00.995147 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr" event={"ID":"49f93fed-b0a8-4c0b-9bb9-16526a966a13","Type":"ContainerDied","Data":"a3a15372b6441bb4b8592be1708fa57630b75451f059a79e46ecb754210a9207"} Oct 30 16:40:00.995584 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:00.995220 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3a15372b6441bb4b8592be1708fa57630b75451f059a79e46ecb754210a9207" Oct 30 16:40:01.159066 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:01.159041 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4" Oct 30 16:40:01.216027 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:01.215995 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e89a950-1f55-4aa9-8e35-59472d90d654-kube-api-access-rvdcd\" (UniqueName: \"kubernetes.io/projected/0e89a950-1f55-4aa9-8e35-59472d90d654-kube-api-access-rvdcd\") pod \"0e89a950-1f55-4aa9-8e35-59472d90d654\" (UID: \"0e89a950-1f55-4aa9-8e35-59472d90d654\") " Oct 30 16:40:01.216209 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:01.216096 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e89a950-1f55-4aa9-8e35-59472d90d654-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e89a950-1f55-4aa9-8e35-59472d90d654-maas-api-openshift-ca\") pod \"0e89a950-1f55-4aa9-8e35-59472d90d654\" (UID: \"0e89a950-1f55-4aa9-8e35-59472d90d654\") " Oct 30 16:40:01.216279 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:01.216226 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e89a950-1f55-4aa9-8e35-59472d90d654-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e89a950-1f55-4aa9-8e35-59472d90d654-maas-api-xks-ca\") pod \"0e89a950-1f55-4aa9-8e35-59472d90d654\" (UID: \"0e89a950-1f55-4aa9-8e35-59472d90d654\") " Oct 30 16:40:01.216542 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:01.216494 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e89a950-1f55-4aa9-8e35-59472d90d654-maas-api-openshift-ca" pod "0e89a950-1f55-4aa9-8e35-59472d90d654" (UID: "0e89a950-1f55-4aa9-8e35-59472d90d654"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:40:01.216807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:01.216749 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e89a950-1f55-4aa9-8e35-59472d90d654-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:01.218312 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:01.218282 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e89a950-1f55-4aa9-8e35-59472d90d654-kube-api-access-rvdcd" pod "0e89a950-1f55-4aa9-8e35-59472d90d654" (UID: "0e89a950-1f55-4aa9-8e35-59472d90d654"). InnerVolumeSpecName "kube-api-access-rvdcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:01.218430 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:01.218409 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e89a950-1f55-4aa9-8e35-59472d90d654-maas-api-xks-ca" pod "0e89a950-1f55-4aa9-8e35-59472d90d654" (UID: "0e89a950-1f55-4aa9-8e35-59472d90d654"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:40:01.318235 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:01.318143 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e89a950-1f55-4aa9-8e35-59472d90d654-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:01.318235 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:01.318180 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvdcd\" (UniqueName: \"kubernetes.io/projected/0e89a950-1f55-4aa9-8e35-59472d90d654-kube-api-access-rvdcd\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:01.592514 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:01.592483 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:02.003787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:02.003689 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4" Oct 30 16:40:02.003787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:02.003726 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4" event={"ID":"0e89a950-1f55-4aa9-8e35-59472d90d654","Type":"ContainerDied","Data":"2eec65e9d416c36cfe9f42dc4ef709945075f64f1c7a183539493dce898c14a8"} Oct 30 16:40:02.004206 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:02.003789 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eec65e9d416c36cfe9f42dc4ef709945075f64f1c7a183539493dce898c14a8" Oct 30 16:40:03.156200 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:03.156165 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:04.332414 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.332381 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z"] Oct 30 16:40:04.333269 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.333248 6202 cpu_manager.go:414] "containerMap: removing container" podUID="0e89a950-1f55-4aa9-8e35-59472d90d654" containerName="revoke-keys" Oct 30 16:40:04.333269 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.333268 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e89a950-1f55-4aa9-8e35-59472d90d654" containerName="revoke-keys" Oct 30 16:40:04.333402 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.333289 6202 cpu_manager.go:414] "containerMap: removing container" podUID="49f93fed-b0a8-4c0b-9bb9-16526a966a13" containerName="revoke-keys" Oct 30 16:40:04.333402 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.333294 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f93fed-b0a8-4c0b-9bb9-16526a966a13" containerName="revoke-keys" Oct 30 16:40:04.333465 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.333423 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e89a950-1f55-4aa9-8e35-59472d90d654" containerName="revoke-keys" Oct 30 16:40:04.333465 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.333434 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f93fed-b0a8-4c0b-9bb9-16526a966a13" containerName="revoke-keys" Oct 30 16:40:04.339501 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.339474 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.376906 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.376873 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z"] Oct 30 16:40:04.462326 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.462293 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.462496 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.462354 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.462556 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.462520 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.462603 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.462586 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51725959-f0c8-4b0f-bd22-547a9995c437-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.462752 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.462729 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51725959-f0c8-4b0f-bd22-547a9995c437-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.462845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.462830 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.462882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.462867 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlvf9\" (UniqueName: \"kubernetes.io/projected/51725959-f0c8-4b0f-bd22-547a9995c437-kube-api-access-jlvf9\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.462953 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.462937 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.462992 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.462982 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51725959-f0c8-4b0f-bd22-547a9995c437-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.564326 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.564290 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.564494 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.564356 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.564494 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.564435 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.564494 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.564472 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51725959-f0c8-4b0f-bd22-547a9995c437-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.564682 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.564552 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51725959-f0c8-4b0f-bd22-547a9995c437-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.564682 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.564601 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.564682 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.564632 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlvf9\" (UniqueName: \"kubernetes.io/projected/51725959-f0c8-4b0f-bd22-547a9995c437-kube-api-access-jlvf9\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.564682 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.564677 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.564900 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.564728 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.564900 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.564742 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51725959-f0c8-4b0f-bd22-547a9995c437-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.564900 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.564866 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.565082 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.565047 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.565268 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.565245 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.565615 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.565596 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51725959-f0c8-4b0f-bd22-547a9995c437-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.567261 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.567240 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.567553 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.567529 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51725959-f0c8-4b0f-bd22-547a9995c437-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.588725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.588655 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51725959-f0c8-4b0f-bd22-547a9995c437-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.588986 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.588965 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlvf9\" (UniqueName: \"kubernetes.io/projected/51725959-f0c8-4b0f-bd22-547a9995c437-kube-api-access-jlvf9\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.657699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.657670 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:04.798008 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.797960 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:04.977586 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:40:04.977543 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51725959_f0c8_4b0f_bd22_547a9995c437.slice/crio-2043764ce2092a62e0c5d9159af0dc9e90ebb9a0e7151c2cd3c38acd888f7bc9 WatchSource:0}: Error finding container 2043764ce2092a62e0c5d9159af0dc9e90ebb9a0e7151c2cd3c38acd888f7bc9: Status 404 returned error can't find the container with id 2043764ce2092a62e0c5d9159af0dc9e90ebb9a0e7151c2cd3c38acd888f7bc9 Oct 30 16:40:04.978734 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.978704 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z"] Oct 30 16:40:04.980634 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.980599 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:40:04.980730 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.980673 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:40:04.980730 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:04.980713 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:40:05.027566 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:05.027539 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" event={"ID":"51725959-f0c8-4b0f-bd22-547a9995c437","Type":"ContainerStarted","Data":"2043764ce2092a62e0c5d9159af0dc9e90ebb9a0e7151c2cd3c38acd888f7bc9"} Oct 30 16:40:06.069448 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:06.069406 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" event={"ID":"51725959-f0c8-4b0f-bd22-547a9995c437","Type":"ContainerStarted","Data":"53e4f54a138ade97e8bccc433bc15259dbd591316a44d5d6e906e787fd598a15"} Oct 30 16:40:06.150055 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:06.149993 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" podStartSLOduration=2.149970595 podStartE2EDuration="2.149970595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:06.126208437 +0000 UTC m=+1119.836663182" watchObservedRunningTime="2026-10-30 16:40:06.149970595 +0000 UTC m=+1119.860425340" Oct 30 16:40:06.663048 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:06.663012 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:06.663403 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:06.663378 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:40:06.663575 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:06.663549 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" podUID="51725959-f0c8-4b0f-bd22-547a9995c437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 16:40:07.326267 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:07.326165 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:07.445403 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:07.445318 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:07.610074 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:07.609940 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:07.660307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:07.660272 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:40:07.660573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:07.660549 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" podUID="51725959-f0c8-4b0f-bd22-547a9995c437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 16:40:08.008849 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:08.008679 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:08.140800 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:08.127885 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:08.204791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:08.203914 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:08.376790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:08.375981 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded" Oct 30 16:40:08.623229 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:08.614850 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:08.623229 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:08.615339 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:40:08.655841 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:08.655138 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:08.660162 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:08.660130 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:40:08.660401 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:08.660373 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" podUID="51725959-f0c8-4b0f-bd22-547a9995c437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 16:40:09.125301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:09.125253 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:09.125974 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:09.125931 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 16:40:09.131880 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:09.131846 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:09.135790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:09.132408 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:40:09.199792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:09.192390 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:09.232794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:09.219262 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:09.350791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:09.331304 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:40:09.424295 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:09.424201 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:09.426230 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:09.426190 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:40:09.663618 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:09.635799 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:09.663618 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:09.661726 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:09.663618 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:09.662072 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:40:09.733022 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:09.732939 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:40:09.736346 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:09.736306 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" podUID="51725959-f0c8-4b0f-bd22-547a9995c437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 16:40:09.826597 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:09.822589 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:40:09.833337 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:09.833306 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:40:09.904132 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:09.868576 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:40:10.016505 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:10.016420 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:40:10.017512 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:10.017487 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:40:10.560575 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:10.560529 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:10.667702 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:10.667667 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:40:10.668014 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:10.667985 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" podUID="51725959-f0c8-4b0f-bd22-547a9995c437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 16:40:11.658186 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:11.658135 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:40:11.658803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:11.658741 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" podUID="51725959-f0c8-4b0f-bd22-547a9995c437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 16:40:11.784307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:11.784271 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq"] Oct 30 16:40:11.789229 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:11.789206 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq" Oct 30 16:40:11.808912 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:11.808885 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq"] Oct 30 16:40:11.815596 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:11.815570 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:40:11.815736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:11.815624 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:40:11.815736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:11.815694 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:40:11.815881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:11.815745 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:40:11.897607 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:11.897565 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtwd\" (UniqueName: \"kubernetes.io/projected/9a14f5f6-8af9-43d7-8775-26852b873c72-kube-api-access-9mtwd\") pod \"maas-api-revoke-sub-3a4fc01e-f9zbq\" (UID: \"9a14f5f6-8af9-43d7-8775-26852b873c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq" Oct 30 16:40:11.897833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:11.897630 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a14f5f6-8af9-43d7-8775-26852b873c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a4fc01e-f9zbq\" (UID: \"9a14f5f6-8af9-43d7-8775-26852b873c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq" Oct 30 16:40:11.897833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:11.897709 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a14f5f6-8af9-43d7-8775-26852b873c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a4fc01e-f9zbq\" (UID: \"9a14f5f6-8af9-43d7-8775-26852b873c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq" Oct 30 16:40:11.999861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:11.999749 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtwd\" (UniqueName: \"kubernetes.io/projected/9a14f5f6-8af9-43d7-8775-26852b873c72-kube-api-access-9mtwd\") pod \"maas-api-revoke-sub-3a4fc01e-f9zbq\" (UID: \"9a14f5f6-8af9-43d7-8775-26852b873c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq" Oct 30 16:40:11.999861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:11.999822 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a14f5f6-8af9-43d7-8775-26852b873c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a4fc01e-f9zbq\" (UID: \"9a14f5f6-8af9-43d7-8775-26852b873c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq" Oct 30 16:40:12.000129 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:11.999877 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a14f5f6-8af9-43d7-8775-26852b873c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a4fc01e-f9zbq\" (UID: \"9a14f5f6-8af9-43d7-8775-26852b873c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq" Oct 30 16:40:12.000741 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:12.000671 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a14f5f6-8af9-43d7-8775-26852b873c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a4fc01e-f9zbq\" (UID: \"9a14f5f6-8af9-43d7-8775-26852b873c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq" Oct 30 16:40:12.002588 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:12.002567 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a14f5f6-8af9-43d7-8775-26852b873c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a4fc01e-f9zbq\" (UID: \"9a14f5f6-8af9-43d7-8775-26852b873c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq" Oct 30 16:40:12.023209 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:12.023184 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtwd\" (UniqueName: \"kubernetes.io/projected/9a14f5f6-8af9-43d7-8775-26852b873c72-kube-api-access-9mtwd\") pod \"maas-api-revoke-sub-3a4fc01e-f9zbq\" (UID: \"9a14f5f6-8af9-43d7-8775-26852b873c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq" Oct 30 16:40:12.103506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:12.103467 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq" Oct 30 16:40:12.267646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:12.267557 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:12.429958 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:12.429930 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq"] Oct 30 16:40:12.431202 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:40:12.431167 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a14f5f6_8af9_43d7_8775_26852b873c72.slice/crio-b4b90d2ca6af76590036600b307ebda62c2bd86fa19b4fb5d0e618402dff3453 WatchSource:0}: Error finding container b4b90d2ca6af76590036600b307ebda62c2bd86fa19b4fb5d0e618402dff3453: Status 404 returned error can't find the container with id b4b90d2ca6af76590036600b307ebda62c2bd86fa19b4fb5d0e618402dff3453 Oct 30 16:40:12.585310 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:12.585270 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq" event={"ID":"9a14f5f6-8af9-43d7-8775-26852b873c72","Type":"ContainerStarted","Data":"b4b90d2ca6af76590036600b307ebda62c2bd86fa19b4fb5d0e618402dff3453"} Oct 30 16:40:12.659083 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:12.659051 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:40:12.659448 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:12.659112 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" podUID="51725959-f0c8-4b0f-bd22-547a9995c437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 16:40:13.600420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:13.600381 6202 generic.go:356] "Generic (PLEG): container finished" podID="9a14f5f6-8af9-43d7-8775-26852b873c72" containerID="d209eda07c2b6e94b871f045dfb95b74ea2406a66dd0639fca5907208dadae0b" exitCode=0 Oct 30 16:40:13.600583 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:13.600466 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq" event={"ID":"9a14f5f6-8af9-43d7-8775-26852b873c72","Type":"ContainerDied","Data":"d209eda07c2b6e94b871f045dfb95b74ea2406a66dd0639fca5907208dadae0b"} Oct 30 16:40:13.872905 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:13.872860 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:13.873998 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:13.873889 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:13.876599 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:13.876578 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:40:14.063994 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:14.063943 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:14.068829 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:40:14.068760 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:40:14.783677 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:14.783654 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq" Oct 30 16:40:14.847568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:14.847528 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a14f5f6-8af9-43d7-8775-26852b873c72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a14f5f6-8af9-43d7-8775-26852b873c72-maas-api-xks-ca\") pod \"9a14f5f6-8af9-43d7-8775-26852b873c72\" (UID: \"9a14f5f6-8af9-43d7-8775-26852b873c72\") " Oct 30 16:40:14.847815 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:14.847794 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a14f5f6-8af9-43d7-8775-26852b873c72-kube-api-access-9mtwd\" (UniqueName: \"kubernetes.io/projected/9a14f5f6-8af9-43d7-8775-26852b873c72-kube-api-access-9mtwd\") pod \"9a14f5f6-8af9-43d7-8775-26852b873c72\" (UID: \"9a14f5f6-8af9-43d7-8775-26852b873c72\") " Oct 30 16:40:14.847896 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:14.847872 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a14f5f6-8af9-43d7-8775-26852b873c72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a14f5f6-8af9-43d7-8775-26852b873c72-maas-api-openshift-ca\") pod \"9a14f5f6-8af9-43d7-8775-26852b873c72\" (UID: \"9a14f5f6-8af9-43d7-8775-26852b873c72\") " Oct 30 16:40:14.849006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:14.848980 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a14f5f6-8af9-43d7-8775-26852b873c72-maas-api-openshift-ca" pod "9a14f5f6-8af9-43d7-8775-26852b873c72" (UID: "9a14f5f6-8af9-43d7-8775-26852b873c72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:40:14.851014 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:14.850988 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a14f5f6-8af9-43d7-8775-26852b873c72-maas-api-xks-ca" pod "9a14f5f6-8af9-43d7-8775-26852b873c72" (UID: "9a14f5f6-8af9-43d7-8775-26852b873c72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:40:14.856332 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:14.856296 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a14f5f6-8af9-43d7-8775-26852b873c72-kube-api-access-9mtwd" pod "9a14f5f6-8af9-43d7-8775-26852b873c72" (UID: "9a14f5f6-8af9-43d7-8775-26852b873c72"). InnerVolumeSpecName "kube-api-access-9mtwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:14.950449 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:14.950372 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a14f5f6-8af9-43d7-8775-26852b873c72-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:14.950449 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:14.950408 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mtwd\" (UniqueName: \"kubernetes.io/projected/9a14f5f6-8af9-43d7-8775-26852b873c72-kube-api-access-9mtwd\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:14.950449 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:14.950423 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a14f5f6-8af9-43d7-8775-26852b873c72-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:15.619837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:15.619792 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq" event={"ID":"9a14f5f6-8af9-43d7-8775-26852b873c72","Type":"ContainerDied","Data":"b4b90d2ca6af76590036600b307ebda62c2bd86fa19b4fb5d0e618402dff3453"} Oct 30 16:40:15.620028 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:15.619862 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4b90d2ca6af76590036600b307ebda62c2bd86fa19b4fb5d0e618402dff3453" Oct 30 16:40:15.620028 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:15.619811 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq" Oct 30 16:40:15.624730 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:15.624638 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:17.184103 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.184069 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:17.575906 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.575868 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f"] Oct 30 16:40:17.576576 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.576558 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9a14f5f6-8af9-43d7-8775-26852b873c72" containerName="revoke-keys" Oct 30 16:40:17.576635 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.576577 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a14f5f6-8af9-43d7-8775-26852b873c72" containerName="revoke-keys" Oct 30 16:40:17.576693 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.576684 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a14f5f6-8af9-43d7-8775-26852b873c72" containerName="revoke-keys" Oct 30 16:40:17.580975 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.580957 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f" Oct 30 16:40:17.600926 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.600899 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f"] Oct 30 16:40:17.607137 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.607107 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:40:17.607312 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.607198 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:40:17.607390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.607318 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:40:17.607390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.607366 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:40:17.685208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.685168 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dafed04-7d9e-40e7-b28e-42b4607598ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfe0871e-c8p7f\" (UID: \"0dafed04-7d9e-40e7-b28e-42b4607598ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f" Oct 30 16:40:17.685417 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.685239 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dafed04-7d9e-40e7-b28e-42b4607598ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfe0871e-c8p7f\" (UID: \"0dafed04-7d9e-40e7-b28e-42b4607598ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f" Oct 30 16:40:17.685489 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.685464 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz955\" (UniqueName: \"kubernetes.io/projected/0dafed04-7d9e-40e7-b28e-42b4607598ba-kube-api-access-nz955\") pod \"maas-api-revoke-sub-dfe0871e-c8p7f\" (UID: \"0dafed04-7d9e-40e7-b28e-42b4607598ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f" Oct 30 16:40:17.786449 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.786408 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dafed04-7d9e-40e7-b28e-42b4607598ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfe0871e-c8p7f\" (UID: \"0dafed04-7d9e-40e7-b28e-42b4607598ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f" Oct 30 16:40:17.786645 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.786482 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dafed04-7d9e-40e7-b28e-42b4607598ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfe0871e-c8p7f\" (UID: \"0dafed04-7d9e-40e7-b28e-42b4607598ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f" Oct 30 16:40:17.786645 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.786592 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz955\" (UniqueName: \"kubernetes.io/projected/0dafed04-7d9e-40e7-b28e-42b4607598ba-kube-api-access-nz955\") pod \"maas-api-revoke-sub-dfe0871e-c8p7f\" (UID: \"0dafed04-7d9e-40e7-b28e-42b4607598ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f" Oct 30 16:40:17.787311 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.787286 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dafed04-7d9e-40e7-b28e-42b4607598ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfe0871e-c8p7f\" (UID: \"0dafed04-7d9e-40e7-b28e-42b4607598ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f" Oct 30 16:40:17.789015 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.788997 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dafed04-7d9e-40e7-b28e-42b4607598ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfe0871e-c8p7f\" (UID: \"0dafed04-7d9e-40e7-b28e-42b4607598ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f" Oct 30 16:40:17.810437 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.810398 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz955\" (UniqueName: \"kubernetes.io/projected/0dafed04-7d9e-40e7-b28e-42b4607598ba-kube-api-access-nz955\") pod \"maas-api-revoke-sub-dfe0871e-c8p7f\" (UID: \"0dafed04-7d9e-40e7-b28e-42b4607598ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f" Oct 30 16:40:17.894532 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:17.894434 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f" Oct 30 16:40:18.205388 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:40:18.205357 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dafed04_7d9e_40e7_b28e_42b4607598ba.slice/crio-ff34bad9bd38f2aeba3c43bd39517c8b528aeac0dd5bcf9c3b57377a686e959f WatchSource:0}: Error finding container ff34bad9bd38f2aeba3c43bd39517c8b528aeac0dd5bcf9c3b57377a686e959f: Status 404 returned error can't find the container with id ff34bad9bd38f2aeba3c43bd39517c8b528aeac0dd5bcf9c3b57377a686e959f Oct 30 16:40:18.206183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:18.206159 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f"] Oct 30 16:40:18.649425 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:18.649381 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f" event={"ID":"0dafed04-7d9e-40e7-b28e-42b4607598ba","Type":"ContainerStarted","Data":"14f44ff63dcb1b9bd5eb58c39ce4c21acc495f2313c0ed71f6b0dfac219be44e"} Oct 30 16:40:18.649629 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:18.649444 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f" event={"ID":"0dafed04-7d9e-40e7-b28e-42b4607598ba","Type":"ContainerStarted","Data":"ff34bad9bd38f2aeba3c43bd39517c8b528aeac0dd5bcf9c3b57377a686e959f"} Oct 30 16:40:18.692039 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:18.691966 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f" podStartSLOduration=1.691946704 podStartE2EDuration="1.691946704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:18.689160649 +0000 UTC m=+1132.399615475" watchObservedRunningTime="2026-10-30 16:40:18.691946704 +0000 UTC m=+1132.402401439" Oct 30 16:40:18.961039 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:18.960991 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:19.363530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.363499 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n"] Oct 30 16:40:19.368157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.368138 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.392239 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.392213 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n"] Oct 30 16:40:19.397524 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.397500 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 16:40:19.397672 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.397570 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 16:40:19.397672 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.397601 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4j6q4\"" Oct 30 16:40:19.397672 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.397611 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 16:40:19.413252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.413221 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.413403 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.413265 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.413403 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.413309 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.413528 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.413469 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.413528 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.413496 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.413632 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.413548 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m67z\" (UniqueName: \"kubernetes.io/projected/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-kube-api-access-5m67z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.413975 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.413938 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.515956 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.515852 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.515956 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.515927 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.516149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.515966 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.516149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.516011 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.516227 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.516171 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.516227 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.516196 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.516324 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.516244 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m67z\" (UniqueName: \"kubernetes.io/projected/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-kube-api-access-5m67z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.517088 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.517054 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.519114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.519089 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.519276 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.519204 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.519276 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.519212 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.519520 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.519498 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.544662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.544632 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.544932 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.544914 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m67z\" (UniqueName: \"kubernetes.io/projected/f39a5526-1dde-4513-a5ad-0c29de3b0fd0-kube-api-access-5m67z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hz72n\" (UID: \"f39a5526-1dde-4513-a5ad-0c29de3b0fd0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:19.659628 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.659592 6202 generic.go:356] "Generic (PLEG): container finished" podID="0dafed04-7d9e-40e7-b28e-42b4607598ba" containerID="14f44ff63dcb1b9bd5eb58c39ce4c21acc495f2313c0ed71f6b0dfac219be44e" exitCode=0 Oct 30 16:40:19.659825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.659673 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f" event={"ID":"0dafed04-7d9e-40e7-b28e-42b4607598ba","Type":"ContainerDied","Data":"14f44ff63dcb1b9bd5eb58c39ce4c21acc495f2313c0ed71f6b0dfac219be44e"} Oct 30 16:40:19.681747 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:19.681717 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:20.003354 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:40:20.003327 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39a5526_1dde_4513_a5ad_0c29de3b0fd0.slice/crio-e693111f8e25b21f8a0ec84649596c9bf0b81a0379da3a33eb65c9ad8a371f1c WatchSource:0}: Error finding container e693111f8e25b21f8a0ec84649596c9bf0b81a0379da3a33eb65c9ad8a371f1c: Status 404 returned error can't find the container with id e693111f8e25b21f8a0ec84649596c9bf0b81a0379da3a33eb65c9ad8a371f1c Oct 30 16:40:20.005556 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:20.005538 6202 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:40:20.006610 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:20.006585 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n"] Oct 30 16:40:20.669688 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:20.669648 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" event={"ID":"f39a5526-1dde-4513-a5ad-0c29de3b0fd0","Type":"ContainerStarted","Data":"e693111f8e25b21f8a0ec84649596c9bf0b81a0379da3a33eb65c9ad8a371f1c"} Oct 30 16:40:20.674493 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:20.674470 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:20.893148 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:20.893121 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f" Oct 30 16:40:20.937105 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:20.937025 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dafed04-7d9e-40e7-b28e-42b4607598ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dafed04-7d9e-40e7-b28e-42b4607598ba-maas-api-openshift-ca\") pod \"0dafed04-7d9e-40e7-b28e-42b4607598ba\" (UID: \"0dafed04-7d9e-40e7-b28e-42b4607598ba\") " Oct 30 16:40:20.937252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:20.937168 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dafed04-7d9e-40e7-b28e-42b4607598ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dafed04-7d9e-40e7-b28e-42b4607598ba-maas-api-xks-ca\") pod \"0dafed04-7d9e-40e7-b28e-42b4607598ba\" (UID: \"0dafed04-7d9e-40e7-b28e-42b4607598ba\") " Oct 30 16:40:20.937318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:20.937303 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dafed04-7d9e-40e7-b28e-42b4607598ba-kube-api-access-nz955\" (UniqueName: \"kubernetes.io/projected/0dafed04-7d9e-40e7-b28e-42b4607598ba-kube-api-access-nz955\") pod \"0dafed04-7d9e-40e7-b28e-42b4607598ba\" (UID: \"0dafed04-7d9e-40e7-b28e-42b4607598ba\") " Oct 30 16:40:20.941171 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:20.941140 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dafed04-7d9e-40e7-b28e-42b4607598ba-maas-api-openshift-ca" pod "0dafed04-7d9e-40e7-b28e-42b4607598ba" (UID: "0dafed04-7d9e-40e7-b28e-42b4607598ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:40:20.947528 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:20.947231 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dafed04-7d9e-40e7-b28e-42b4607598ba-maas-api-xks-ca" pod "0dafed04-7d9e-40e7-b28e-42b4607598ba" (UID: "0dafed04-7d9e-40e7-b28e-42b4607598ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:40:20.947663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:20.947623 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dafed04-7d9e-40e7-b28e-42b4607598ba-kube-api-access-nz955" pod "0dafed04-7d9e-40e7-b28e-42b4607598ba" (UID: "0dafed04-7d9e-40e7-b28e-42b4607598ba"). InnerVolumeSpecName "kube-api-access-nz955". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:21.039407 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:21.039369 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dafed04-7d9e-40e7-b28e-42b4607598ba-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:21.039407 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:21.039407 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dafed04-7d9e-40e7-b28e-42b4607598ba-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:21.039599 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:21.039422 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz955\" (UniqueName: \"kubernetes.io/projected/0dafed04-7d9e-40e7-b28e-42b4607598ba-kube-api-access-nz955\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:21.684167 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:21.684122 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f" event={"ID":"0dafed04-7d9e-40e7-b28e-42b4607598ba","Type":"ContainerDied","Data":"ff34bad9bd38f2aeba3c43bd39517c8b528aeac0dd5bcf9c3b57377a686e959f"} Oct 30 16:40:21.684624 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:21.684225 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff34bad9bd38f2aeba3c43bd39517c8b528aeac0dd5bcf9c3b57377a686e959f" Oct 30 16:40:21.684624 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:21.684294 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f" Oct 30 16:40:21.814349 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:21.814304 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:22.838160 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:22.838127 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:22.925630 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:22.922552 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:40:22.925630 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:22.922641 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:40:23.712893 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:23.712641 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" event={"ID":"f39a5526-1dde-4513-a5ad-0c29de3b0fd0","Type":"ContainerStarted","Data":"ec03a89435d7af030beff1ce82d65e918b3a1d0428457b6dbd364de2699d8d17"} Oct 30 16:40:23.712893 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:23.712819 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:23.795029 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:23.794911 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" podStartSLOduration=1.8831183679999999 podStartE2EDuration="4.794890124s" totalImagesPullingTime="2.911771756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:23.792310409 +0000 UTC m=+1137.502765154" watchObservedRunningTime="2026-10-30 16:40:23.794890124 +0000 UTC m=+1137.505344870" Oct 30 16:40:24.700570 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:24.700539 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:24.724539 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:24.724498 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hz72n" Oct 30 16:40:25.212279 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.212249 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv"] Oct 30 16:40:25.212983 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.212969 6202 cpu_manager.go:414] "containerMap: removing container" podUID="0dafed04-7d9e-40e7-b28e-42b4607598ba" containerName="revoke-keys" Oct 30 16:40:25.213032 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.212985 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dafed04-7d9e-40e7-b28e-42b4607598ba" containerName="revoke-keys" Oct 30 16:40:25.213659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.213631 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dafed04-7d9e-40e7-b28e-42b4607598ba" containerName="revoke-keys" Oct 30 16:40:25.223885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.223856 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv" Oct 30 16:40:25.239054 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.238441 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv"] Oct 30 16:40:25.243808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.240613 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:40:25.243808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.240721 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:40:25.243808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.240758 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:40:25.243808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.240901 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:40:25.324241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.324209 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ddb302-59b5-476e-a04a-5c2186ff2581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ee42639-htrzv\" (UID: \"56ddb302-59b5-476e-a04a-5c2186ff2581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv" Oct 30 16:40:25.324410 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.324258 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb58x\" (UniqueName: \"kubernetes.io/projected/56ddb302-59b5-476e-a04a-5c2186ff2581-kube-api-access-qb58x\") pod \"maas-api-revoke-sub-5ee42639-htrzv\" (UID: \"56ddb302-59b5-476e-a04a-5c2186ff2581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv" Oct 30 16:40:25.324410 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.324376 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ddb302-59b5-476e-a04a-5c2186ff2581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ee42639-htrzv\" (UID: \"56ddb302-59b5-476e-a04a-5c2186ff2581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv" Oct 30 16:40:25.425284 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.425255 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ddb302-59b5-476e-a04a-5c2186ff2581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ee42639-htrzv\" (UID: \"56ddb302-59b5-476e-a04a-5c2186ff2581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv" Oct 30 16:40:25.425450 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.425306 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb58x\" (UniqueName: \"kubernetes.io/projected/56ddb302-59b5-476e-a04a-5c2186ff2581-kube-api-access-qb58x\") pod \"maas-api-revoke-sub-5ee42639-htrzv\" (UID: \"56ddb302-59b5-476e-a04a-5c2186ff2581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv" Oct 30 16:40:25.425450 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.425354 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ddb302-59b5-476e-a04a-5c2186ff2581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ee42639-htrzv\" (UID: \"56ddb302-59b5-476e-a04a-5c2186ff2581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv" Oct 30 16:40:25.425918 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.425896 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ddb302-59b5-476e-a04a-5c2186ff2581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ee42639-htrzv\" (UID: \"56ddb302-59b5-476e-a04a-5c2186ff2581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv" Oct 30 16:40:25.427741 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.427724 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ddb302-59b5-476e-a04a-5c2186ff2581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ee42639-htrzv\" (UID: \"56ddb302-59b5-476e-a04a-5c2186ff2581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv" Oct 30 16:40:25.448691 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.448661 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb58x\" (UniqueName: \"kubernetes.io/projected/56ddb302-59b5-476e-a04a-5c2186ff2581-kube-api-access-qb58x\") pod \"maas-api-revoke-sub-5ee42639-htrzv\" (UID: \"56ddb302-59b5-476e-a04a-5c2186ff2581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv" Oct 30 16:40:25.539647 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.539563 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv" Oct 30 16:40:25.846987 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:25.846959 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv"] Oct 30 16:40:26.557328 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:26.557255 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:26.739496 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:26.739451 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv" event={"ID":"56ddb302-59b5-476e-a04a-5c2186ff2581","Type":"ContainerStarted","Data":"f20f4cd1b0ee8ec9c4a36857a7a0cff8802a2f702f5d6032456111ef299ac981"} Oct 30 16:40:26.739682 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:26.739517 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv" event={"ID":"56ddb302-59b5-476e-a04a-5c2186ff2581","Type":"ContainerStarted","Data":"de960350aa3930eaa19e6e98c942ffc4846585c6f4b58aa8ddf9f3803c173192"} Oct 30 16:40:26.785095 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:26.785044 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv" podStartSLOduration=1.785030156 podStartE2EDuration="1.785030156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:26.779382835 +0000 UTC m=+1140.489837578" watchObservedRunningTime="2026-10-30 16:40:26.785030156 +0000 UTC m=+1140.495484901" Oct 30 16:40:27.074794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:27.073945 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88"] Oct 30 16:40:27.146791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:27.136341 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322ea210-dcc88"] Oct 30 16:40:27.976056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:27.976014 6202 generic.go:356] "Generic (PLEG): container finished" podID="56ddb302-59b5-476e-a04a-5c2186ff2581" containerID="f20f4cd1b0ee8ec9c4a36857a7a0cff8802a2f702f5d6032456111ef299ac981" exitCode=0 Oct 30 16:40:27.976536 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:27.976510 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv" event={"ID":"56ddb302-59b5-476e-a04a-5c2186ff2581","Type":"ContainerDied","Data":"f20f4cd1b0ee8ec9c4a36857a7a0cff8802a2f702f5d6032456111ef299ac981"} Oct 30 16:40:28.582796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:28.578134 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="092b952e-c47c-4ee9-9af6-29aca8023e0a" path="/var/lib/kubelet/pods/092b952e-c47c-4ee9-9af6-29aca8023e0a/volumes" Oct 30 16:40:28.742803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:28.738277 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 16:40:29.043797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:29.042871 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:29.043797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:29.043239 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 16:40:29.125924 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:29.125876 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 16:40:29.148984 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:29.148915 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:29.160935 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:29.160855 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:40:29.456955 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:29.456914 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:40:29.457871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:29.457839 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" podUID="c3f813c4-823b-4109-a2e4-6027d5fabf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:29.747183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:29.739931 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:29.747183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:29.740312 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:40:29.839295 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:29.803332 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:29.839295 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:29.832298 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:29.839295 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:29.836910 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:30.063088 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.047519 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:30.063088 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.048562 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:30.063088 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.049749 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:40:30.063088 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.049830 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:40:30.179362 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.170867 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:40:30.179362 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.170980 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 16:40:30.193855 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.185476 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:30.262240 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.257840 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:40:30.300265 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.288094 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:40:30.300265 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.288152 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:40:30.322657 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.322628 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv" Oct 30 16:40:30.461212 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.455714 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:40:30.540799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.501144 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56ddb302-59b5-476e-a04a-5c2186ff2581-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ddb302-59b5-476e-a04a-5c2186ff2581-maas-api-openshift-ca\") pod \"56ddb302-59b5-476e-a04a-5c2186ff2581\" (UID: \"56ddb302-59b5-476e-a04a-5c2186ff2581\") " Oct 30 16:40:30.551291 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.551160 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56ddb302-59b5-476e-a04a-5c2186ff2581-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ddb302-59b5-476e-a04a-5c2186ff2581-maas-api-xks-ca\") pod \"56ddb302-59b5-476e-a04a-5c2186ff2581\" (UID: \"56ddb302-59b5-476e-a04a-5c2186ff2581\") " Oct 30 16:40:30.551536 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.551516 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ddb302-59b5-476e-a04a-5c2186ff2581-kube-api-access-qb58x\" (UniqueName: \"kubernetes.io/projected/56ddb302-59b5-476e-a04a-5c2186ff2581-kube-api-access-qb58x\") pod \"56ddb302-59b5-476e-a04a-5c2186ff2581\" (UID: \"56ddb302-59b5-476e-a04a-5c2186ff2581\") " Oct 30 16:40:30.552977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.505201 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ddb302-59b5-476e-a04a-5c2186ff2581-maas-api-openshift-ca" pod "56ddb302-59b5-476e-a04a-5c2186ff2581" (UID: "56ddb302-59b5-476e-a04a-5c2186ff2581"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:40:30.558433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.556865 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded" Oct 30 16:40:30.570812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.570788 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:40:30.571074 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.571057 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:40:30.575356 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.575330 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ddb302-59b5-476e-a04a-5c2186ff2581-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:30.602445 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.602414 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56ddb302-59b5-476e-a04a-5c2186ff2581-maas-api-xks-ca" pod "56ddb302-59b5-476e-a04a-5c2186ff2581" (UID: "56ddb302-59b5-476e-a04a-5c2186ff2581"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:40:30.618934 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.618900 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ddb302-59b5-476e-a04a-5c2186ff2581-kube-api-access-qb58x" pod "56ddb302-59b5-476e-a04a-5c2186ff2581" (UID: "56ddb302-59b5-476e-a04a-5c2186ff2581"). InnerVolumeSpecName "kube-api-access-qb58x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:30.674293 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.674265 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:40:30.684146 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.684119 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:40:30.692817 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.690826 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ddb302-59b5-476e-a04a-5c2186ff2581-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:30.692817 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.690856 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb58x\" (UniqueName: \"kubernetes.io/projected/56ddb302-59b5-476e-a04a-5c2186ff2581-kube-api-access-qb58x\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:30.696383 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:30.695949 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:40:31.327295 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:31.327267 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv" Oct 30 16:40:31.327295 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:31.327287 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv" event={"ID":"56ddb302-59b5-476e-a04a-5c2186ff2581","Type":"ContainerDied","Data":"de960350aa3930eaa19e6e98c942ffc4846585c6f4b58aa8ddf9f3803c173192"} Oct 30 16:40:31.327842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:31.327363 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de960350aa3930eaa19e6e98c942ffc4846585c6f4b58aa8ddf9f3803c173192" Oct 30 16:40:31.975958 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:31.975916 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:32.411820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.411790 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7"] Oct 30 16:40:32.412438 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.412415 6202 cpu_manager.go:414] "containerMap: removing container" podUID="56ddb302-59b5-476e-a04a-5c2186ff2581" containerName="revoke-keys" Oct 30 16:40:32.412438 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.412435 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56ddb302-59b5-476e-a04a-5c2186ff2581" containerName="revoke-keys" Oct 30 16:40:32.412595 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.412555 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="56ddb302-59b5-476e-a04a-5c2186ff2581" containerName="revoke-keys" Oct 30 16:40:32.416621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.416602 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7" Oct 30 16:40:32.419825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.419796 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416dd70d-f8a3-453f-b00b-7cd422ae3768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6828c9dc-q6pt7\" (UID: \"416dd70d-f8a3-453f-b00b-7cd422ae3768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7" Oct 30 16:40:32.420088 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.420067 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416dd70d-f8a3-453f-b00b-7cd422ae3768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6828c9dc-q6pt7\" (UID: \"416dd70d-f8a3-453f-b00b-7cd422ae3768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7" Oct 30 16:40:32.420195 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.420103 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch5cg\" (UniqueName: \"kubernetes.io/projected/416dd70d-f8a3-453f-b00b-7cd422ae3768-kube-api-access-ch5cg\") pod \"maas-api-revoke-sub-6828c9dc-q6pt7\" (UID: \"416dd70d-f8a3-453f-b00b-7cd422ae3768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7" Oct 30 16:40:32.431245 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.431220 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:40:32.431579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.431554 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7"] Oct 30 16:40:32.436972 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.436948 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:40:32.437153 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.436977 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:40:32.442759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.442733 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:40:32.521406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.521372 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416dd70d-f8a3-453f-b00b-7cd422ae3768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6828c9dc-q6pt7\" (UID: \"416dd70d-f8a3-453f-b00b-7cd422ae3768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7" Oct 30 16:40:32.521406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.521411 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch5cg\" (UniqueName: \"kubernetes.io/projected/416dd70d-f8a3-453f-b00b-7cd422ae3768-kube-api-access-ch5cg\") pod \"maas-api-revoke-sub-6828c9dc-q6pt7\" (UID: \"416dd70d-f8a3-453f-b00b-7cd422ae3768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7" Oct 30 16:40:32.521640 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.521563 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416dd70d-f8a3-453f-b00b-7cd422ae3768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6828c9dc-q6pt7\" (UID: \"416dd70d-f8a3-453f-b00b-7cd422ae3768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7" Oct 30 16:40:32.522104 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.522082 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416dd70d-f8a3-453f-b00b-7cd422ae3768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6828c9dc-q6pt7\" (UID: \"416dd70d-f8a3-453f-b00b-7cd422ae3768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7" Oct 30 16:40:32.524052 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.524032 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416dd70d-f8a3-453f-b00b-7cd422ae3768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6828c9dc-q6pt7\" (UID: \"416dd70d-f8a3-453f-b00b-7cd422ae3768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7" Oct 30 16:40:32.547956 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.547921 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch5cg\" (UniqueName: \"kubernetes.io/projected/416dd70d-f8a3-453f-b00b-7cd422ae3768-kube-api-access-ch5cg\") pod \"maas-api-revoke-sub-6828c9dc-q6pt7\" (UID: \"416dd70d-f8a3-453f-b00b-7cd422ae3768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7" Oct 30 16:40:32.731853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:32.731752 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7" Oct 30 16:40:33.034469 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:40:33.034427 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416dd70d_f8a3_453f_b00b_7cd422ae3768.slice/crio-a17278bbb983baa62a24086e2e8b9086744ee98c0ca1eeef1f0313a9cc097d4a WatchSource:0}: Error finding container a17278bbb983baa62a24086e2e8b9086744ee98c0ca1eeef1f0313a9cc097d4a: Status 404 returned error can't find the container with id a17278bbb983baa62a24086e2e8b9086744ee98c0ca1eeef1f0313a9cc097d4a Oct 30 16:40:33.036963 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:33.036939 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7"] Oct 30 16:40:33.346148 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:33.346111 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7" event={"ID":"416dd70d-f8a3-453f-b00b-7cd422ae3768","Type":"ContainerStarted","Data":"a17278bbb983baa62a24086e2e8b9086744ee98c0ca1eeef1f0313a9cc097d4a"} Oct 30 16:40:33.580484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:33.580445 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:34.360467 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:34.360433 6202 generic.go:356] "Generic (PLEG): container finished" podID="416dd70d-f8a3-453f-b00b-7cd422ae3768" containerID="1528e918cf4a3df4b1568c2d527e868cd69aac8860e79fbf369c9cc856eeb572" exitCode=0 Oct 30 16:40:34.360654 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:34.360520 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7" event={"ID":"416dd70d-f8a3-453f-b00b-7cd422ae3768","Type":"ContainerDied","Data":"1528e918cf4a3df4b1568c2d527e868cd69aac8860e79fbf369c9cc856eeb572"} Oct 30 16:40:35.111205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:35.111173 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:35.539728 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:35.539702 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7" Oct 30 16:40:35.568315 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:35.568286 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416dd70d-f8a3-453f-b00b-7cd422ae3768-kube-api-access-ch5cg\" (UniqueName: \"kubernetes.io/projected/416dd70d-f8a3-453f-b00b-7cd422ae3768-kube-api-access-ch5cg\") pod \"416dd70d-f8a3-453f-b00b-7cd422ae3768\" (UID: \"416dd70d-f8a3-453f-b00b-7cd422ae3768\") " Oct 30 16:40:35.568476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:35.568338 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/416dd70d-f8a3-453f-b00b-7cd422ae3768-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416dd70d-f8a3-453f-b00b-7cd422ae3768-maas-api-xks-ca\") pod \"416dd70d-f8a3-453f-b00b-7cd422ae3768\" (UID: \"416dd70d-f8a3-453f-b00b-7cd422ae3768\") " Oct 30 16:40:35.568687 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:35.568670 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416dd70d-f8a3-453f-b00b-7cd422ae3768-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416dd70d-f8a3-453f-b00b-7cd422ae3768-maas-api-openshift-ca\") pod \"416dd70d-f8a3-453f-b00b-7cd422ae3768\" (UID: \"416dd70d-f8a3-453f-b00b-7cd422ae3768\") " Oct 30 16:40:35.569026 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:35.569005 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416dd70d-f8a3-453f-b00b-7cd422ae3768-maas-api-openshift-ca" pod "416dd70d-f8a3-453f-b00b-7cd422ae3768" (UID: "416dd70d-f8a3-453f-b00b-7cd422ae3768"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:40:35.569189 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:35.569172 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416dd70d-f8a3-453f-b00b-7cd422ae3768-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:35.571201 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:35.571170 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416dd70d-f8a3-453f-b00b-7cd422ae3768-kube-api-access-ch5cg" pod "416dd70d-f8a3-453f-b00b-7cd422ae3768" (UID: "416dd70d-f8a3-453f-b00b-7cd422ae3768"). InnerVolumeSpecName "kube-api-access-ch5cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:35.571305 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:35.571203 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416dd70d-f8a3-453f-b00b-7cd422ae3768-maas-api-xks-ca" pod "416dd70d-f8a3-453f-b00b-7cd422ae3768" (UID: "416dd70d-f8a3-453f-b00b-7cd422ae3768"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:40:35.670657 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:35.670582 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch5cg\" (UniqueName: \"kubernetes.io/projected/416dd70d-f8a3-453f-b00b-7cd422ae3768-kube-api-access-ch5cg\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:35.670657 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:35.670614 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416dd70d-f8a3-453f-b00b-7cd422ae3768-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:36.383264 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:36.383234 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7" Oct 30 16:40:36.383264 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:36.383252 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7" event={"ID":"416dd70d-f8a3-453f-b00b-7cd422ae3768","Type":"ContainerDied","Data":"a17278bbb983baa62a24086e2e8b9086744ee98c0ca1eeef1f0313a9cc097d4a"} Oct 30 16:40:36.383721 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:36.383296 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a17278bbb983baa62a24086e2e8b9086744ee98c0ca1eeef1f0313a9cc097d4a" Oct 30 16:40:36.645127 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:36.645038 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:36.914375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:36.914291 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:37.361420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:37.361389 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:38.850987 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:38.850957 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:41.582035 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:41.581986 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:41.582961 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:41.582931 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:40:41.679966 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:41.679907 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:40:41.686886 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:41.686850 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:41.705909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:41.705872 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 16:40:41.848019 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:41.847900 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:41.965790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:41.964861 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded" Oct 30 16:40:42.101917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:42.101757 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:42.308917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:42.308872 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:42.438180 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:42.437895 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:42.452464 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:42.452423 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:42.596977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:42.595048 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:42.596977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:42.595409 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 16:40:42.596977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:42.596011 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:40:42.596977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:42.596069 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:40:42.737845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:42.724533 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:42.743189 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:42.738192 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:40:42.743189 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:42.738573 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 16:40:42.743189 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:42.739591 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:40:42.743189 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:42.739647 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:40:42.743189 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:42.739685 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:40:42.838924 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:42.838878 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:43.104781 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:43.104723 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:43.106552 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:43.106531 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:40:43.208278 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:43.208231 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:40:43.208534 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:43.208497 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" podUID="ec24f992-d313-4d6d-987d-98c4c8361df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:43.233368 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:43.233326 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:40:43.248618 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:43.248590 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:40:43.460848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:43.454166 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:43.460848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:43.455624 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:40:43.628975 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:43.609203 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 16:40:43.628975 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:43.610367 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:40:43.766172 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:43.764891 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:43.848791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:43.848183 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:40:44.005790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:44.004922 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:40:44.096923 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:44.096894 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:40:44.105122 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:44.105100 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:40:44.124448 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:44.124300 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:40:44.138366 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:40:44.138330 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:40:44.545833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:44.545719 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:45.221321 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.221285 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:45.657888 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.657857 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4"] Oct 30 16:40:45.659221 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.659198 6202 cpu_manager.go:414] "containerMap: removing container" podUID="416dd70d-f8a3-453f-b00b-7cd422ae3768" containerName="revoke-keys" Oct 30 16:40:45.659221 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.659218 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416dd70d-f8a3-453f-b00b-7cd422ae3768" containerName="revoke-keys" Oct 30 16:40:45.659424 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.659363 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="416dd70d-f8a3-453f-b00b-7cd422ae3768" containerName="revoke-keys" Oct 30 16:40:45.663947 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.663921 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4" Oct 30 16:40:45.683095 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.683073 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4"] Oct 30 16:40:45.689571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.689327 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:40:45.689571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.689461 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:40:45.689571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.689534 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:40:45.690541 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.690509 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:40:45.783661 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.783622 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g4q9\" (UniqueName: \"kubernetes.io/projected/938846c6-098f-493f-834f-d3c77839aece-kube-api-access-2g4q9\") pod \"maas-api-revoke-sub-83c165f8-b44h4\" (UID: \"938846c6-098f-493f-834f-d3c77839aece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4" Oct 30 16:40:45.783862 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.783813 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938846c6-098f-493f-834f-d3c77839aece-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83c165f8-b44h4\" (UID: \"938846c6-098f-493f-834f-d3c77839aece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4" Oct 30 16:40:45.783862 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.783858 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938846c6-098f-493f-834f-d3c77839aece-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83c165f8-b44h4\" (UID: \"938846c6-098f-493f-834f-d3c77839aece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4" Oct 30 16:40:45.885602 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.885562 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g4q9\" (UniqueName: \"kubernetes.io/projected/938846c6-098f-493f-834f-d3c77839aece-kube-api-access-2g4q9\") pod \"maas-api-revoke-sub-83c165f8-b44h4\" (UID: \"938846c6-098f-493f-834f-d3c77839aece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4" Oct 30 16:40:45.885797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.885703 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938846c6-098f-493f-834f-d3c77839aece-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83c165f8-b44h4\" (UID: \"938846c6-098f-493f-834f-d3c77839aece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4" Oct 30 16:40:45.885797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.885757 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938846c6-098f-493f-834f-d3c77839aece-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83c165f8-b44h4\" (UID: \"938846c6-098f-493f-834f-d3c77839aece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4" Oct 30 16:40:45.886717 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.886684 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938846c6-098f-493f-834f-d3c77839aece-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83c165f8-b44h4\" (UID: \"938846c6-098f-493f-834f-d3c77839aece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4" Oct 30 16:40:45.888224 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.888200 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938846c6-098f-493f-834f-d3c77839aece-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83c165f8-b44h4\" (UID: \"938846c6-098f-493f-834f-d3c77839aece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4" Oct 30 16:40:45.908898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.908838 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g4q9\" (UniqueName: \"kubernetes.io/projected/938846c6-098f-493f-834f-d3c77839aece-kube-api-access-2g4q9\") pod \"maas-api-revoke-sub-83c165f8-b44h4\" (UID: \"938846c6-098f-493f-834f-d3c77839aece\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4" Oct 30 16:40:45.977930 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:45.977905 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4" Oct 30 16:40:46.289441 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:40:46.289397 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938846c6_098f_493f_834f_d3c77839aece.slice/crio-71db822b6cf33ace4b1f490041905de1c7fb76f36826b9996f6903c5faeb36c1 WatchSource:0}: Error finding container 71db822b6cf33ace4b1f490041905de1c7fb76f36826b9996f6903c5faeb36c1: Status 404 returned error can't find the container with id 71db822b6cf33ace4b1f490041905de1c7fb76f36826b9996f6903c5faeb36c1 Oct 30 16:40:46.289814 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:46.289670 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4"] Oct 30 16:40:46.801062 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:46.800971 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j"] Oct 30 16:40:46.805654 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:46.805634 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j" Oct 30 16:40:46.826644 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:46.826611 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j"] Oct 30 16:40:46.897170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:46.897129 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bddc700-6xc6j\" (UID: \"22f9d3a6-5911-45b0-84fb-12b6ad4274f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j" Oct 30 16:40:46.897363 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:46.897343 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpkmw\" (UniqueName: \"kubernetes.io/projected/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-kube-api-access-lpkmw\") pod \"maas-api-revoke-sub-3bddc700-6xc6j\" (UID: \"22f9d3a6-5911-45b0-84fb-12b6ad4274f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j" Oct 30 16:40:46.897429 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:46.897401 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bddc700-6xc6j\" (UID: \"22f9d3a6-5911-45b0-84fb-12b6ad4274f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j" Oct 30 16:40:46.999029 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:46.998984 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bddc700-6xc6j\" (UID: \"22f9d3a6-5911-45b0-84fb-12b6ad4274f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j" Oct 30 16:40:46.999236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:46.999170 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpkmw\" (UniqueName: \"kubernetes.io/projected/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-kube-api-access-lpkmw\") pod \"maas-api-revoke-sub-3bddc700-6xc6j\" (UID: \"22f9d3a6-5911-45b0-84fb-12b6ad4274f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j" Oct 30 16:40:46.999311 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:46.999234 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bddc700-6xc6j\" (UID: \"22f9d3a6-5911-45b0-84fb-12b6ad4274f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j" Oct 30 16:40:47.000183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:47.000153 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bddc700-6xc6j\" (UID: \"22f9d3a6-5911-45b0-84fb-12b6ad4274f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j" Oct 30 16:40:47.002146 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:47.002121 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bddc700-6xc6j\" (UID: \"22f9d3a6-5911-45b0-84fb-12b6ad4274f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j" Oct 30 16:40:47.022719 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:47.022688 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpkmw\" (UniqueName: \"kubernetes.io/projected/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-kube-api-access-lpkmw\") pod \"maas-api-revoke-sub-3bddc700-6xc6j\" (UID: \"22f9d3a6-5911-45b0-84fb-12b6ad4274f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j" Oct 30 16:40:47.067507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:47.067431 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:47.077983 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:47.077949 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4" event={"ID":"938846c6-098f-493f-834f-d3c77839aece","Type":"ContainerStarted","Data":"1f0c0eace06cd246e78882bb72d8644cede8b4758c72807b4ac4933777906de2"} Oct 30 16:40:47.078121 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:47.078013 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4" event={"ID":"938846c6-098f-493f-834f-d3c77839aece","Type":"ContainerStarted","Data":"71db822b6cf33ace4b1f490041905de1c7fb76f36826b9996f6903c5faeb36c1"} Oct 30 16:40:47.124103 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:47.124039 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4" podStartSLOduration=2.124016007 podStartE2EDuration="2.124016007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:47.119803733 +0000 UTC m=+1160.830258469" watchObservedRunningTime="2026-10-30 16:40:47.124016007 +0000 UTC m=+1160.834470755" Oct 30 16:40:47.126077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:47.125550 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j" Oct 30 16:40:47.457616 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:47.457581 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j"] Oct 30 16:40:48.088230 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:48.088185 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j" event={"ID":"22f9d3a6-5911-45b0-84fb-12b6ad4274f0","Type":"ContainerStarted","Data":"c74861b6080b8cc6f504716993a7a8b990784fc9eb767fdc8fd887d6de13d906"} Oct 30 16:40:48.088422 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:48.088255 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j" event={"ID":"22f9d3a6-5911-45b0-84fb-12b6ad4274f0","Type":"ContainerStarted","Data":"0e1e175a8d4289f0ec987712ac1a814efd168f945e65393e5dfd47f773f95552"} Oct 30 16:40:48.089889 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:48.089862 6202 generic.go:356] "Generic (PLEG): container finished" podID="938846c6-098f-493f-834f-d3c77839aece" containerID="1f0c0eace06cd246e78882bb72d8644cede8b4758c72807b4ac4933777906de2" exitCode=0 Oct 30 16:40:48.090003 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:48.089947 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4" event={"ID":"938846c6-098f-493f-834f-d3c77839aece","Type":"ContainerDied","Data":"1f0c0eace06cd246e78882bb72d8644cede8b4758c72807b4ac4933777906de2"} Oct 30 16:40:48.134575 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:48.134522 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j" podStartSLOduration=2.134507335 podStartE2EDuration="2.134507335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:48.129504035 +0000 UTC m=+1161.839958790" watchObservedRunningTime="2026-10-30 16:40:48.134507335 +0000 UTC m=+1161.844962119" Oct 30 16:40:48.663653 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:48.663542 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:49.100953 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:49.100920 6202 generic.go:356] "Generic (PLEG): container finished" podID="22f9d3a6-5911-45b0-84fb-12b6ad4274f0" containerID="c74861b6080b8cc6f504716993a7a8b990784fc9eb767fdc8fd887d6de13d906" exitCode=0 Oct 30 16:40:49.101150 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:49.100996 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j" event={"ID":"22f9d3a6-5911-45b0-84fb-12b6ad4274f0","Type":"ContainerDied","Data":"c74861b6080b8cc6f504716993a7a8b990784fc9eb767fdc8fd887d6de13d906"} Oct 30 16:40:49.304001 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:49.303972 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4" Oct 30 16:40:49.434149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:49.434069 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/938846c6-098f-493f-834f-d3c77839aece-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938846c6-098f-493f-834f-d3c77839aece-maas-api-xks-ca\") pod \"938846c6-098f-493f-834f-d3c77839aece\" (UID: \"938846c6-098f-493f-834f-d3c77839aece\") " Oct 30 16:40:49.434290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:49.434205 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938846c6-098f-493f-834f-d3c77839aece-kube-api-access-2g4q9\" (UniqueName: \"kubernetes.io/projected/938846c6-098f-493f-834f-d3c77839aece-kube-api-access-2g4q9\") pod \"938846c6-098f-493f-834f-d3c77839aece\" (UID: \"938846c6-098f-493f-834f-d3c77839aece\") " Oct 30 16:40:49.434290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:49.434240 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/938846c6-098f-493f-834f-d3c77839aece-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938846c6-098f-493f-834f-d3c77839aece-maas-api-openshift-ca\") pod \"938846c6-098f-493f-834f-d3c77839aece\" (UID: \"938846c6-098f-493f-834f-d3c77839aece\") " Oct 30 16:40:49.434642 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:49.434615 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938846c6-098f-493f-834f-d3c77839aece-maas-api-openshift-ca" pod "938846c6-098f-493f-834f-d3c77839aece" (UID: "938846c6-098f-493f-834f-d3c77839aece"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:40:49.434804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:49.434787 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938846c6-098f-493f-834f-d3c77839aece-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:49.436413 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:49.436383 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938846c6-098f-493f-834f-d3c77839aece-kube-api-access-2g4q9" pod "938846c6-098f-493f-834f-d3c77839aece" (UID: "938846c6-098f-493f-834f-d3c77839aece"). InnerVolumeSpecName "kube-api-access-2g4q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:49.436477 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:49.436407 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938846c6-098f-493f-834f-d3c77839aece-maas-api-xks-ca" pod "938846c6-098f-493f-834f-d3c77839aece" (UID: "938846c6-098f-493f-834f-d3c77839aece"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:40:49.536667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:49.536624 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938846c6-098f-493f-834f-d3c77839aece-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:49.536667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:49.536663 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g4q9\" (UniqueName: \"kubernetes.io/projected/938846c6-098f-493f-834f-d3c77839aece-kube-api-access-2g4q9\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:50.110913 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:50.110880 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4" Oct 30 16:40:50.110913 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:50.110905 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4" event={"ID":"938846c6-098f-493f-834f-d3c77839aece","Type":"ContainerDied","Data":"71db822b6cf33ace4b1f490041905de1c7fb76f36826b9996f6903c5faeb36c1"} Oct 30 16:40:50.111578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:50.110956 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71db822b6cf33ace4b1f490041905de1c7fb76f36826b9996f6903c5faeb36c1" Oct 30 16:40:50.180346 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:50.180183 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:50.327617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:50.327591 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j" Oct 30 16:40:50.447538 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:50.447456 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-maas-api-xks-ca\") pod \"22f9d3a6-5911-45b0-84fb-12b6ad4274f0\" (UID: \"22f9d3a6-5911-45b0-84fb-12b6ad4274f0\") " Oct 30 16:40:50.447686 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:50.447601 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-maas-api-openshift-ca\") pod \"22f9d3a6-5911-45b0-84fb-12b6ad4274f0\" (UID: \"22f9d3a6-5911-45b0-84fb-12b6ad4274f0\") " Oct 30 16:40:50.447739 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:50.447726 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-kube-api-access-lpkmw\" (UniqueName: \"kubernetes.io/projected/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-kube-api-access-lpkmw\") pod \"22f9d3a6-5911-45b0-84fb-12b6ad4274f0\" (UID: \"22f9d3a6-5911-45b0-84fb-12b6ad4274f0\") " Oct 30 16:40:50.448091 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:50.448058 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-maas-api-openshift-ca" pod "22f9d3a6-5911-45b0-84fb-12b6ad4274f0" (UID: "22f9d3a6-5911-45b0-84fb-12b6ad4274f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:40:50.448258 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:50.448241 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:50.450491 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:50.450461 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-maas-api-xks-ca" pod "22f9d3a6-5911-45b0-84fb-12b6ad4274f0" (UID: "22f9d3a6-5911-45b0-84fb-12b6ad4274f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:40:50.450575 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:50.450468 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-kube-api-access-lpkmw" pod "22f9d3a6-5911-45b0-84fb-12b6ad4274f0" (UID: "22f9d3a6-5911-45b0-84fb-12b6ad4274f0"). InnerVolumeSpecName "kube-api-access-lpkmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:50.549470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:50.549432 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpkmw\" (UniqueName: \"kubernetes.io/projected/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-kube-api-access-lpkmw\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:50.549470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:50.549462 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f9d3a6-5911-45b0-84fb-12b6ad4274f0-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:40:51.121260 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:51.121225 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j" Oct 30 16:40:51.121649 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:51.121259 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j" event={"ID":"22f9d3a6-5911-45b0-84fb-12b6ad4274f0","Type":"ContainerDied","Data":"0e1e175a8d4289f0ec987712ac1a814efd168f945e65393e5dfd47f773f95552"} Oct 30 16:40:51.121649 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:51.121305 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e1e175a8d4289f0ec987712ac1a814efd168f945e65393e5dfd47f773f95552" Oct 30 16:40:51.973225 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:51.973187 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:52.033298 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:52.033267 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m"] Oct 30 16:40:52.048022 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:52.047970 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b29a217e-m5s8m"] Oct 30 16:40:52.411084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:52.411052 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74e99d6c-d7a8-446b-a852-6e2c62cc0016" path="/var/lib/kubelet/pods/74e99d6c-d7a8-446b-a852-6e2c62cc0016/volumes" Oct 30 16:40:53.511665 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:53.511632 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:54.045484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:54.045440 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:54.171521 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:54.171483 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:54.763833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:54.763798 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:55.049854 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:55.049751 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl"] Oct 30 16:40:55.058757 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:55.058729 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e944294-z2pwl"] Oct 30 16:40:55.153459 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:55.153428 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:56.484792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:56.484315 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51a9d020-824f-4d23-bf54-09d28522a7f4" path="/var/lib/kubelet/pods/51a9d020-824f-4d23-bf54-09d28522a7f4/volumes" Oct 30 16:40:57.013794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.012838 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:57.101792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.100841 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:57.131793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.131489 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:57.240039 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.238912 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:40:57.603793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.603170 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks"] Oct 30 16:40:57.604300 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.604199 6202 cpu_manager.go:414] "containerMap: removing container" podUID="22f9d3a6-5911-45b0-84fb-12b6ad4274f0" containerName="revoke-keys" Oct 30 16:40:57.604300 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.604220 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f9d3a6-5911-45b0-84fb-12b6ad4274f0" containerName="revoke-keys" Oct 30 16:40:57.604300 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.604266 6202 cpu_manager.go:414] "containerMap: removing container" podUID="938846c6-098f-493f-834f-d3c77839aece" containerName="revoke-keys" Oct 30 16:40:57.604300 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.604274 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938846c6-098f-493f-834f-d3c77839aece" containerName="revoke-keys" Oct 30 16:40:57.604519 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.604453 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="938846c6-098f-493f-834f-d3c77839aece" containerName="revoke-keys" Oct 30 16:40:57.604519 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.604478 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f9d3a6-5911-45b0-84fb-12b6ad4274f0" containerName="revoke-keys" Oct 30 16:40:57.626791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.624185 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:57.632801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.629438 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks" Oct 30 16:40:57.642853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.631756 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks"] Oct 30 16:40:57.664828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.663390 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:40:57.664828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.663536 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:40:57.664828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.663605 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:40:57.690423 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.689050 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:40:57.784794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.758583 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57258ca6-00aa-4c78-a3d1-10f414b58247-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0539a2b1-4hvks\" (UID: \"57258ca6-00aa-4c78-a3d1-10f414b58247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks" Oct 30 16:40:57.813794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.789964 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57258ca6-00aa-4c78-a3d1-10f414b58247-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0539a2b1-4hvks\" (UID: \"57258ca6-00aa-4c78-a3d1-10f414b58247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks" Oct 30 16:40:57.814219 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.814183 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thvng\" (UniqueName: \"kubernetes.io/projected/57258ca6-00aa-4c78-a3d1-10f414b58247-kube-api-access-thvng\") pod \"maas-api-revoke-sub-0539a2b1-4hvks\" (UID: \"57258ca6-00aa-4c78-a3d1-10f414b58247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks" Oct 30 16:40:57.879851 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.847534 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:57.947794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.880133 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded" Oct 30 16:40:57.972794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.944214 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57258ca6-00aa-4c78-a3d1-10f414b58247-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0539a2b1-4hvks\" (UID: \"57258ca6-00aa-4c78-a3d1-10f414b58247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks" Oct 30 16:40:57.972794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.966997 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57258ca6-00aa-4c78-a3d1-10f414b58247-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0539a2b1-4hvks\" (UID: \"57258ca6-00aa-4c78-a3d1-10f414b58247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks" Oct 30 16:40:57.972794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.967143 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thvng\" (UniqueName: \"kubernetes.io/projected/57258ca6-00aa-4c78-a3d1-10f414b58247-kube-api-access-thvng\") pod \"maas-api-revoke-sub-0539a2b1-4hvks\" (UID: \"57258ca6-00aa-4c78-a3d1-10f414b58247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks" Oct 30 16:40:57.972794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.968387 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57258ca6-00aa-4c78-a3d1-10f414b58247-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0539a2b1-4hvks\" (UID: \"57258ca6-00aa-4c78-a3d1-10f414b58247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks" Oct 30 16:40:57.997929 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:57.997869 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded" Oct 30 16:40:58.029804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.015993 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 16:40:58.029804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.016546 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:40:58.029804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.025573 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thvng\" (UniqueName: \"kubernetes.io/projected/57258ca6-00aa-4c78-a3d1-10f414b58247-kube-api-access-thvng\") pod \"maas-api-revoke-sub-0539a2b1-4hvks\" (UID: \"57258ca6-00aa-4c78-a3d1-10f414b58247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks" Oct 30 16:40:58.055793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.055188 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57258ca6-00aa-4c78-a3d1-10f414b58247-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0539a2b1-4hvks\" (UID: \"57258ca6-00aa-4c78-a3d1-10f414b58247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks" Oct 30 16:40:58.106802 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.106350 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:58.112267 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.108034 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:40:58.156081 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.154920 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 16:40:58.182349 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.159230 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:40:58.296803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.258438 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks" Oct 30 16:40:58.782619 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.269433 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:40:58.783239 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.783207 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" podUID="ec24f992-d313-4d6d-987d-98c4c8361df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:58.783362 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.270558 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:58.783454 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.625053 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:58.783549 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.269474 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:40:58.923416 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.892432 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:40:58.923416 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.892505 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:40:58.953822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.944386 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:58.981819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.972438 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:40:59.010708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:58.997885 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded" Oct 30 16:40:59.010708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:59.009885 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:40:59.038450 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:59.020183 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:59.111090 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:59.109140 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 16:40:59.190577 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:59.155837 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:59.533051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:59.533002 6202 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:40:59.544208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:59.544130 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" podUID="c3f813c4-823b-4109-a2e4-6027d5fabf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:59.880165 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:59.880120 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:59.885398 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:59.881235 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:59.950008 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:59.949960 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:59.956171 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:59.950853 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:40:59.962819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:40:59.957557 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:00.076919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:00.076876 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded" Oct 30 16:41:00.077470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:00.077442 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 16:41:00.083136 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:00.081446 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:41:00.129604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:00.129570 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr"] Oct 30 16:41:00.146436 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:00.146359 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:41:00.146600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:00.146455 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:41:00.146600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:00.146504 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:41:00.146600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:00.146552 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:41:00.183784 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:00.170368 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:41:00.215839 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:00.215813 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:41:00.231248 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:00.230918 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e33859a-2kvtr"] Oct 30 16:41:00.235982 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:00.235962 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:41:00.396865 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:41:00.394799 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57258ca6_00aa_4c78_a3d1_10f414b58247.slice/crio-008e2c30bfa18b1f5af3cf4177c8c0e233eda7fab5d40201661e9d7a187a18b9 WatchSource:0}: Error finding container 008e2c30bfa18b1f5af3cf4177c8c0e233eda7fab5d40201661e9d7a187a18b9: Status 404 returned error can't find the container with id 008e2c30bfa18b1f5af3cf4177c8c0e233eda7fab5d40201661e9d7a187a18b9 Oct 30 16:41:00.396865 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:00.395139 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks"] Oct 30 16:41:00.412198 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:00.412164 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5fbbcce-bb92-4df9-abba-0901cee17ce3" path="/var/lib/kubelet/pods/b5fbbcce-bb92-4df9-abba-0901cee17ce3/volumes" Oct 30 16:41:00.744018 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:00.743979 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:01.116193 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:01.116153 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks" event={"ID":"57258ca6-00aa-4c78-a3d1-10f414b58247","Type":"ContainerStarted","Data":"403873e8f0995424f767a7814e16265b410b61fed4453fe69f96780b18f8dc9f"} Oct 30 16:41:01.116647 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:01.116218 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks" event={"ID":"57258ca6-00aa-4c78-a3d1-10f414b58247","Type":"ContainerStarted","Data":"008e2c30bfa18b1f5af3cf4177c8c0e233eda7fab5d40201661e9d7a187a18b9"} Oct 30 16:41:01.165588 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:01.165526 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks" podStartSLOduration=4.165507428 podStartE2EDuration="4.165507428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:01.158950983 +0000 UTC m=+1174.869405728" watchObservedRunningTime="2026-10-30 16:41:01.165507428 +0000 UTC m=+1174.875962172" Oct 30 16:41:01.495651 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:01.495620 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:01.605070 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:01.605033 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:01.832306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:01.832275 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:02.126881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:02.126795 6202 generic.go:356] "Generic (PLEG): container finished" podID="57258ca6-00aa-4c78-a3d1-10f414b58247" containerID="403873e8f0995424f767a7814e16265b410b61fed4453fe69f96780b18f8dc9f" exitCode=0 Oct 30 16:41:02.127247 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:02.126875 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks" event={"ID":"57258ca6-00aa-4c78-a3d1-10f414b58247","Type":"ContainerDied","Data":"403873e8f0995424f767a7814e16265b410b61fed4453fe69f96780b18f8dc9f"} Oct 30 16:41:02.176688 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:02.176657 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs"] Oct 30 16:41:02.181623 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:02.181599 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs" Oct 30 16:41:02.201490 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:02.201465 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs"] Oct 30 16:41:02.257941 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:02.257907 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6456ffb9-7rxgs\" (UID: \"9a5940bd-e36f-448b-a43d-bb2febc2b8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs" Oct 30 16:41:02.258167 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:02.258148 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79htj\" (UniqueName: \"kubernetes.io/projected/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-kube-api-access-79htj\") pod \"maas-api-revoke-sub-6456ffb9-7rxgs\" (UID: \"9a5940bd-e36f-448b-a43d-bb2febc2b8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs" Oct 30 16:41:02.258333 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:02.258313 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6456ffb9-7rxgs\" (UID: \"9a5940bd-e36f-448b-a43d-bb2febc2b8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs" Oct 30 16:41:02.341529 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:02.341487 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:02.359539 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:02.359506 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79htj\" (UniqueName: \"kubernetes.io/projected/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-kube-api-access-79htj\") pod \"maas-api-revoke-sub-6456ffb9-7rxgs\" (UID: \"9a5940bd-e36f-448b-a43d-bb2febc2b8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs" Oct 30 16:41:02.359748 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:02.359638 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6456ffb9-7rxgs\" (UID: \"9a5940bd-e36f-448b-a43d-bb2febc2b8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs" Oct 30 16:41:02.359933 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:02.359757 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6456ffb9-7rxgs\" (UID: \"9a5940bd-e36f-448b-a43d-bb2febc2b8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs" Oct 30 16:41:02.360812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:02.360786 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6456ffb9-7rxgs\" (UID: \"9a5940bd-e36f-448b-a43d-bb2febc2b8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs" Oct 30 16:41:02.362544 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:02.362507 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6456ffb9-7rxgs\" (UID: \"9a5940bd-e36f-448b-a43d-bb2febc2b8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs" Oct 30 16:41:02.383130 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:02.383059 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79htj\" (UniqueName: \"kubernetes.io/projected/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-kube-api-access-79htj\") pod \"maas-api-revoke-sub-6456ffb9-7rxgs\" (UID: \"9a5940bd-e36f-448b-a43d-bb2febc2b8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs" Oct 30 16:41:02.453888 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:02.453840 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:02.501023 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:02.500986 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs" Oct 30 16:41:02.887246 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:02.887204 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs"] Oct 30 16:41:03.136172 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:03.136139 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs" event={"ID":"9a5940bd-e36f-448b-a43d-bb2febc2b8c9","Type":"ContainerStarted","Data":"2546ed242baf7bc3ba2995cb006626c1a04acb728b8cf01656bb9f5d89240a7f"} Oct 30 16:41:03.300726 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:03.300703 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks" Oct 30 16:41:03.409662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:03.407376 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57258ca6-00aa-4c78-a3d1-10f414b58247-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57258ca6-00aa-4c78-a3d1-10f414b58247-maas-api-xks-ca\") pod \"57258ca6-00aa-4c78-a3d1-10f414b58247\" (UID: \"57258ca6-00aa-4c78-a3d1-10f414b58247\") " Oct 30 16:41:03.409662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:03.407505 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57258ca6-00aa-4c78-a3d1-10f414b58247-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57258ca6-00aa-4c78-a3d1-10f414b58247-maas-api-openshift-ca\") pod \"57258ca6-00aa-4c78-a3d1-10f414b58247\" (UID: \"57258ca6-00aa-4c78-a3d1-10f414b58247\") " Oct 30 16:41:03.409662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:03.407561 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57258ca6-00aa-4c78-a3d1-10f414b58247-kube-api-access-thvng\" (UniqueName: \"kubernetes.io/projected/57258ca6-00aa-4c78-a3d1-10f414b58247-kube-api-access-thvng\") pod \"57258ca6-00aa-4c78-a3d1-10f414b58247\" (UID: \"57258ca6-00aa-4c78-a3d1-10f414b58247\") " Oct 30 16:41:03.409662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:03.409468 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57258ca6-00aa-4c78-a3d1-10f414b58247-maas-api-openshift-ca" pod "57258ca6-00aa-4c78-a3d1-10f414b58247" (UID: "57258ca6-00aa-4c78-a3d1-10f414b58247"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:41:03.411526 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:03.411490 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57258ca6-00aa-4c78-a3d1-10f414b58247-maas-api-xks-ca" pod "57258ca6-00aa-4c78-a3d1-10f414b58247" (UID: "57258ca6-00aa-4c78-a3d1-10f414b58247"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:41:03.418753 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:03.418716 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57258ca6-00aa-4c78-a3d1-10f414b58247-kube-api-access-thvng" pod "57258ca6-00aa-4c78-a3d1-10f414b58247" (UID: "57258ca6-00aa-4c78-a3d1-10f414b58247"). InnerVolumeSpecName "kube-api-access-thvng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:41:03.509817 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:03.509651 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57258ca6-00aa-4c78-a3d1-10f414b58247-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:41:03.509817 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:03.509682 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57258ca6-00aa-4c78-a3d1-10f414b58247-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:41:03.509817 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:03.509696 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thvng\" (UniqueName: \"kubernetes.io/projected/57258ca6-00aa-4c78-a3d1-10f414b58247-kube-api-access-thvng\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:41:04.072503 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.072466 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb"] Oct 30 16:41:04.073523 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.073502 6202 cpu_manager.go:414] "containerMap: removing container" podUID="57258ca6-00aa-4c78-a3d1-10f414b58247" containerName="revoke-keys" Oct 30 16:41:04.073599 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.073527 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57258ca6-00aa-4c78-a3d1-10f414b58247" containerName="revoke-keys" Oct 30 16:41:04.073699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.073688 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="57258ca6-00aa-4c78-a3d1-10f414b58247" containerName="revoke-keys" Oct 30 16:41:04.077949 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.077909 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.109998 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.109964 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb"] Oct 30 16:41:04.117887 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.117847 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt26z\" (UniqueName: \"kubernetes.io/projected/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-kube-api-access-rt26z\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.118037 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.118006 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.118100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.118048 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.118151 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.118114 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.118151 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.118140 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.118290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.118270 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.118337 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.118306 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.118549 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.118454 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.118549 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.118523 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.168650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.168614 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks" event={"ID":"57258ca6-00aa-4c78-a3d1-10f414b58247","Type":"ContainerDied","Data":"008e2c30bfa18b1f5af3cf4177c8c0e233eda7fab5d40201661e9d7a187a18b9"} Oct 30 16:41:04.169124 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.168667 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="008e2c30bfa18b1f5af3cf4177c8c0e233eda7fab5d40201661e9d7a187a18b9" Oct 30 16:41:04.169124 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.168729 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks" Oct 30 16:41:04.171333 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.171305 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp"] Oct 30 16:41:04.173107 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.173076 6202 generic.go:356] "Generic (PLEG): container finished" podID="9a5940bd-e36f-448b-a43d-bb2febc2b8c9" containerID="49271aff4561a7679c36050a88569c00749b12279905b1c82948094c0e3ae1ae" exitCode=0 Oct 30 16:41:04.176077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.176053 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs" event={"ID":"9a5940bd-e36f-448b-a43d-bb2febc2b8c9","Type":"ContainerDied","Data":"49271aff4561a7679c36050a88569c00749b12279905b1c82948094c0e3ae1ae"} Oct 30 16:41:04.176184 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.176146 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.185800 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.185746 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:04.197992 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.197927 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp"] Oct 30 16:41:04.220736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.220697 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.221035 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.221013 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.221152 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.221134 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.222008 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.221988 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.222261 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.222233 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.228003 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.227978 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d5lm\" (UniqueName: \"kubernetes.io/projected/4951f605-1e39-41f0-8d0a-2b23791653dd-kube-api-access-7d5lm\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.259866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.239989 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.259866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.240079 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.259866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.240104 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.259866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.240155 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4951f605-1e39-41f0-8d0a-2b23791653dd-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.259866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.240193 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.259866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.240216 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.259866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.240271 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.259866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.240313 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.259866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.240400 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.259866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.240441 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt26z\" (UniqueName: \"kubernetes.io/projected/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-kube-api-access-rt26z\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.259866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.240537 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.259866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.240565 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.259866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.221886 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.259866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.227903 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.259866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.241098 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.262262 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.261913 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.264466 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.264425 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.315537 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.315504 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.316048 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.316026 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.317049 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.316891 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt26z\" (UniqueName: \"kubernetes.io/projected/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-kube-api-access-rt26z\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.317156 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.317093 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:04.318354 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.318329 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.342345 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.342320 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.342478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.342384 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4951f605-1e39-41f0-8d0a-2b23791653dd-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.342478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.342416 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.342478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.342458 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.342631 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.342535 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.342631 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.342611 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.342719 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.342686 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.342785 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.342720 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.342854 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.342798 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d5lm\" (UniqueName: \"kubernetes.io/projected/4951f605-1e39-41f0-8d0a-2b23791653dd-kube-api-access-7d5lm\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.342963 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.342939 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.343172 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.343144 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.343310 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.343292 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.343380 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.343296 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.343618 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.343597 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4951f605-1e39-41f0-8d0a-2b23791653dd-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.345384 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.345365 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.345384 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.345375 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.367079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.367042 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.367343 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.367319 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d5lm\" (UniqueName: \"kubernetes.io/projected/4951f605-1e39-41f0-8d0a-2b23791653dd-kube-api-access-7d5lm\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.396257 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.396225 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:04.493268 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.493203 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:04.505007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.504878 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:04.734757 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.734730 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb"] Oct 30 16:41:04.734896 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:41:04.734811 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcf87ea_7a55_4f30_9349_b50f3f1bf5f6.slice/crio-9419b449f9cdd53ac2de07f10c4d05a7ee979a5c70aaae6c08571c5f7e7e239e WatchSource:0}: Error finding container 9419b449f9cdd53ac2de07f10c4d05a7ee979a5c70aaae6c08571c5f7e7e239e: Status 404 returned error can't find the container with id 9419b449f9cdd53ac2de07f10c4d05a7ee979a5c70aaae6c08571c5f7e7e239e Oct 30 16:41:04.737404 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.737370 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:04.737482 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.737440 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:04.737482 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.737469 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:04.836087 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.836059 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp"] Oct 30 16:41:04.839626 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.839584 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:04.839748 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.839719 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:04.848952 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.848913 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:04.888758 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:04.888189 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:05.012993 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.012861 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:05.053133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.053103 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl"] Oct 30 16:41:05.062442 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.062375 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-639bde01-lzghl"] Oct 30 16:41:05.243097 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.243057 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" event={"ID":"4951f605-1e39-41f0-8d0a-2b23791653dd","Type":"ContainerStarted","Data":"3a3887b53d5fade58db199dcd2945e18802feea2160f6b4ace28bf945edca57d"} Oct 30 16:41:05.245158 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.245131 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" event={"ID":"4951f605-1e39-41f0-8d0a-2b23791653dd","Type":"ContainerStarted","Data":"edb430413b4f6d10a8cc4bf625d558978e7734064b74c362c4be8dd7a57495a4"} Oct 30 16:41:05.256785 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.249090 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" event={"ID":"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6","Type":"ContainerStarted","Data":"f1490f678d537ce07537b3b72cee7adcd65ddff0cc9c7996e18aa926d57fc4d9"} Oct 30 16:41:05.257039 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.257012 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" event={"ID":"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6","Type":"ContainerStarted","Data":"9419b449f9cdd53ac2de07f10c4d05a7ee979a5c70aaae6c08571c5f7e7e239e"} Oct 30 16:41:05.296220 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.294900 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" podStartSLOduration=1.294741941 podStartE2EDuration="1.294741941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:05.291964484 +0000 UTC m=+1179.002419229" watchObservedRunningTime="2026-10-30 16:41:05.294741941 +0000 UTC m=+1179.005196685" Oct 30 16:41:05.352193 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.352132 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" podStartSLOduration=1.352112245 podStartE2EDuration="1.352112245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:05.349236039 +0000 UTC m=+1179.059690784" watchObservedRunningTime="2026-10-30 16:41:05.352112245 +0000 UTC m=+1179.062566990" Oct 30 16:41:05.400400 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.400209 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:05.400565 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.400532 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 16:41:05.400659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.400577 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" podUID="ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 16:41:05.480896 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.478333 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs" Oct 30 16:41:05.504086 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.502377 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:05.504086 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.502674 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 16:41:05.504086 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.502725 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" podUID="4951f605-1e39-41f0-8d0a-2b23791653dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 16:41:05.597861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.589573 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-maas-api-openshift-ca\") pod \"9a5940bd-e36f-448b-a43d-bb2febc2b8c9\" (UID: \"9a5940bd-e36f-448b-a43d-bb2febc2b8c9\") " Oct 30 16:41:05.597861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.589623 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-maas-api-xks-ca\") pod \"9a5940bd-e36f-448b-a43d-bb2febc2b8c9\" (UID: \"9a5940bd-e36f-448b-a43d-bb2febc2b8c9\") " Oct 30 16:41:05.597861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.589662 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-kube-api-access-79htj\" (UniqueName: \"kubernetes.io/projected/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-kube-api-access-79htj\") pod \"9a5940bd-e36f-448b-a43d-bb2febc2b8c9\" (UID: \"9a5940bd-e36f-448b-a43d-bb2febc2b8c9\") " Oct 30 16:41:05.597861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.592486 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-maas-api-openshift-ca" pod "9a5940bd-e36f-448b-a43d-bb2febc2b8c9" (UID: "9a5940bd-e36f-448b-a43d-bb2febc2b8c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:41:05.606714 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.606675 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-maas-api-xks-ca" pod "9a5940bd-e36f-448b-a43d-bb2febc2b8c9" (UID: "9a5940bd-e36f-448b-a43d-bb2febc2b8c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:41:05.609263 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.609223 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-kube-api-access-79htj" pod "9a5940bd-e36f-448b-a43d-bb2febc2b8c9" (UID: "9a5940bd-e36f-448b-a43d-bb2febc2b8c9"). InnerVolumeSpecName "kube-api-access-79htj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:41:05.703213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.703168 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:41:05.703213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.703208 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:41:05.703462 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:05.703225 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79htj\" (UniqueName: \"kubernetes.io/projected/9a5940bd-e36f-448b-a43d-bb2febc2b8c9-kube-api-access-79htj\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:41:06.307391 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:06.307361 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs" Oct 30 16:41:06.307866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:06.307398 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs" event={"ID":"9a5940bd-e36f-448b-a43d-bb2febc2b8c9","Type":"ContainerDied","Data":"2546ed242baf7bc3ba2995cb006626c1a04acb728b8cf01656bb9f5d89240a7f"} Oct 30 16:41:06.307866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:06.307454 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2546ed242baf7bc3ba2995cb006626c1a04acb728b8cf01656bb9f5d89240a7f" Oct 30 16:41:06.397203 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:06.397159 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 16:41:06.397362 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:06.397229 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" podUID="ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 16:41:06.411875 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:06.411715 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd7a75d-3328-436a-8e21-8e8088de4079" path="/var/lib/kubelet/pods/cfd7a75d-3328-436a-8e21-8e8088de4079/volumes" Oct 30 16:41:06.494255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:06.494210 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 16:41:06.494467 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:06.494275 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" podUID="4951f605-1e39-41f0-8d0a-2b23791653dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 16:41:06.609925 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:06.609876 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:06.736409 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:06.736377 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:07.397385 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:07.397342 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 16:41:07.397816 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:07.397408 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" podUID="ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 16:41:07.494676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:07.494629 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 16:41:07.494863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:07.494711 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" podUID="4951f605-1e39-41f0-8d0a-2b23791653dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 16:41:08.053791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:08.043075 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:08.398231 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:08.398186 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 16:41:08.398741 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:08.398257 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" podUID="ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 16:41:08.418472 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:08.418445 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:08.494791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:08.494170 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 16:41:08.494791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:08.494245 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" podUID="4951f605-1e39-41f0-8d0a-2b23791653dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 16:41:08.760849 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:08.759964 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:09.397283 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:09.397241 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 16:41:09.397445 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:09.397304 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" podUID="ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 16:41:09.493883 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:09.493846 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 16:41:09.494245 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:09.493912 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" podUID="4951f605-1e39-41f0-8d0a-2b23791653dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 16:41:10.035201 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.035169 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p"] Oct 30 16:41:10.089793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.089445 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2570c8b8-m4p4p"] Oct 30 16:41:10.220084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.216170 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5"] Oct 30 16:41:10.220084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.217396 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9a5940bd-e36f-448b-a43d-bb2febc2b8c9" containerName="revoke-keys" Oct 30 16:41:10.220084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.217413 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a5940bd-e36f-448b-a43d-bb2febc2b8c9" containerName="revoke-keys" Oct 30 16:41:10.220084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.217601 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a5940bd-e36f-448b-a43d-bb2febc2b8c9" containerName="revoke-keys" Oct 30 16:41:10.229289 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.227848 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5" Oct 30 16:41:10.229289 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.228188 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5"] Oct 30 16:41:10.253791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.249087 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:41:10.253791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.249879 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:41:10.298846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.295191 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:41:10.298846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.297459 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:10.298846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.297738 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:41:10.407885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.407851 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 16:41:10.408119 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.407909 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" podUID="ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 16:41:10.417178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.417127 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a30cd40-6649-46fc-98ab-3b398fb53d38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ec27a38-sgzq5\" (UID: \"2a30cd40-6649-46fc-98ab-3b398fb53d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5" Oct 30 16:41:10.417400 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.417363 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb2dc\" (UniqueName: \"kubernetes.io/projected/2a30cd40-6649-46fc-98ab-3b398fb53d38-kube-api-access-sb2dc\") pod \"maas-api-revoke-sub-2ec27a38-sgzq5\" (UID: \"2a30cd40-6649-46fc-98ab-3b398fb53d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5" Oct 30 16:41:10.417585 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.417555 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a30cd40-6649-46fc-98ab-3b398fb53d38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ec27a38-sgzq5\" (UID: \"2a30cd40-6649-46fc-98ab-3b398fb53d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5" Oct 30 16:41:10.418187 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.418161 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2770411-8647-4b26-98dc-ab9818be3d4f" path="/var/lib/kubelet/pods/c2770411-8647-4b26-98dc-ab9818be3d4f/volumes" Oct 30 16:41:10.494357 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.494309 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 16:41:10.494884 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.494405 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" podUID="4951f605-1e39-41f0-8d0a-2b23791653dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 16:41:10.519421 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.519376 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb2dc\" (UniqueName: \"kubernetes.io/projected/2a30cd40-6649-46fc-98ab-3b398fb53d38-kube-api-access-sb2dc\") pod \"maas-api-revoke-sub-2ec27a38-sgzq5\" (UID: \"2a30cd40-6649-46fc-98ab-3b398fb53d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5" Oct 30 16:41:10.519593 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.519562 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a30cd40-6649-46fc-98ab-3b398fb53d38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ec27a38-sgzq5\" (UID: \"2a30cd40-6649-46fc-98ab-3b398fb53d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5" Oct 30 16:41:10.519667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.519637 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a30cd40-6649-46fc-98ab-3b398fb53d38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ec27a38-sgzq5\" (UID: \"2a30cd40-6649-46fc-98ab-3b398fb53d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5" Oct 30 16:41:10.520366 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.520341 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a30cd40-6649-46fc-98ab-3b398fb53d38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ec27a38-sgzq5\" (UID: \"2a30cd40-6649-46fc-98ab-3b398fb53d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5" Oct 30 16:41:10.523020 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.523001 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a30cd40-6649-46fc-98ab-3b398fb53d38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ec27a38-sgzq5\" (UID: \"2a30cd40-6649-46fc-98ab-3b398fb53d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5" Oct 30 16:41:10.563046 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.562959 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb2dc\" (UniqueName: \"kubernetes.io/projected/2a30cd40-6649-46fc-98ab-3b398fb53d38-kube-api-access-sb2dc\") pod \"maas-api-revoke-sub-2ec27a38-sgzq5\" (UID: \"2a30cd40-6649-46fc-98ab-3b398fb53d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5" Oct 30 16:41:10.606012 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.605976 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5" Oct 30 16:41:10.727925 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.727849 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:10.983908 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:10.976024 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5"] Oct 30 16:41:11.225807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:11.225483 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:11.401602 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:11.401168 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 16:41:11.401602 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:11.401228 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" podUID="ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 16:41:11.452618 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:11.445250 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:11.479639 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:11.479557 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5" event={"ID":"2a30cd40-6649-46fc-98ab-3b398fb53d38","Type":"ContainerStarted","Data":"d3918a54aa7cc5d712259eb177373e183fcc0df0eb5b6033281a0abdc6e8d2b8"} Oct 30 16:41:11.501801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:11.497178 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 16:41:11.501801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:11.497232 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" podUID="4951f605-1e39-41f0-8d0a-2b23791653dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 16:41:12.087258 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:12.087220 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:12.496501 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:12.496394 6202 generic.go:356] "Generic (PLEG): container finished" podID="2a30cd40-6649-46fc-98ab-3b398fb53d38" containerID="de7758be79136db98681aa638138a24e76d47e26cc575e9eb44070a93eb025f1" exitCode=0 Oct 30 16:41:12.496713 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:12.496696 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5" event={"ID":"2a30cd40-6649-46fc-98ab-3b398fb53d38","Type":"ContainerDied","Data":"de7758be79136db98681aa638138a24e76d47e26cc575e9eb44070a93eb025f1"} Oct 30 16:41:12.628824 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:12.628751 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:12.629585 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:12.629556 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:12.630519 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:12.630501 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:41:13.066504 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:13.064574 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:13.066504 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:13.065905 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:13.067108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:13.066881 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:41:13.690611 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:13.690579 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5" Oct 30 16:41:13.711577 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:13.711539 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a30cd40-6649-46fc-98ab-3b398fb53d38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a30cd40-6649-46fc-98ab-3b398fb53d38-maas-api-xks-ca\") pod \"2a30cd40-6649-46fc-98ab-3b398fb53d38\" (UID: \"2a30cd40-6649-46fc-98ab-3b398fb53d38\") " Oct 30 16:41:13.711717 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:13.711639 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a30cd40-6649-46fc-98ab-3b398fb53d38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a30cd40-6649-46fc-98ab-3b398fb53d38-maas-api-openshift-ca\") pod \"2a30cd40-6649-46fc-98ab-3b398fb53d38\" (UID: \"2a30cd40-6649-46fc-98ab-3b398fb53d38\") " Oct 30 16:41:13.711858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:13.711839 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a30cd40-6649-46fc-98ab-3b398fb53d38-kube-api-access-sb2dc\" (UniqueName: \"kubernetes.io/projected/2a30cd40-6649-46fc-98ab-3b398fb53d38-kube-api-access-sb2dc\") pod \"2a30cd40-6649-46fc-98ab-3b398fb53d38\" (UID: \"2a30cd40-6649-46fc-98ab-3b398fb53d38\") " Oct 30 16:41:13.712215 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:13.712120 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a30cd40-6649-46fc-98ab-3b398fb53d38-maas-api-openshift-ca" pod "2a30cd40-6649-46fc-98ab-3b398fb53d38" (UID: "2a30cd40-6649-46fc-98ab-3b398fb53d38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:41:13.712671 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:13.712649 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a30cd40-6649-46fc-98ab-3b398fb53d38-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:41:13.713932 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:13.713912 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a30cd40-6649-46fc-98ab-3b398fb53d38-maas-api-xks-ca" pod "2a30cd40-6649-46fc-98ab-3b398fb53d38" (UID: "2a30cd40-6649-46fc-98ab-3b398fb53d38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:41:13.714405 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:13.714371 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a30cd40-6649-46fc-98ab-3b398fb53d38-kube-api-access-sb2dc" pod "2a30cd40-6649-46fc-98ab-3b398fb53d38" (UID: "2a30cd40-6649-46fc-98ab-3b398fb53d38"). InnerVolumeSpecName "kube-api-access-sb2dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:41:13.814133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:13.814043 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb2dc\" (UniqueName: \"kubernetes.io/projected/2a30cd40-6649-46fc-98ab-3b398fb53d38-kube-api-access-sb2dc\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:41:13.814133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:13.814075 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a30cd40-6649-46fc-98ab-3b398fb53d38-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:41:13.884118 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:13.884083 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:14.071552 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:41:14.071518 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:41:14.385922 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:14.385843 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:14.514520 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:14.514490 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5" Oct 30 16:41:14.514665 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:14.514522 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5" event={"ID":"2a30cd40-6649-46fc-98ab-3b398fb53d38","Type":"ContainerDied","Data":"d3918a54aa7cc5d712259eb177373e183fcc0df0eb5b6033281a0abdc6e8d2b8"} Oct 30 16:41:14.514665 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:14.514573 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3918a54aa7cc5d712259eb177373e183fcc0df0eb5b6033281a0abdc6e8d2b8" Oct 30 16:41:14.537029 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:14.537001 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:15.773459 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:15.773424 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:15.995926 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:15.995885 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:16.157487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:16.157433 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:16.189470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:16.189441 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424"] Oct 30 16:41:16.190094 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:16.190080 6202 cpu_manager.go:414] "containerMap: removing container" podUID="2a30cd40-6649-46fc-98ab-3b398fb53d38" containerName="revoke-keys" Oct 30 16:41:16.190146 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:16.190095 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a30cd40-6649-46fc-98ab-3b398fb53d38" containerName="revoke-keys" Oct 30 16:41:16.190240 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:16.190230 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a30cd40-6649-46fc-98ab-3b398fb53d38" containerName="revoke-keys" Oct 30 16:41:16.194038 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:16.194015 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424" Oct 30 16:41:16.214182 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:16.214158 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424"] Oct 30 16:41:16.223012 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:16.222985 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4db2924d-dockercfg-5s6qs\"" Oct 30 16:41:16.223148 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:16.223068 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4db2924d\"" Oct 30 16:41:16.271852 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:16.271802 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7000c6f9-beba-4fda-aebc-ce1098c7ee41-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424\" (UID: \"7000c6f9-beba-4fda-aebc-ce1098c7ee41\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424" Oct 30 16:41:16.313789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:16.294940 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw7xt\" (UniqueName: \"kubernetes.io/projected/7000c6f9-beba-4fda-aebc-ce1098c7ee41-kube-api-access-fw7xt\") pod \"payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424\" (UID: \"7000c6f9-beba-4fda-aebc-ce1098c7ee41\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424" Oct 30 16:41:16.425020 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:16.424925 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw7xt\" (UniqueName: \"kubernetes.io/projected/7000c6f9-beba-4fda-aebc-ce1098c7ee41-kube-api-access-fw7xt\") pod \"payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424\" (UID: \"7000c6f9-beba-4fda-aebc-ce1098c7ee41\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424" Oct 30 16:41:16.425193 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:16.425056 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7000c6f9-beba-4fda-aebc-ce1098c7ee41-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424\" (UID: \"7000c6f9-beba-4fda-aebc-ce1098c7ee41\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424" Oct 30 16:41:16.425920 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:16.425894 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7000c6f9-beba-4fda-aebc-ce1098c7ee41-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424\" (UID: \"7000c6f9-beba-4fda-aebc-ce1098c7ee41\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424" Oct 30 16:41:16.450223 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:16.450152 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw7xt\" (UniqueName: \"kubernetes.io/projected/7000c6f9-beba-4fda-aebc-ce1098c7ee41-kube-api-access-fw7xt\") pod \"payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424\" (UID: \"7000c6f9-beba-4fda-aebc-ce1098c7ee41\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424" Oct 30 16:41:16.509549 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:16.509515 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424" Oct 30 16:41:16.846209 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:41:16.846178 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7000c6f9_beba_4fda_aebc_ce1098c7ee41.slice/crio-e3d9ae62b32b684f581ae5dea3ce0e6bac7e946b2487478d3e42165ff66d36dd WatchSource:0}: Error finding container e3d9ae62b32b684f581ae5dea3ce0e6bac7e946b2487478d3e42165ff66d36dd: Status 404 returned error can't find the container with id e3d9ae62b32b684f581ae5dea3ce0e6bac7e946b2487478d3e42165ff66d36dd Oct 30 16:41:16.848273 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:16.848244 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424"] Oct 30 16:41:17.557065 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:17.557023 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424" event={"ID":"7000c6f9-beba-4fda-aebc-ce1098c7ee41","Type":"ContainerStarted","Data":"e26b33a04fbc72e3e3799d25661b34ad50f4f5bb9db19f46f5b4213d6365b451"} Oct 30 16:41:17.557308 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:17.557091 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424" event={"ID":"7000c6f9-beba-4fda-aebc-ce1098c7ee41","Type":"ContainerStarted","Data":"e3d9ae62b32b684f581ae5dea3ce0e6bac7e946b2487478d3e42165ff66d36dd"} Oct 30 16:41:17.557469 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:17.557372 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424" Oct 30 16:41:17.598828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:17.598736 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424" podStartSLOduration=1.59871976 podStartE2EDuration="1.59871976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:17.597089497 +0000 UTC m=+1191.307544244" watchObservedRunningTime="2026-10-30 16:41:17.59871976 +0000 UTC m=+1191.309174521" Oct 30 16:41:17.768853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:17.768794 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:19.033572 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.033530 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm"] Oct 30 16:41:19.048115 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.048081 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5b0469-jw7bm"] Oct 30 16:41:19.337574 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.337545 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:19.460234 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.460199 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:19.552156 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.552124 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r"] Oct 30 16:41:19.557184 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.557160 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.588290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.587547 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/970bb829-3799-40ba-a22d-f1e1ae061631-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.588290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.587621 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/970bb829-3799-40ba-a22d-f1e1ae061631-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.588290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.587697 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.588290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.587823 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.588290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.588007 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.588290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.588079 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.588290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.588281 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.588627 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.588366 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8tk5\" (UniqueName: \"kubernetes.io/projected/970bb829-3799-40ba-a22d-f1e1ae061631-kube-api-access-r8tk5\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.591752 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.591727 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r"] Oct 30 16:41:19.594520 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.594491 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/970bb829-3799-40ba-a22d-f1e1ae061631-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.696369 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.696331 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.696541 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.696392 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.696541 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.696453 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8tk5\" (UniqueName: \"kubernetes.io/projected/970bb829-3799-40ba-a22d-f1e1ae061631-kube-api-access-r8tk5\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.696541 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.696512 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/970bb829-3799-40ba-a22d-f1e1ae061631-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.696733 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.696672 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/970bb829-3799-40ba-a22d-f1e1ae061631-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.696733 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.696699 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.696733 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.696725 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/970bb829-3799-40ba-a22d-f1e1ae061631-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.696911 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.696806 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.696911 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.696831 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.696911 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.696872 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.697198 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.697173 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.697265 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.697208 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.697435 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.697415 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/970bb829-3799-40ba-a22d-f1e1ae061631-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.697481 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.697457 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.698912 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.698886 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.699218 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.699195 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/970bb829-3799-40ba-a22d-f1e1ae061631-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.720162 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.720136 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/970bb829-3799-40ba-a22d-f1e1ae061631-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.720257 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.720143 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8tk5\" (UniqueName: \"kubernetes.io/projected/970bb829-3799-40ba-a22d-f1e1ae061631-kube-api-access-r8tk5\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:19.875733 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:19.875647 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:20.202284 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:41:20.202255 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod970bb829_3799_40ba_a22d_f1e1ae061631.slice/crio-a8ae791ae5e4596f7ca85497917f11211546d9ea851f843c4cad8d01e433d871 WatchSource:0}: Error finding container a8ae791ae5e4596f7ca85497917f11211546d9ea851f843c4cad8d01e433d871: Status 404 returned error can't find the container with id a8ae791ae5e4596f7ca85497917f11211546d9ea851f843c4cad8d01e433d871 Oct 30 16:41:20.205252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.205210 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:20.205350 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.205298 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:20.205390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.205348 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:20.205969 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.205867 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r"] Oct 30 16:41:20.414072 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.414029 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3445db7b-baec-4665-b2ad-274e0c0f5da1" path="/var/lib/kubelet/pods/3445db7b-baec-4665-b2ad-274e0c0f5da1/volumes" Oct 30 16:41:20.578793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.574918 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9"] Oct 30 16:41:20.588789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.585743 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9" Oct 30 16:41:20.606573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.606484 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:41:20.613561 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.612916 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeb9db09-b582-470c-bcb2-b9a22839fc71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca519378-25kf9\" (UID: \"eeb9db09-b582-470c-bcb2-b9a22839fc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9" Oct 30 16:41:20.613561 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.613015 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cs57\" (UniqueName: \"kubernetes.io/projected/eeb9db09-b582-470c-bcb2-b9a22839fc71-kube-api-access-4cs57\") pod \"maas-api-revoke-sub-ca519378-25kf9\" (UID: \"eeb9db09-b582-470c-bcb2-b9a22839fc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9" Oct 30 16:41:20.613561 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.613099 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeb9db09-b582-470c-bcb2-b9a22839fc71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca519378-25kf9\" (UID: \"eeb9db09-b582-470c-bcb2-b9a22839fc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9" Oct 30 16:41:20.618108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.617799 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:41:20.618108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.617901 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:41:20.618108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.617967 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:41:20.632835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.626121 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9"] Oct 30 16:41:20.632835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.629802 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" event={"ID":"970bb829-3799-40ba-a22d-f1e1ae061631","Type":"ContainerStarted","Data":"8a39b3ba925f37d64a88a7b2a4e699344740cdb0deb9aa169baeec6a394c820e"} Oct 30 16:41:20.632835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.629861 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" event={"ID":"970bb829-3799-40ba-a22d-f1e1ae061631","Type":"ContainerStarted","Data":"a8ae791ae5e4596f7ca85497917f11211546d9ea851f843c4cad8d01e433d871"} Oct 30 16:41:20.686017 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.684009 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" podStartSLOduration=1.68399053 podStartE2EDuration="1.68399053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:20.680063746 +0000 UTC m=+1194.390518490" watchObservedRunningTime="2026-10-30 16:41:20.68399053 +0000 UTC m=+1194.394445275" Oct 30 16:41:20.715673 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.715639 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeb9db09-b582-470c-bcb2-b9a22839fc71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca519378-25kf9\" (UID: \"eeb9db09-b582-470c-bcb2-b9a22839fc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9" Oct 30 16:41:20.716464 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.716435 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeb9db09-b582-470c-bcb2-b9a22839fc71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca519378-25kf9\" (UID: \"eeb9db09-b582-470c-bcb2-b9a22839fc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9" Oct 30 16:41:20.716648 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.716627 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeb9db09-b582-470c-bcb2-b9a22839fc71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca519378-25kf9\" (UID: \"eeb9db09-b582-470c-bcb2-b9a22839fc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9" Oct 30 16:41:20.716836 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.716813 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cs57\" (UniqueName: \"kubernetes.io/projected/eeb9db09-b582-470c-bcb2-b9a22839fc71-kube-api-access-4cs57\") pod \"maas-api-revoke-sub-ca519378-25kf9\" (UID: \"eeb9db09-b582-470c-bcb2-b9a22839fc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9" Oct 30 16:41:20.721039 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.721013 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeb9db09-b582-470c-bcb2-b9a22839fc71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca519378-25kf9\" (UID: \"eeb9db09-b582-470c-bcb2-b9a22839fc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9" Oct 30 16:41:20.741066 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.741036 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cs57\" (UniqueName: \"kubernetes.io/projected/eeb9db09-b582-470c-bcb2-b9a22839fc71-kube-api-access-4cs57\") pod \"maas-api-revoke-sub-ca519378-25kf9\" (UID: \"eeb9db09-b582-470c-bcb2-b9a22839fc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9" Oct 30 16:41:20.876811 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.876706 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:20.935526 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:20.935486 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9" Oct 30 16:41:21.260846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:21.260664 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:21.285514 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:21.285484 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9"] Oct 30 16:41:21.285709 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:41:21.285683 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice/crio-647c759f8fdeb97871011460477612086d143d400bebcaba160fb016d105cf43 WatchSource:0}: Error finding container 647c759f8fdeb97871011460477612086d143d400bebcaba160fb016d105cf43: Status 404 returned error can't find the container with id 647c759f8fdeb97871011460477612086d143d400bebcaba160fb016d105cf43 Oct 30 16:41:21.686755 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:21.686712 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9" event={"ID":"eeb9db09-b582-470c-bcb2-b9a22839fc71","Type":"ContainerStarted","Data":"647c759f8fdeb97871011460477612086d143d400bebcaba160fb016d105cf43"} Oct 30 16:41:21.877610 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:21.877574 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 16:41:21.877805 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:21.877636 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" podUID="970bb829-3799-40ba-a22d-f1e1ae061631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 16:41:22.070873 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:22.070746 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd"] Oct 30 16:41:22.076121 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:22.076086 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57a984c2-q7kmd"] Oct 30 16:41:22.412461 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:22.412432 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b354d37-f9fd-4db3-9881-ea315888fd99" path="/var/lib/kubelet/pods/9b354d37-f9fd-4db3-9881-ea315888fd99/volumes" Oct 30 16:41:22.696605 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:22.696515 6202 generic.go:356] "Generic (PLEG): container finished" podID="eeb9db09-b582-470c-bcb2-b9a22839fc71" containerID="bc14d88d9a035e2603e3e85501fddd33ab1f3bdf6f4e2f764cb3f8ef17488a71" exitCode=0 Oct 30 16:41:22.696794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:22.696600 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9" event={"ID":"eeb9db09-b582-470c-bcb2-b9a22839fc71","Type":"ContainerDied","Data":"bc14d88d9a035e2603e3e85501fddd33ab1f3bdf6f4e2f764cb3f8ef17488a71"} Oct 30 16:41:22.880723 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:22.880683 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 16:41:22.881071 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:22.881047 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" podUID="970bb829-3799-40ba-a22d-f1e1ae061631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 16:41:23.119790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:23.111847 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8"] Oct 30 16:41:23.195792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:23.184926 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a1c2927-2mqg8"] Oct 30 16:41:23.637803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:23.634396 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:23.877389 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:23.877332 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 16:41:23.877715 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:23.877685 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" podUID="970bb829-3799-40ba-a22d-f1e1ae061631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 16:41:24.463793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:24.450060 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bf28f2b-6a23-40aa-822b-bdf67321ae75" path="/var/lib/kubelet/pods/6bf28f2b-6a23-40aa-822b-bdf67321ae75/volumes" Oct 30 16:41:24.852632 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:24.816282 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Oct 30 16:41:25.061025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:24.924176 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 16:41:25.061025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:24.924239 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" podUID="970bb829-3799-40ba-a22d-f1e1ae061631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 16:41:25.098793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:25.089948 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.236856 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:25.236264 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 16:41:25.236856 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:25.236689 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.237110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:25.236958 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:41:25.289792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:25.237247 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded" Oct 30 16:41:25.289792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:25.237540 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.289792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:25.237846 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 16:41:25.289792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:25.239877 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 16:41:25.604872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:25.549297 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.656389 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:25.655987 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9" Oct 30 16:41:26.021137 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:25.938979 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 16:41:26.021137 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:25.939043 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" podUID="970bb829-3799-40ba-a22d-f1e1ae061631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 16:41:26.151858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:26.112570 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:26.235795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:26.153853 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:41:26.304861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:26.237047 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 16:41:26.361246 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:26.348742 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:41:26.527440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:26.237116 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:26.527440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:26.237657 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:26.527440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:26.237686 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded" Oct 30 16:41:26.527440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:26.254927 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 16:41:26.527440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:26.257866 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:26.527440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:26.267082 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeb9db09-b582-470c-bcb2-b9a22839fc71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeb9db09-b582-470c-bcb2-b9a22839fc71-maas-api-xks-ca\") pod \"eeb9db09-b582-470c-bcb2-b9a22839fc71\" (UID: \"eeb9db09-b582-470c-bcb2-b9a22839fc71\") " Oct 30 16:41:26.527440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:26.480055 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeb9db09-b582-470c-bcb2-b9a22839fc71-kube-api-access-4cs57\" (UniqueName: \"kubernetes.io/projected/eeb9db09-b582-470c-bcb2-b9a22839fc71-kube-api-access-4cs57\") pod \"eeb9db09-b582-470c-bcb2-b9a22839fc71\" (UID: \"eeb9db09-b582-470c-bcb2-b9a22839fc71\") " Oct 30 16:41:26.527440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:26.480140 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeb9db09-b582-470c-bcb2-b9a22839fc71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeb9db09-b582-470c-bcb2-b9a22839fc71-maas-api-openshift-ca\") pod \"eeb9db09-b582-470c-bcb2-b9a22839fc71\" (UID: \"eeb9db09-b582-470c-bcb2-b9a22839fc71\") " Oct 30 16:41:26.527440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:26.237085 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 16:41:26.675794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:26.669246 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb9db09-b582-470c-bcb2-b9a22839fc71-maas-api-openshift-ca" pod "eeb9db09-b582-470c-bcb2-b9a22839fc71" (UID: "eeb9db09-b582-470c-bcb2-b9a22839fc71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:41:26.805094 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:26.778242 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeb9db09-b582-470c-bcb2-b9a22839fc71-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:41:26.943848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:26.934991 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb9db09-b582-470c-bcb2-b9a22839fc71-kube-api-access-4cs57" pod "eeb9db09-b582-470c-bcb2-b9a22839fc71" (UID: "eeb9db09-b582-470c-bcb2-b9a22839fc71"). InnerVolumeSpecName "kube-api-access-4cs57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:41:26.998804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:26.955484 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 16:41:27.078355 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.078313 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" podUID="970bb829-3799-40ba-a22d-f1e1ae061631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 16:41:27.151676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:26.990986 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cs57\" (UniqueName: \"kubernetes.io/projected/eeb9db09-b582-470c-bcb2-b9a22839fc71-kube-api-access-4cs57\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:41:27.151676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.078250 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:41:27.151676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.149951 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" podUID="35198b3d-3a56-45c7-8d36-d4de1780a938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.172902 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.172855 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 16:41:27.192234 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.192169 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb9db09-b582-470c-bcb2-b9a22839fc71-maas-api-xks-ca" pod "eeb9db09-b582-470c-bcb2-b9a22839fc71" (UID: "eeb9db09-b582-470c-bcb2-b9a22839fc71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:41:27.201850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.198796 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9" Oct 30 16:41:27.280644 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.280516 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:41:27.281276 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.280918 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" podUID="05d8ddc3-7f1c-4c56-a906-3722b011ab90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.308816 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.301226 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeb9db09-b582-470c-bcb2-b9a22839fc71-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:41:27.361508 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.361463 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 16:41:27.483858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.483427 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 16:41:27.483858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.483711 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.484402 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.484190 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:41:27.484402 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.484362 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded" Oct 30 16:41:27.497798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.485463 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.591810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.589936 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.696817 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:41:27.696016 6202 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.254s" Oct 30 16:41:27.724208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.724178 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:41:27.724558 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.724541 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:41:27.724816 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.724794 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9" event={"ID":"eeb9db09-b582-470c-bcb2-b9a22839fc71","Type":"ContainerDied","Data":"647c759f8fdeb97871011460477612086d143d400bebcaba160fb016d105cf43"} Oct 30 16:41:27.724957 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.724942 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="647c759f8fdeb97871011460477612086d143d400bebcaba160fb016d105cf43" Oct 30 16:41:27.725105 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.725093 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:41:27.725266 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.725254 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:41:27.725417 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.725404 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:41:27.725575 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.725562 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:41:27.767430 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.767293 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:27.802833 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:41:27.793888 6202 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice/crio-647c759f8fdeb97871011460477612086d143d400bebcaba160fb016d105cf43\": RecentStats: unable to find data in memory cache]" Oct 30 16:41:27.807091 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:41:27.807050 6202 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice/crio-647c759f8fdeb97871011460477612086d143d400bebcaba160fb016d105cf43\": RecentStats: unable to find data in memory cache]" Oct 30 16:41:28.053894 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:28.035877 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:28.328112 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:28.327626 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:41:28.328112 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:28.327692 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" podUID="ec24f992-d313-4d6d-987d-98c4c8361df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:28.417615 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:27.882674 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 16:41:28.422824 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:28.417887 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" podUID="970bb829-3799-40ba-a22d-f1e1ae061631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 16:41:28.510804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:28.495933 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 16:41:29.016858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.014277 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 16:41:29.036366 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.034927 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 16:41:29.036781 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.034978 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:41:29.037128 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.034998 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:41:29.037509 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.035042 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded" Oct 30 16:41:29.050845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.035069 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 16:41:29.054783 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.035096 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 16:41:29.181930 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.181641 6202 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:41:29.181930 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.181705 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" podUID="ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:29.270103 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.270004 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 16:41:29.270383 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.270352 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" podUID="970bb829-3799-40ba-a22d-f1e1ae061631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 16:41:29.307886 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.307857 6202 scope.go:122] "RemoveContainer" containerID="2ea8c89386266f67a58d7eea217e87f68e1b762e1665ef5c56d4049daef08de0" Oct 30 16:41:29.406579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.406546 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:41:29.407108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.406876 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:41:29.407108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.406920 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:41:29.407108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.406968 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424" Oct 30 16:41:29.407108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.407013 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:41:29.407108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.407042 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:41:29.407108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.407070 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:41:29.407108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.407097 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:41:29.413463 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.413438 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:41:29.581870 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:41:29.574654 6202 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice/crio-647c759f8fdeb97871011460477612086d143d400bebcaba160fb016d105cf43\": RecentStats: unable to find data in memory cache]" Oct 30 16:41:29.610464 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:41:29.610419 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea8c89386266f67a58d7eea217e87f68e1b762e1665ef5c56d4049daef08de0\": container with ID starting with 2ea8c89386266f67a58d7eea217e87f68e1b762e1665ef5c56d4049daef08de0 not found: ID does not exist" containerID="2ea8c89386266f67a58d7eea217e87f68e1b762e1665ef5c56d4049daef08de0" Oct 30 16:41:29.610464 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.610464 6202 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"2ea8c89386266f67a58d7eea217e87f68e1b762e1665ef5c56d4049daef08de0\": container with ID starting with 2ea8c89386266f67a58d7eea217e87f68e1b762e1665ef5c56d4049daef08de0 not found: ID does not exist" Oct 30 16:41:29.623349 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:41:29.623280 6202 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"2ea8c89386266f67a58d7eea217e87f68e1b762e1665ef5c56d4049daef08de0\": container with ID starting with 2ea8c89386266f67a58d7eea217e87f68e1b762e1665ef5c56d4049daef08de0 not found: ID does not exist" Oct 30 16:41:29.691124 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.691011 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:29.888838 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.888090 6202 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 16:41:29.888838 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:29.888147 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" podUID="970bb829-3799-40ba-a22d-f1e1ae061631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 16:41:31.082820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:31.082759 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24"] Oct 30 16:41:31.092505 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:31.092477 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b97bb07-28j24"] Oct 30 16:41:31.259842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:31.259815 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:31.260612 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:31.260591 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:31.261752 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:31.261720 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:41:31.378901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:31.378759 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:31.895927 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:41:31.895890 6202 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice/crio-647c759f8fdeb97871011460477612086d143d400bebcaba160fb016d105cf43\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice\": RecentStats: unable to find data in memory cache]" Oct 30 16:41:32.411333 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:32.411300 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b6ee8e3-149f-40f2-9e0f-586559326f88" path="/var/lib/kubelet/pods/6b6ee8e3-149f-40f2-9e0f-586559326f88/volumes" Oct 30 16:41:33.007827 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:33.007757 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:33.135894 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:33.135861 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:33.276613 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:33.276510 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:33.415283 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:33.415248 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:33.873426 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:33.873393 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97"] Oct 30 16:41:33.874715 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:33.874684 6202 cpu_manager.go:414] "containerMap: removing container" podUID="eeb9db09-b582-470c-bcb2-b9a22839fc71" containerName="revoke-keys" Oct 30 16:41:33.874715 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:33.874715 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb9db09-b582-470c-bcb2-b9a22839fc71" containerName="revoke-keys" Oct 30 16:41:33.874981 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:33.874962 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb9db09-b582-470c-bcb2-b9a22839fc71" containerName="revoke-keys" Oct 30 16:41:33.882195 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:33.882170 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:33.906065 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:33.906037 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97"] Oct 30 16:41:33.911820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:33.909974 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-a475e0dd-gw-openshift-default-dockercfg-bq2nd\"" Oct 30 16:41:33.936109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:33.936083 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:34.005456 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.005405 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-workload-certs\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.005648 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.005508 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e492a729-2008-4c88-a7b9-25caea1a5983-istio-podinfo\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.005648 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.005542 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-istio-envoy\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.005648 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.005626 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e492a729-2008-4c88-a7b9-25caea1a5983-istio-token\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.005900 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.005654 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-workload-socket\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.005900 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.005754 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-istio-data\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.005900 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.005807 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-credential-socket\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.005900 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.005882 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glm2v\" (UniqueName: \"kubernetes.io/projected/e492a729-2008-4c88-a7b9-25caea1a5983-kube-api-access-glm2v\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.006120 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.005916 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e492a729-2008-4c88-a7b9-25caea1a5983-istiod-ca-cert\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.104446 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.104404 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:34.109113 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.108894 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glm2v\" (UniqueName: \"kubernetes.io/projected/e492a729-2008-4c88-a7b9-25caea1a5983-kube-api-access-glm2v\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.109113 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.108957 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e492a729-2008-4c88-a7b9-25caea1a5983-istiod-ca-cert\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.109113 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.109059 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-workload-certs\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.109399 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.109140 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e492a729-2008-4c88-a7b9-25caea1a5983-istio-podinfo\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.109399 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.109170 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-istio-envoy\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.109399 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.109248 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e492a729-2008-4c88-a7b9-25caea1a5983-istio-token\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.109399 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.109274 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-workload-socket\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.109604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.109405 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-istio-data\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.109604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.109442 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-credential-socket\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.110158 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.110132 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-credential-socket\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.110552 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.110529 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-workload-socket\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.111026 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.111003 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-istio-data\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.111375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.111350 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e492a729-2008-4c88-a7b9-25caea1a5983-istiod-ca-cert\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.111461 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.111355 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-workload-certs\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.121333 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.117207 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e492a729-2008-4c88-a7b9-25caea1a5983-istio-podinfo\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.121333 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.119542 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-istio-envoy\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.134613 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.134552 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glm2v\" (UniqueName: \"kubernetes.io/projected/e492a729-2008-4c88-a7b9-25caea1a5983-kube-api-access-glm2v\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.144277 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.144249 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e492a729-2008-4c88-a7b9-25caea1a5983-istio-token\") pod \"e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.207422 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.207388 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:34.570990 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:41:34.570957 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode492a729_2008_4c88_a7b9_25caea1a5983.slice/crio-dfa0b0239b3c656c8cb05c62f20b268cfd5dfae427238edfe9594c978b9281ce WatchSource:0}: Error finding container dfa0b0239b3c656c8cb05c62f20b268cfd5dfae427238edfe9594c978b9281ce: Status 404 returned error can't find the container with id dfa0b0239b3c656c8cb05c62f20b268cfd5dfae427238edfe9594c978b9281ce Oct 30 16:41:34.571573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.571553 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97"] Oct 30 16:41:34.574106 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.574052 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:34.574208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.574126 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:34.574208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:34.574158 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:35.001079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.001046 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5"] Oct 30 16:41:35.006171 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.006149 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5" Oct 30 16:41:35.026081 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.026054 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5"] Oct 30 16:41:35.032431 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.032328 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:41:35.032431 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.032401 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:41:35.032622 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.032199 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:41:35.032991 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.032959 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:41:35.049791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.043800 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66"] Oct 30 16:41:35.055165 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.055110 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt"] Oct 30 16:41:35.061504 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.061482 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa490f0-lmc66"] Oct 30 16:41:35.068459 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.068431 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgjm2\" (UniqueName: \"kubernetes.io/projected/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-kube-api-access-zgjm2\") pod \"maas-api-revoke-sub-c928adc4-wb6p5\" (UID: \"964580e0-4bc1-4b73-8694-3e2dcb3da8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5" Oct 30 16:41:35.068591 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.068558 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c928adc4-wb6p5\" (UID: \"964580e0-4bc1-4b73-8694-3e2dcb3da8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5" Oct 30 16:41:35.068668 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.068634 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c928adc4-wb6p5\" (UID: \"964580e0-4bc1-4b73-8694-3e2dcb3da8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5" Oct 30 16:41:35.073288 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.073262 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80392282-n2vbt"] Oct 30 16:41:35.169913 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.169880 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c928adc4-wb6p5\" (UID: \"964580e0-4bc1-4b73-8694-3e2dcb3da8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5" Oct 30 16:41:35.170202 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.170179 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgjm2\" (UniqueName: \"kubernetes.io/projected/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-kube-api-access-zgjm2\") pod \"maas-api-revoke-sub-c928adc4-wb6p5\" (UID: \"964580e0-4bc1-4b73-8694-3e2dcb3da8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5" Oct 30 16:41:35.170326 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.170309 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c928adc4-wb6p5\" (UID: \"964580e0-4bc1-4b73-8694-3e2dcb3da8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5" Oct 30 16:41:35.170706 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.170680 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c928adc4-wb6p5\" (UID: \"964580e0-4bc1-4b73-8694-3e2dcb3da8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5" Oct 30 16:41:35.172957 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.172931 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c928adc4-wb6p5\" (UID: \"964580e0-4bc1-4b73-8694-3e2dcb3da8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5" Oct 30 16:41:35.194143 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.194115 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgjm2\" (UniqueName: \"kubernetes.io/projected/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-kube-api-access-zgjm2\") pod \"maas-api-revoke-sub-c928adc4-wb6p5\" (UID: \"964580e0-4bc1-4b73-8694-3e2dcb3da8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5" Oct 30 16:41:35.237933 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.237900 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:35.321834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.321760 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5" Oct 30 16:41:35.469286 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.469242 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" event={"ID":"e492a729-2008-4c88-a7b9-25caea1a5983","Type":"ContainerStarted","Data":"b3606144e02ad57741bf1d25970e8f45e2e0e8684fd3c8c5a8b5a45359891b8c"} Oct 30 16:41:35.469455 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.469336 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" event={"ID":"e492a729-2008-4c88-a7b9-25caea1a5983","Type":"ContainerStarted","Data":"dfa0b0239b3c656c8cb05c62f20b268cfd5dfae427238edfe9594c978b9281ce"} Oct 30 16:41:35.521535 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.521483 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" podStartSLOduration=2.521468656 podStartE2EDuration="2.521468656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:35.518075572 +0000 UTC m=+1209.228530318" watchObservedRunningTime="2026-10-30 16:41:35.521468656 +0000 UTC m=+1209.231923399" Oct 30 16:41:35.674843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:35.670808 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5"] Oct 30 16:41:36.208252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:36.208155 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:36.214012 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:36.213991 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:36.413989 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:36.413952 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e0253d5-274d-4e10-9331-954e74065e17" path="/var/lib/kubelet/pods/6e0253d5-274d-4e10-9331-954e74065e17/volumes" Oct 30 16:41:36.414597 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:36.414580 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77b79ab5-a82f-416b-8fbe-a4c2de84a128" path="/var/lib/kubelet/pods/77b79ab5-a82f-416b-8fbe-a4c2de84a128/volumes" Oct 30 16:41:36.479254 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:36.479163 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5" event={"ID":"964580e0-4bc1-4b73-8694-3e2dcb3da8eb","Type":"ContainerStarted","Data":"079d0b337c49150fe916672a6e70a18b0fe42e27afff84f4c68761cff665bbac"} Oct 30 16:41:36.479254 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:36.479221 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5" event={"ID":"964580e0-4bc1-4b73-8694-3e2dcb3da8eb","Type":"ContainerStarted","Data":"ea5b44b369429b7b00c6df58420cd55ade8b29a9dfc31a2b43611929c84a8152"} Oct 30 16:41:36.479504 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:36.479484 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:36.480384 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:36.480366 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:41:36.522050 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:36.521996 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5" podStartSLOduration=2.521982042 podStartE2EDuration="2.521982042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:36.51793071 +0000 UTC m=+1210.228385453" watchObservedRunningTime="2026-10-30 16:41:36.521982042 +0000 UTC m=+1210.232436785" Oct 30 16:41:37.052899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:37.052860 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:37.415875 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:37.415817 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:37.489392 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:37.489356 6202 generic.go:356] "Generic (PLEG): container finished" podID="964580e0-4bc1-4b73-8694-3e2dcb3da8eb" containerID="079d0b337c49150fe916672a6e70a18b0fe42e27afff84f4c68761cff665bbac" exitCode=0 Oct 30 16:41:37.489556 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:37.489445 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5" event={"ID":"964580e0-4bc1-4b73-8694-3e2dcb3da8eb","Type":"ContainerDied","Data":"079d0b337c49150fe916672a6e70a18b0fe42e27afff84f4c68761cff665bbac"} Oct 30 16:41:37.534243 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:37.534156 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:38.249482 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.249442 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:38.342481 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.342452 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6"] Oct 30 16:41:38.347105 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.347086 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6" Oct 30 16:41:38.368412 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.368389 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6"] Oct 30 16:41:38.375197 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.375174 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a475e0dd\"" Oct 30 16:41:38.428256 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.428218 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:38.434278 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.434246 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ef6485-8b36-4bb2-b643-31b06cdedddd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6\" (UID: \"51ef6485-8b36-4bb2-b643-31b06cdedddd\") " pod="openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6" Oct 30 16:41:38.434397 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.434311 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7vfj\" (UniqueName: \"kubernetes.io/projected/51ef6485-8b36-4bb2-b643-31b06cdedddd-kube-api-access-k7vfj\") pod \"payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6\" (UID: \"51ef6485-8b36-4bb2-b643-31b06cdedddd\") " pod="openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6" Oct 30 16:41:38.616822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.613400 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ef6485-8b36-4bb2-b643-31b06cdedddd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6\" (UID: \"51ef6485-8b36-4bb2-b643-31b06cdedddd\") " pod="openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6" Oct 30 16:41:38.616822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.613548 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7vfj\" (UniqueName: \"kubernetes.io/projected/51ef6485-8b36-4bb2-b643-31b06cdedddd-kube-api-access-k7vfj\") pod \"payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6\" (UID: \"51ef6485-8b36-4bb2-b643-31b06cdedddd\") " pod="openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6" Oct 30 16:41:38.616822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.615122 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ef6485-8b36-4bb2-b643-31b06cdedddd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6\" (UID: \"51ef6485-8b36-4bb2-b643-31b06cdedddd\") " pod="openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6" Oct 30 16:41:38.657985 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.651514 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7vfj\" (UniqueName: \"kubernetes.io/projected/51ef6485-8b36-4bb2-b643-31b06cdedddd-kube-api-access-k7vfj\") pod \"payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6\" (UID: \"51ef6485-8b36-4bb2-b643-31b06cdedddd\") " pod="openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6" Oct 30 16:41:38.664061 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.663970 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6" Oct 30 16:41:38.872338 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.871931 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5" Oct 30 16:41:38.930441 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.930407 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-maas-api-xks-ca\") pod \"964580e0-4bc1-4b73-8694-3e2dcb3da8eb\" (UID: \"964580e0-4bc1-4b73-8694-3e2dcb3da8eb\") " Oct 30 16:41:38.930719 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.930697 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-kube-api-access-zgjm2\" (UniqueName: \"kubernetes.io/projected/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-kube-api-access-zgjm2\") pod \"964580e0-4bc1-4b73-8694-3e2dcb3da8eb\" (UID: \"964580e0-4bc1-4b73-8694-3e2dcb3da8eb\") " Oct 30 16:41:38.931380 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.931358 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-maas-api-openshift-ca\") pod \"964580e0-4bc1-4b73-8694-3e2dcb3da8eb\" (UID: \"964580e0-4bc1-4b73-8694-3e2dcb3da8eb\") " Oct 30 16:41:38.931818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.931757 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-maas-api-openshift-ca" pod "964580e0-4bc1-4b73-8694-3e2dcb3da8eb" (UID: "964580e0-4bc1-4b73-8694-3e2dcb3da8eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:41:38.932417 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.932354 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:41:38.933731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.933692 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-kube-api-access-zgjm2" pod "964580e0-4bc1-4b73-8694-3e2dcb3da8eb" (UID: "964580e0-4bc1-4b73-8694-3e2dcb3da8eb"). InnerVolumeSpecName "kube-api-access-zgjm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:41:38.934281 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:38.934220 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-maas-api-xks-ca" pod "964580e0-4bc1-4b73-8694-3e2dcb3da8eb" (UID: "964580e0-4bc1-4b73-8694-3e2dcb3da8eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:41:39.008621 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:41:39.008585 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ef6485_8b36_4bb2_b643_31b06cdedddd.slice/crio-bf6d178a9215287868f6e03f1238e09c520263df7030d6ed3b15c275a815c04c WatchSource:0}: Error finding container bf6d178a9215287868f6e03f1238e09c520263df7030d6ed3b15c275a815c04c: Status 404 returned error can't find the container with id bf6d178a9215287868f6e03f1238e09c520263df7030d6ed3b15c275a815c04c Oct 30 16:41:39.009117 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:39.009095 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6"] Oct 30 16:41:39.033621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:39.033592 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:41:39.033621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:39.033617 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgjm2\" (UniqueName: \"kubernetes.io/projected/964580e0-4bc1-4b73-8694-3e2dcb3da8eb-kube-api-access-zgjm2\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:41:39.160520 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:39.160445 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:39.637460 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:39.637433 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5" Oct 30 16:41:39.637962 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:39.637526 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5" event={"ID":"964580e0-4bc1-4b73-8694-3e2dcb3da8eb","Type":"ContainerDied","Data":"ea5b44b369429b7b00c6df58420cd55ade8b29a9dfc31a2b43611929c84a8152"} Oct 30 16:41:39.637962 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:39.637589 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea5b44b369429b7b00c6df58420cd55ade8b29a9dfc31a2b43611929c84a8152" Oct 30 16:41:39.640492 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:39.639724 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6" event={"ID":"51ef6485-8b36-4bb2-b643-31b06cdedddd","Type":"ContainerStarted","Data":"521126afbe5eda38b1d1c14944406bf1241d1a0adad558ca533a3e501094096d"} Oct 30 16:41:39.640492 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:39.639812 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6" event={"ID":"51ef6485-8b36-4bb2-b643-31b06cdedddd","Type":"ContainerStarted","Data":"bf6d178a9215287868f6e03f1238e09c520263df7030d6ed3b15c275a815c04c"} Oct 30 16:41:39.640492 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:39.639963 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6" Oct 30 16:41:39.668072 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:41:39.668047 6202 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice/crio-647c759f8fdeb97871011460477612086d143d400bebcaba160fb016d105cf43\": RecentStats: unable to find data in memory cache]" Oct 30 16:41:39.685530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:39.685480 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6" podStartSLOduration=1.685467508 podStartE2EDuration="1.685467508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:39.680314427 +0000 UTC m=+1213.390769182" watchObservedRunningTime="2026-10-30 16:41:39.685467508 +0000 UTC m=+1213.395922267" Oct 30 16:41:40.753140 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:40.753104 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:42.465855 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:42.465817 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:44.072752 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:41:44.072703 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:41:44.274490 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:44.274455 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:45.036749 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:45.036713 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7"] Oct 30 16:41:45.050839 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:45.050799 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db40c21-6w6k7"] Oct 30 16:41:45.994182 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:45.994148 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:46.411725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:46.411693 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5be78108-87f2-41b7-bee1-0b404be3614b" path="/var/lib/kubelet/pods/5be78108-87f2-41b7-bee1-0b404be3614b/volumes" Oct 30 16:41:46.979110 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:41:46.979069 6202 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice/crio-647c759f8fdeb97871011460477612086d143d400bebcaba160fb016d105cf43\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice\": RecentStats: unable to find data in memory cache]" Oct 30 16:41:47.492951 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:47.492918 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:49.503863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:49.503754 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:49.706799 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:41:49.706062 6202 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice/crio-647c759f8fdeb97871011460477612086d143d400bebcaba160fb016d105cf43\": RecentStats: unable to find data in memory cache]" Oct 30 16:41:50.664793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:50.664040 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6" Oct 30 16:41:51.507139 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:51.507037 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:51.939411 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:51.939368 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:52.068297 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:52.066916 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 16:41:52.068297 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:52.067327 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:52.068297 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:52.067881 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:41:52.470700 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:52.470659 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:52.505801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:52.471759 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:52.508110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:52.507738 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:52.552793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:52.539385 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:41:52.552793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:52.539444 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" podUID="dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:52.560900 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:52.560875 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:41:53.041437 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:53.041392 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:53.093595 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:53.093547 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:53.135096 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:53.135047 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:53.161899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:53.161853 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:41:53.162120 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:53.162103 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:41:53.162349 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:53.162334 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:41:53.313510 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:53.313426 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:41:53.368947 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:53.365942 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:41:53.533923 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:53.533859 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:56.661139 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.661097 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf"] Oct 30 16:41:56.662031 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.662012 6202 cpu_manager.go:414] "containerMap: removing container" podUID="964580e0-4bc1-4b73-8694-3e2dcb3da8eb" containerName="revoke-keys" Oct 30 16:41:56.662031 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.662031 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="964580e0-4bc1-4b73-8694-3e2dcb3da8eb" containerName="revoke-keys" Oct 30 16:41:56.662147 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.662137 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="964580e0-4bc1-4b73-8694-3e2dcb3da8eb" containerName="revoke-keys" Oct 30 16:41:56.665840 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.665821 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf" Oct 30 16:41:56.685676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.685646 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf"] Oct 30 16:41:56.694810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.691315 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:41:56.694810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.691398 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:41:56.694810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.691460 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:41:56.694810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.692106 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:41:56.763002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.762962 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662a08aa-69c4-49cb-9b70-14f59ee316c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f654570d-wtqlf\" (UID: \"662a08aa-69c4-49cb-9b70-14f59ee316c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf" Oct 30 16:41:56.763199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.763043 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662a08aa-69c4-49cb-9b70-14f59ee316c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f654570d-wtqlf\" (UID: \"662a08aa-69c4-49cb-9b70-14f59ee316c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf" Oct 30 16:41:56.763258 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.763240 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvs4g\" (UniqueName: \"kubernetes.io/projected/662a08aa-69c4-49cb-9b70-14f59ee316c5-kube-api-access-gvs4g\") pod \"maas-api-revoke-keys-e2e-ait-f654570d-wtqlf\" (UID: \"662a08aa-69c4-49cb-9b70-14f59ee316c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf" Oct 30 16:41:56.846624 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.846586 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:56.864711 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.864675 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvs4g\" (UniqueName: \"kubernetes.io/projected/662a08aa-69c4-49cb-9b70-14f59ee316c5-kube-api-access-gvs4g\") pod \"maas-api-revoke-keys-e2e-ait-f654570d-wtqlf\" (UID: \"662a08aa-69c4-49cb-9b70-14f59ee316c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf" Oct 30 16:41:56.864937 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.864812 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662a08aa-69c4-49cb-9b70-14f59ee316c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f654570d-wtqlf\" (UID: \"662a08aa-69c4-49cb-9b70-14f59ee316c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf" Oct 30 16:41:56.864937 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.864851 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662a08aa-69c4-49cb-9b70-14f59ee316c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f654570d-wtqlf\" (UID: \"662a08aa-69c4-49cb-9b70-14f59ee316c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf" Oct 30 16:41:56.867157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.866700 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662a08aa-69c4-49cb-9b70-14f59ee316c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f654570d-wtqlf\" (UID: \"662a08aa-69c4-49cb-9b70-14f59ee316c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf" Oct 30 16:41:56.892920 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.892882 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvs4g\" (UniqueName: \"kubernetes.io/projected/662a08aa-69c4-49cb-9b70-14f59ee316c5-kube-api-access-gvs4g\") pod \"maas-api-revoke-keys-e2e-ait-f654570d-wtqlf\" (UID: \"662a08aa-69c4-49cb-9b70-14f59ee316c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf" Oct 30 16:41:56.893405 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.893368 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662a08aa-69c4-49cb-9b70-14f59ee316c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f654570d-wtqlf\" (UID: \"662a08aa-69c4-49cb-9b70-14f59ee316c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf" Oct 30 16:41:56.980339 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:56.980245 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf" Oct 30 16:41:57.021707 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:57.021672 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:57.048667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:57.048628 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85"] Oct 30 16:41:57.053524 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:57.053493 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dccf310-w9f85"] Oct 30 16:41:57.180300 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:57.166838 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:57.306615 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:41:57.306575 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662a08aa_69c4_49cb_9b70_14f59ee316c5.slice/crio-2d5cf63e10c6573420b7101b9d9823b5a380fde112083f57b9426a0be10d54ad WatchSource:0}: Error finding container 2d5cf63e10c6573420b7101b9d9823b5a380fde112083f57b9426a0be10d54ad: Status 404 returned error can't find the container with id 2d5cf63e10c6573420b7101b9d9823b5a380fde112083f57b9426a0be10d54ad Oct 30 16:41:57.308009 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:57.307885 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf"] Oct 30 16:41:57.408252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:57.408215 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf" event={"ID":"662a08aa-69c4-49cb-9b70-14f59ee316c5","Type":"ContainerStarted","Data":"2d5cf63e10c6573420b7101b9d9823b5a380fde112083f57b9426a0be10d54ad"} Oct 30 16:41:57.906251 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:57.906152 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:58.417179 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:58.417131 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88ef6c68-ddb8-4e03-afbf-d8a908183cc2" path="/var/lib/kubelet/pods/88ef6c68-ddb8-4e03-afbf-d8a908183cc2/volumes" Oct 30 16:41:58.421861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:58.421828 6202 generic.go:356] "Generic (PLEG): container finished" podID="662a08aa-69c4-49cb-9b70-14f59ee316c5" containerID="677fbbf12c35a66f1612c9310a6246ad838c755b6bf403c7b12d7812cdc10039" exitCode=0 Oct 30 16:41:58.422482 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:58.421912 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf" event={"ID":"662a08aa-69c4-49cb-9b70-14f59ee316c5","Type":"ContainerDied","Data":"677fbbf12c35a66f1612c9310a6246ad838c755b6bf403c7b12d7812cdc10039"} Oct 30 16:41:59.615049 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:59.615011 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:59.619436 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:59.619406 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf" Oct 30 16:41:59.714731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:59.712642 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/662a08aa-69c4-49cb-9b70-14f59ee316c5-kube-api-access-gvs4g\" (UniqueName: \"kubernetes.io/projected/662a08aa-69c4-49cb-9b70-14f59ee316c5-kube-api-access-gvs4g\") pod \"662a08aa-69c4-49cb-9b70-14f59ee316c5\" (UID: \"662a08aa-69c4-49cb-9b70-14f59ee316c5\") " Oct 30 16:41:59.714731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:59.713177 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/662a08aa-69c4-49cb-9b70-14f59ee316c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662a08aa-69c4-49cb-9b70-14f59ee316c5-maas-api-xks-ca\") pod \"662a08aa-69c4-49cb-9b70-14f59ee316c5\" (UID: \"662a08aa-69c4-49cb-9b70-14f59ee316c5\") " Oct 30 16:41:59.714731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:59.713559 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/662a08aa-69c4-49cb-9b70-14f59ee316c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662a08aa-69c4-49cb-9b70-14f59ee316c5-maas-api-openshift-ca\") pod \"662a08aa-69c4-49cb-9b70-14f59ee316c5\" (UID: \"662a08aa-69c4-49cb-9b70-14f59ee316c5\") " Oct 30 16:41:59.715291 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:59.715235 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662a08aa-69c4-49cb-9b70-14f59ee316c5-maas-api-openshift-ca" pod "662a08aa-69c4-49cb-9b70-14f59ee316c5" (UID: "662a08aa-69c4-49cb-9b70-14f59ee316c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:41:59.717464 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:59.717435 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662a08aa-69c4-49cb-9b70-14f59ee316c5-kube-api-access-gvs4g" pod "662a08aa-69c4-49cb-9b70-14f59ee316c5" (UID: "662a08aa-69c4-49cb-9b70-14f59ee316c5"). InnerVolumeSpecName "kube-api-access-gvs4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:41:59.721104 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:59.721073 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662a08aa-69c4-49cb-9b70-14f59ee316c5-maas-api-xks-ca" pod "662a08aa-69c4-49cb-9b70-14f59ee316c5" (UID: "662a08aa-69c4-49cb-9b70-14f59ee316c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:41:59.799394 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:59.799349 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:59.815558 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:59.815486 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662a08aa-69c4-49cb-9b70-14f59ee316c5-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:41:59.815558 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:59.815521 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662a08aa-69c4-49cb-9b70-14f59ee316c5-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:41:59.815558 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:41:59.815540 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvs4g\" (UniqueName: \"kubernetes.io/projected/662a08aa-69c4-49cb-9b70-14f59ee316c5-kube-api-access-gvs4g\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:41:59.818427 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:41:59.818393 6202 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice/crio-647c759f8fdeb97871011460477612086d143d400bebcaba160fb016d105cf43\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice\": RecentStats: unable to find data in memory cache]" Oct 30 16:42:00.124885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.124847 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8"] Oct 30 16:42:00.125572 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.125559 6202 cpu_manager.go:414] "containerMap: removing container" podUID="662a08aa-69c4-49cb-9b70-14f59ee316c5" containerName="revoke-keys" Oct 30 16:42:00.125622 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.125574 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="662a08aa-69c4-49cb-9b70-14f59ee316c5" containerName="revoke-keys" Oct 30 16:42:00.125708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.125698 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="662a08aa-69c4-49cb-9b70-14f59ee316c5" containerName="revoke-keys" Oct 30 16:42:00.130030 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.130009 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8" Oct 30 16:42:00.154417 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.154385 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8"] Oct 30 16:42:00.221145 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.221114 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-745h4\" (UniqueName: \"kubernetes.io/projected/b0d10705-542d-4993-9657-c51e0da16e31-kube-api-access-745h4\") pod \"maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8\" (UID: \"b0d10705-542d-4993-9657-c51e0da16e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8" Oct 30 16:42:00.221348 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.221279 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d10705-542d-4993-9657-c51e0da16e31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8\" (UID: \"b0d10705-542d-4993-9657-c51e0da16e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8" Oct 30 16:42:00.221441 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.221423 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d10705-542d-4993-9657-c51e0da16e31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8\" (UID: \"b0d10705-542d-4993-9657-c51e0da16e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8" Oct 30 16:42:00.323443 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.323411 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d10705-542d-4993-9657-c51e0da16e31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8\" (UID: \"b0d10705-542d-4993-9657-c51e0da16e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8" Oct 30 16:42:00.323641 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.323530 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d10705-542d-4993-9657-c51e0da16e31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8\" (UID: \"b0d10705-542d-4993-9657-c51e0da16e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8" Oct 30 16:42:00.323734 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.323717 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-745h4\" (UniqueName: \"kubernetes.io/projected/b0d10705-542d-4993-9657-c51e0da16e31-kube-api-access-745h4\") pod \"maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8\" (UID: \"b0d10705-542d-4993-9657-c51e0da16e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8" Oct 30 16:42:00.324394 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.324357 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d10705-542d-4993-9657-c51e0da16e31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8\" (UID: \"b0d10705-542d-4993-9657-c51e0da16e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8" Oct 30 16:42:00.326046 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.326019 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d10705-542d-4993-9657-c51e0da16e31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8\" (UID: \"b0d10705-542d-4993-9657-c51e0da16e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8" Oct 30 16:42:00.347640 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.347605 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-745h4\" (UniqueName: \"kubernetes.io/projected/b0d10705-542d-4993-9657-c51e0da16e31-kube-api-access-745h4\") pod \"maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8\" (UID: \"b0d10705-542d-4993-9657-c51e0da16e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8" Oct 30 16:42:00.358367 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.358336 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:00.445565 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.445531 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8" Oct 30 16:42:00.446642 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.446609 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf" event={"ID":"662a08aa-69c4-49cb-9b70-14f59ee316c5","Type":"ContainerDied","Data":"2d5cf63e10c6573420b7101b9d9823b5a380fde112083f57b9426a0be10d54ad"} Oct 30 16:42:00.446864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.446671 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d5cf63e10c6573420b7101b9d9823b5a380fde112083f57b9426a0be10d54ad" Oct 30 16:42:00.446864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.446740 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf" Oct 30 16:42:00.783331 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:42:00.783301 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d10705_542d_4993_9657_c51e0da16e31.slice/crio-0e0a8283656adbc823dba625a425c052fac4bba4774e5de32d56940797b7d40f WatchSource:0}: Error finding container 0e0a8283656adbc823dba625a425c052fac4bba4774e5de32d56940797b7d40f: Status 404 returned error can't find the container with id 0e0a8283656adbc823dba625a425c052fac4bba4774e5de32d56940797b7d40f Oct 30 16:42:00.786057 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:00.786029 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8"] Oct 30 16:42:01.458369 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:01.458329 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8" event={"ID":"b0d10705-542d-4993-9657-c51e0da16e31","Type":"ContainerStarted","Data":"12b1915199cfbcb372a889fd514085107177b2d2d3d10f0e1c40e6a160871007"} Oct 30 16:42:01.458559 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:01.458384 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8" event={"ID":"b0d10705-542d-4993-9657-c51e0da16e31","Type":"ContainerStarted","Data":"0e0a8283656adbc823dba625a425c052fac4bba4774e5de32d56940797b7d40f"} Oct 30 16:42:01.502551 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:01.502495 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8" podStartSLOduration=1.5024798000000001 podStartE2EDuration="1.5024798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:01.498376881 +0000 UTC m=+1235.208831625" watchObservedRunningTime="2026-10-30 16:42:01.5024798 +0000 UTC m=+1235.212934541" Oct 30 16:42:01.843296 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:01.843258 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:01.894001 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:42:01.893963 6202 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice/crio-647c759f8fdeb97871011460477612086d143d400bebcaba160fb016d105cf43\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice\": RecentStats: unable to find data in memory cache]" Oct 30 16:42:02.471421 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:02.471385 6202 generic.go:356] "Generic (PLEG): container finished" podID="b0d10705-542d-4993-9657-c51e0da16e31" containerID="12b1915199cfbcb372a889fd514085107177b2d2d3d10f0e1c40e6a160871007" exitCode=0 Oct 30 16:42:02.471653 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:02.471478 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8" event={"ID":"b0d10705-542d-4993-9657-c51e0da16e31","Type":"ContainerDied","Data":"12b1915199cfbcb372a889fd514085107177b2d2d3d10f0e1c40e6a160871007"} Oct 30 16:42:02.929041 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:02.929006 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl"] Oct 30 16:42:02.929442 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:02.929247 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" podUID="8fa1e96c-f52b-4de4-8949-c22c45d6d842" containerName="istio-proxy" containerID="cri-o://3e38c7fb3bd2a1ac8ac9b841a1908bacc03ecfd5e8f62b8d06db5776284b080c" gracePeriod=30 Oct 30 16:42:03.485497 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:03.485464 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:03.704350 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:03.704322 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8" Oct 30 16:42:03.788897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:03.788726 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0d10705-542d-4993-9657-c51e0da16e31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d10705-542d-4993-9657-c51e0da16e31-maas-api-openshift-ca\") pod \"b0d10705-542d-4993-9657-c51e0da16e31\" (UID: \"b0d10705-542d-4993-9657-c51e0da16e31\") " Oct 30 16:42:03.789596 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:03.789021 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d10705-542d-4993-9657-c51e0da16e31-kube-api-access-745h4\" (UniqueName: \"kubernetes.io/projected/b0d10705-542d-4993-9657-c51e0da16e31-kube-api-access-745h4\") pod \"b0d10705-542d-4993-9657-c51e0da16e31\" (UID: \"b0d10705-542d-4993-9657-c51e0da16e31\") " Oct 30 16:42:03.789596 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:03.789166 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0d10705-542d-4993-9657-c51e0da16e31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d10705-542d-4993-9657-c51e0da16e31-maas-api-xks-ca\") pod \"b0d10705-542d-4993-9657-c51e0da16e31\" (UID: \"b0d10705-542d-4993-9657-c51e0da16e31\") " Oct 30 16:42:03.789596 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:03.789267 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d10705-542d-4993-9657-c51e0da16e31-maas-api-openshift-ca" pod "b0d10705-542d-4993-9657-c51e0da16e31" (UID: "b0d10705-542d-4993-9657-c51e0da16e31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:03.789812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:03.789738 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d10705-542d-4993-9657-c51e0da16e31-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:03.791782 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:03.791736 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d10705-542d-4993-9657-c51e0da16e31-maas-api-xks-ca" pod "b0d10705-542d-4993-9657-c51e0da16e31" (UID: "b0d10705-542d-4993-9657-c51e0da16e31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:03.792074 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:03.792055 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d10705-542d-4993-9657-c51e0da16e31-kube-api-access-745h4" pod "b0d10705-542d-4993-9657-c51e0da16e31" (UID: "b0d10705-542d-4993-9657-c51e0da16e31"). InnerVolumeSpecName "kube-api-access-745h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:03.890980 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:03.890944 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-745h4\" (UniqueName: \"kubernetes.io/projected/b0d10705-542d-4993-9657-c51e0da16e31-kube-api-access-745h4\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:03.890980 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:03.890976 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d10705-542d-4993-9657-c51e0da16e31-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:04.412853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.412820 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg"] Oct 30 16:42:04.413719 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.413695 6202 cpu_manager.go:414] "containerMap: removing container" podUID="b0d10705-542d-4993-9657-c51e0da16e31" containerName="revoke-keys" Oct 30 16:42:04.413874 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.413721 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d10705-542d-4993-9657-c51e0da16e31" containerName="revoke-keys" Oct 30 16:42:04.413953 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.413937 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d10705-542d-4993-9657-c51e0da16e31" containerName="revoke-keys" Oct 30 16:42:04.418149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.418125 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.422112 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.422090 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg"] Oct 30 16:42:04.427046 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.427026 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.447164 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.447129 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg"] Oct 30 16:42:04.466571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.466541 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg"] Oct 30 16:42:04.495469 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.495437 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8" event={"ID":"b0d10705-542d-4993-9657-c51e0da16e31","Type":"ContainerDied","Data":"0e0a8283656adbc823dba625a425c052fac4bba4774e5de32d56940797b7d40f"} Oct 30 16:42:04.495618 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.495490 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e0a8283656adbc823dba625a425c052fac4bba4774e5de32d56940797b7d40f" Oct 30 16:42:04.495618 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.495553 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8" Oct 30 16:42:04.499302 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.499274 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.499487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.499362 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.499487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.499421 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs24n\" (UniqueName: \"kubernetes.io/projected/e8bd28b1-844d-4b4f-bed7-542915f85c47-kube-api-access-qs24n\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.499487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.499451 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.499678 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.499595 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.499678 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.499656 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zdht\" (UniqueName: \"kubernetes.io/projected/80a5af6e-f20f-4745-83d3-fd5df7661cc3-kube-api-access-9zdht\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.499843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.499825 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.499909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.499890 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.499961 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.499947 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.499997 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.499979 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.500048 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.500035 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.500124 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.500088 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.500158 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.500128 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bd28b1-844d-4b4f-bed7-542915f85c47-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.500191 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.500166 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.500271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.500255 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.500311 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.500293 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.500345 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.500327 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.500402 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.500388 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.602014 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.601979 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.602306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602028 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.602306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602062 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.602306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602115 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.602306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602181 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.602306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602240 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.602306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602281 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs24n\" (UniqueName: \"kubernetes.io/projected/e8bd28b1-844d-4b4f-bed7-542915f85c47-kube-api-access-qs24n\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.602306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602308 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.602942 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602341 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.602942 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602394 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zdht\" (UniqueName: \"kubernetes.io/projected/80a5af6e-f20f-4745-83d3-fd5df7661cc3-kube-api-access-9zdht\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.602942 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602474 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.602942 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602519 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.602942 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602562 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.602942 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602592 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.602942 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602660 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.602942 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602719 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.602942 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602728 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.602942 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602795 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bd28b1-844d-4b4f-bed7-542915f85c47-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.602942 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602840 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.603528 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.602943 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.603528 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.603003 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.603528 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.603032 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.603528 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.603283 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.603748 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.603715 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.603878 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.603852 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.603952 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.603872 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.604050 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.604027 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.604106 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.604043 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bd28b1-844d-4b4f-bed7-542915f85c47-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.604651 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.604628 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.605312 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.605284 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.605480 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.605464 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.605612 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.605597 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.625279 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.625248 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.625445 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.625425 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zdht\" (UniqueName: \"kubernetes.io/projected/80a5af6e-f20f-4745-83d3-fd5df7661cc3-kube-api-access-9zdht\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.625534 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.625496 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs24n\" (UniqueName: \"kubernetes.io/projected/e8bd28b1-844d-4b4f-bed7-542915f85c47-kube-api-access-qs24n\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.625664 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.625645 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:04.766253 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.766169 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:04.782729 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:04.782696 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:05.102715 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:42:05.102673 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a5af6e_f20f_4745_83d3_fd5df7661cc3.slice/crio-de3c101c1a1919aa9b7d04091c643230cdc5fdc623f8bf04a9624013e5ddd31b WatchSource:0}: Error finding container de3c101c1a1919aa9b7d04091c643230cdc5fdc623f8bf04a9624013e5ddd31b: Status 404 returned error can't find the container with id de3c101c1a1919aa9b7d04091c643230cdc5fdc623f8bf04a9624013e5ddd31b Oct 30 16:42:05.104495 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:05.104443 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg"] Oct 30 16:42:05.105459 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:05.105424 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:05.105545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:05.105490 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:05.105545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:05.105520 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:05.137737 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:42:05.137707 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bd28b1_844d_4b4f_bed7_542915f85c47.slice/crio-56ebd316f465301d95789716ea22e8ce211f5b7fbd011103bff8d60a2350490e WatchSource:0}: Error finding container 56ebd316f465301d95789716ea22e8ce211f5b7fbd011103bff8d60a2350490e: Status 404 returned error can't find the container with id 56ebd316f465301d95789716ea22e8ce211f5b7fbd011103bff8d60a2350490e Oct 30 16:42:05.152377 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:05.152343 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg"] Oct 30 16:42:05.156287 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:05.156244 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:05.156411 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:05.156381 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:05.156489 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:05.156468 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:05.308344 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:05.308251 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:05.511526 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:05.511420 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" event={"ID":"80a5af6e-f20f-4745-83d3-fd5df7661cc3","Type":"ContainerStarted","Data":"c9d2dd04ff1419c63bdaeab4082fabb27a4ac267999a9a240987d8b00f65d8d8"} Oct 30 16:42:05.515084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:05.513606 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" event={"ID":"80a5af6e-f20f-4745-83d3-fd5df7661cc3","Type":"ContainerStarted","Data":"de3c101c1a1919aa9b7d04091c643230cdc5fdc623f8bf04a9624013e5ddd31b"} Oct 30 16:42:05.519589 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:05.519542 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" event={"ID":"e8bd28b1-844d-4b4f-bed7-542915f85c47","Type":"ContainerStarted","Data":"491f6cb6efb758dd1b31f9909e957fa419b32c053ee233695b38cc2ebf184f8e"} Oct 30 16:42:05.519889 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:05.519850 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" event={"ID":"e8bd28b1-844d-4b4f-bed7-542915f85c47","Type":"ContainerStarted","Data":"56ebd316f465301d95789716ea22e8ce211f5b7fbd011103bff8d60a2350490e"} Oct 30 16:42:05.564961 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:05.564898 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" podStartSLOduration=1.564878792 podStartE2EDuration="1.564878792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:05.558639287 +0000 UTC m=+1239.269094036" watchObservedRunningTime="2026-10-30 16:42:05.564878792 +0000 UTC m=+1239.275333537" Oct 30 16:42:05.607339 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:05.607264 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" podStartSLOduration=1.607245737 podStartE2EDuration="1.607245737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:05.605277832 +0000 UTC m=+1239.315732575" watchObservedRunningTime="2026-10-30 16:42:05.607245737 +0000 UTC m=+1239.317700483" Oct 30 16:42:05.767130 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:05.767041 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:05.783785 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:05.783735 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:06.418248 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.418215 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj"] Oct 30 16:42:06.423114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.423095 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj" Oct 30 16:42:06.464808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.463933 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:42:06.464808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.464696 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:42:06.464808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.464757 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:42:06.485791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.478023 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:42:06.514694 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.514590 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj"] Oct 30 16:42:06.583798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.579888 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae22304-8037-4bb2-9b6c-35f547407dd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dccd7785-pnrtj\" (UID: \"5ae22304-8037-4bb2-9b6c-35f547407dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj" Oct 30 16:42:06.583798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.579981 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcfdk\" (UniqueName: \"kubernetes.io/projected/5ae22304-8037-4bb2-9b6c-35f547407dd8-kube-api-access-vcfdk\") pod \"maas-api-revoke-sub-dccd7785-pnrtj\" (UID: \"5ae22304-8037-4bb2-9b6c-35f547407dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj" Oct 30 16:42:06.583798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.580047 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae22304-8037-4bb2-9b6c-35f547407dd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dccd7785-pnrtj\" (UID: \"5ae22304-8037-4bb2-9b6c-35f547407dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj" Oct 30 16:42:06.685128 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.685034 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae22304-8037-4bb2-9b6c-35f547407dd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dccd7785-pnrtj\" (UID: \"5ae22304-8037-4bb2-9b6c-35f547407dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj" Oct 30 16:42:06.687502 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.687473 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcfdk\" (UniqueName: \"kubernetes.io/projected/5ae22304-8037-4bb2-9b6c-35f547407dd8-kube-api-access-vcfdk\") pod \"maas-api-revoke-sub-dccd7785-pnrtj\" (UID: \"5ae22304-8037-4bb2-9b6c-35f547407dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj" Oct 30 16:42:06.688323 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.688300 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae22304-8037-4bb2-9b6c-35f547407dd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dccd7785-pnrtj\" (UID: \"5ae22304-8037-4bb2-9b6c-35f547407dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj" Oct 30 16:42:06.693395 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.686904 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae22304-8037-4bb2-9b6c-35f547407dd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dccd7785-pnrtj\" (UID: \"5ae22304-8037-4bb2-9b6c-35f547407dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj" Oct 30 16:42:06.706015 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.705990 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae22304-8037-4bb2-9b6c-35f547407dd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dccd7785-pnrtj\" (UID: \"5ae22304-8037-4bb2-9b6c-35f547407dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj" Oct 30 16:42:06.712319 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.712297 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcfdk\" (UniqueName: \"kubernetes.io/projected/5ae22304-8037-4bb2-9b6c-35f547407dd8-kube-api-access-vcfdk\") pod \"maas-api-revoke-sub-dccd7785-pnrtj\" (UID: \"5ae22304-8037-4bb2-9b6c-35f547407dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj" Oct 30 16:42:06.742050 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.742009 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj" Oct 30 16:42:06.799982 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.799843 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:42:06.800478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.800444 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" podUID="e8bd28b1-844d-4b4f-bed7-542915f85c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 16:42:06.802676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.802649 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 16:42:06.806970 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:06.806941 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" podUID="80a5af6e-f20f-4745-83d3-fd5df7661cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 16:42:07.002030 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:07.001746 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:07.218200 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:07.218159 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj"] Oct 30 16:42:07.511792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:07.505414 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:07.737555 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:07.737505 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:07.778814 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:07.778538 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 16:42:07.778814 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:07.778596 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" podUID="80a5af6e-f20f-4745-83d3-fd5df7661cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 16:42:07.783422 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:07.783308 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:42:07.783422 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:07.783371 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" podUID="e8bd28b1-844d-4b4f-bed7-542915f85c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 16:42:07.799932 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:07.787104 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj" event={"ID":"5ae22304-8037-4bb2-9b6c-35f547407dd8","Type":"ContainerStarted","Data":"0d3923adf33cc09e56cd355783a5069234348236016a073c92e98ec04d504062"} Oct 30 16:42:08.795991 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:08.768881 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 16:42:08.795991 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:08.768961 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" podUID="80a5af6e-f20f-4745-83d3-fd5df7661cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 16:42:08.814437 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:08.814286 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:42:08.815193 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:08.814683 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" podUID="e8bd28b1-844d-4b4f-bed7-542915f85c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 16:42:08.833246 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:08.833220 6202 generic.go:356] "Generic (PLEG): container finished" podID="8fa1e96c-f52b-4de4-8949-c22c45d6d842" containerID="3e38c7fb3bd2a1ac8ac9b841a1908bacc03ecfd5e8f62b8d06db5776284b080c" exitCode=0 Oct 30 16:42:08.833610 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:08.833561 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" event={"ID":"8fa1e96c-f52b-4de4-8949-c22c45d6d842","Type":"ContainerDied","Data":"3e38c7fb3bd2a1ac8ac9b841a1908bacc03ecfd5e8f62b8d06db5776284b080c"} Oct 30 16:42:08.836650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:08.836137 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj" event={"ID":"5ae22304-8037-4bb2-9b6c-35f547407dd8","Type":"ContainerStarted","Data":"bea80a946f03ce22338c17d69bfabaa053bdd828fca4ec934cc0adfde6f95ca4"} Oct 30 16:42:08.934095 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:08.934035 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj" podStartSLOduration=2.934016042 podStartE2EDuration="2.934016042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:08.923278742 +0000 UTC m=+1242.633733498" watchObservedRunningTime="2026-10-30 16:42:08.934016042 +0000 UTC m=+1242.644470791" Oct 30 16:42:09.038790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.036259 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:09.099594 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.087084 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:42:09.280842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.276734 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-podinfo\") pod \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " Oct 30 16:42:09.280842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.276834 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-credential-socket\") pod \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " Oct 30 16:42:09.280842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.276888 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-workload-socket\") pod \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " Oct 30 16:42:09.280842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.277034 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-data\") pod \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " Oct 30 16:42:09.280842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.277069 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-envoy\") pod \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " Oct 30 16:42:09.280842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.277166 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-token\" (UniqueName: \"kubernetes.io/projected/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-token\") pod \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " Oct 30 16:42:09.280842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.277280 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-workload-certs\") pod \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " Oct 30 16:42:09.280842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.277322 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istiod-ca-cert\") pod \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " Oct 30 16:42:09.280842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.277351 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa1e96c-f52b-4de4-8949-c22c45d6d842-kube-api-access-nrrz6\" (UniqueName: \"kubernetes.io/projected/8fa1e96c-f52b-4de4-8949-c22c45d6d842-kube-api-access-nrrz6\") pod \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\" (UID: \"8fa1e96c-f52b-4de4-8949-c22c45d6d842\") " Oct 30 16:42:09.322839 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.322802 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-data" pod "8fa1e96c-f52b-4de4-8949-c22c45d6d842" (UID: "8fa1e96c-f52b-4de4-8949-c22c45d6d842"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:09.338660 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.334534 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-workload-certs" pod "8fa1e96c-f52b-4de4-8949-c22c45d6d842" (UID: "8fa1e96c-f52b-4de4-8949-c22c45d6d842"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:09.338826 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.334726 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-workload-socket" pod "8fa1e96c-f52b-4de4-8949-c22c45d6d842" (UID: "8fa1e96c-f52b-4de4-8949-c22c45d6d842"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:09.343243 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.339232 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istiod-ca-cert" pod "8fa1e96c-f52b-4de4-8949-c22c45d6d842" (UID: "8fa1e96c-f52b-4de4-8949-c22c45d6d842"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:09.347260 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.346077 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-credential-socket" pod "8fa1e96c-f52b-4de4-8949-c22c45d6d842" (UID: "8fa1e96c-f52b-4de4-8949-c22c45d6d842"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:09.384268 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.384231 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:09.384491 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.384473 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:09.385054 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.385033 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:09.385213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.385200 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:09.385318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.385307 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:09.560850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.544008 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa1e96c-f52b-4de4-8949-c22c45d6d842-kube-api-access-nrrz6" pod "8fa1e96c-f52b-4de4-8949-c22c45d6d842" (UID: "8fa1e96c-f52b-4de4-8949-c22c45d6d842"). InnerVolumeSpecName "kube-api-access-nrrz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:09.560850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.544210 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-token" pod "8fa1e96c-f52b-4de4-8949-c22c45d6d842" (UID: "8fa1e96c-f52b-4de4-8949-c22c45d6d842"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:09.594848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.592974 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-envoy" pod "8fa1e96c-f52b-4de4-8949-c22c45d6d842" (UID: "8fa1e96c-f52b-4de4-8949-c22c45d6d842"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:09.596251 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.596215 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:09.596416 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.596400 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrrz6\" (UniqueName: \"kubernetes.io/projected/8fa1e96c-f52b-4de4-8949-c22c45d6d842-kube-api-access-nrrz6\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:09.596540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.596527 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:09.613745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.613706 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-podinfo" pod "8fa1e96c-f52b-4de4-8949-c22c45d6d842" (UID: "8fa1e96c-f52b-4de4-8949-c22c45d6d842"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:42:09.700314 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.700281 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa1e96c-f52b-4de4-8949-c22c45d6d842-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:09.768257 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.768220 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 16:42:09.768570 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.768534 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" podUID="80a5af6e-f20f-4745-83d3-fd5df7661cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 16:42:09.788140 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.788030 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:42:09.788140 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.788084 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" podUID="e8bd28b1-844d-4b4f-bed7-542915f85c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 16:42:09.970056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.970007 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" event={"ID":"8fa1e96c-f52b-4de4-8949-c22c45d6d842","Type":"ContainerDied","Data":"455c909908141d760fe57b50b39d6b941b8a753bab098186440021f96c3e6e1e"} Oct 30 16:42:09.970661 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.970641 6202 scope.go:122] "RemoveContainer" containerID="3e38c7fb3bd2a1ac8ac9b841a1908bacc03ecfd5e8f62b8d06db5776284b080c" Oct 30 16:42:09.976023 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:09.975985 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl" Oct 30 16:42:10.031748 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:10.031710 6202 generic.go:356] "Generic (PLEG): container finished" podID="5ae22304-8037-4bb2-9b6c-35f547407dd8" containerID="bea80a946f03ce22338c17d69bfabaa053bdd828fca4ec934cc0adfde6f95ca4" exitCode=0 Oct 30 16:42:10.031990 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:10.031961 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj" event={"ID":"5ae22304-8037-4bb2-9b6c-35f547407dd8","Type":"ContainerDied","Data":"bea80a946f03ce22338c17d69bfabaa053bdd828fca4ec934cc0adfde6f95ca4"} Oct 30 16:42:10.166893 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:10.164863 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl"] Oct 30 16:42:10.231395 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:10.227822 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-57mzl"] Oct 30 16:42:10.290794 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:42:10.290683 6202 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa1e96c_f52b_4de4_8949_c22c45d6d842.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa1e96c_f52b_4de4_8949_c22c45d6d842.slice/crio-455c909908141d760fe57b50b39d6b941b8a753bab098186440021f96c3e6e1e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice/crio-647c759f8fdeb97871011460477612086d143d400bebcaba160fb016d105cf43\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice\": RecentStats: unable to find data in memory cache]" Oct 30 16:42:10.414239 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:10.414206 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa1e96c-f52b-4de4-8949-c22c45d6d842" path="/var/lib/kubelet/pods/8fa1e96c-f52b-4de4-8949-c22c45d6d842/volumes" Oct 30 16:42:10.723500 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:10.723468 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424"] Oct 30 16:42:10.767251 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:10.767215 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 16:42:10.769246 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:10.769211 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" podUID="80a5af6e-f20f-4745-83d3-fd5df7661cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 16:42:10.784438 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:10.784385 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:42:10.862817 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:10.831023 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" podUID="e8bd28b1-844d-4b4f-bed7-542915f85c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 16:42:10.862817 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:10.833272 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:11.153022 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:11.066481 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:11.854794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:11.853786 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:42:11.854794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:11.853858 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" podUID="e8bd28b1-844d-4b4f-bed7-542915f85c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 16:42:11.902800 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:11.900255 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 16:42:11.902800 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:11.900339 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" podUID="80a5af6e-f20f-4745-83d3-fd5df7661cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 16:42:12.010984 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.006496 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:12.113448 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.113312 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj" Oct 30 16:42:12.211838 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.211797 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae22304-8037-4bb2-9b6c-35f547407dd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae22304-8037-4bb2-9b6c-35f547407dd8-maas-api-openshift-ca\") pod \"5ae22304-8037-4bb2-9b6c-35f547407dd8\" (UID: \"5ae22304-8037-4bb2-9b6c-35f547407dd8\") " Oct 30 16:42:12.212333 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.211887 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae22304-8037-4bb2-9b6c-35f547407dd8-kube-api-access-vcfdk\" (UniqueName: \"kubernetes.io/projected/5ae22304-8037-4bb2-9b6c-35f547407dd8-kube-api-access-vcfdk\") pod \"5ae22304-8037-4bb2-9b6c-35f547407dd8\" (UID: \"5ae22304-8037-4bb2-9b6c-35f547407dd8\") " Oct 30 16:42:12.212333 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.212108 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ae22304-8037-4bb2-9b6c-35f547407dd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae22304-8037-4bb2-9b6c-35f547407dd8-maas-api-xks-ca\") pod \"5ae22304-8037-4bb2-9b6c-35f547407dd8\" (UID: \"5ae22304-8037-4bb2-9b6c-35f547407dd8\") " Oct 30 16:42:12.212333 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.212266 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae22304-8037-4bb2-9b6c-35f547407dd8-maas-api-openshift-ca" pod "5ae22304-8037-4bb2-9b6c-35f547407dd8" (UID: "5ae22304-8037-4bb2-9b6c-35f547407dd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:12.212882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.212858 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae22304-8037-4bb2-9b6c-35f547407dd8-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:12.214406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.214378 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae22304-8037-4bb2-9b6c-35f547407dd8-maas-api-xks-ca" pod "5ae22304-8037-4bb2-9b6c-35f547407dd8" (UID: "5ae22304-8037-4bb2-9b6c-35f547407dd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:12.214406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.214388 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae22304-8037-4bb2-9b6c-35f547407dd8-kube-api-access-vcfdk" pod "5ae22304-8037-4bb2-9b6c-35f547407dd8" (UID: "5ae22304-8037-4bb2-9b6c-35f547407dd8"). InnerVolumeSpecName "kube-api-access-vcfdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:12.314943 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.314895 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae22304-8037-4bb2-9b6c-35f547407dd8-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:12.314943 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.314933 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcfdk\" (UniqueName: \"kubernetes.io/projected/5ae22304-8037-4bb2-9b6c-35f547407dd8-kube-api-access-vcfdk\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:12.358462 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.358314 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:12.367442 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.367350 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj" event={"ID":"5ae22304-8037-4bb2-9b6c-35f547407dd8","Type":"ContainerDied","Data":"0d3923adf33cc09e56cd355783a5069234348236016a073c92e98ec04d504062"} Oct 30 16:42:12.367587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.367448 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d3923adf33cc09e56cd355783a5069234348236016a073c92e98ec04d504062" Oct 30 16:42:12.367587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.367516 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj" Oct 30 16:42:12.767471 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.767372 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 16:42:12.767471 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.767440 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" podUID="80a5af6e-f20f-4745-83d3-fd5df7661cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 16:42:12.783976 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.783929 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:42:12.784223 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:12.784005 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" podUID="e8bd28b1-844d-4b4f-bed7-542915f85c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 16:42:14.079659 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:42:14.079624 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:42:14.241606 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:14.241558 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:14.819922 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:14.819884 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:42:14.820106 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:14.819971 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" podUID="80a5af6e-f20f-4745-83d3-fd5df7661cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:14.828943 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:14.828907 6202 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:42:14.829059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:14.828983 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" podUID="e8bd28b1-844d-4b4f-bed7-542915f85c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:15.305287 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:15.304161 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:15.305287 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:15.305178 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:15.307354 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:15.307328 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:42:15.465233 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:15.465196 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:15.468242 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:15.468215 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:15.469893 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:15.469876 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:42:15.724415 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:15.724379 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424" podUID="7000c6f9-beba-4fda-aebc-ce1098c7ee41" containerName="payload-pre-processing" containerID="cri-o://e26b33a04fbc72e3e3799d25661b34ad50f4f5bb9db19f46f5b4213d6365b451" gracePeriod=25 Oct 30 16:42:16.021107 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:16.021080 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424" Oct 30 16:42:16.120255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:16.120217 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7000c6f9-beba-4fda-aebc-ce1098c7ee41-kube-api-access-fw7xt\" (UniqueName: \"kubernetes.io/projected/7000c6f9-beba-4fda-aebc-ce1098c7ee41-kube-api-access-fw7xt\") pod \"7000c6f9-beba-4fda-aebc-ce1098c7ee41\" (UID: \"7000c6f9-beba-4fda-aebc-ce1098c7ee41\") " Oct 30 16:42:16.120589 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:16.120568 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7000c6f9-beba-4fda-aebc-ce1098c7ee41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7000c6f9-beba-4fda-aebc-ce1098c7ee41-plugins-config-volume\") pod \"7000c6f9-beba-4fda-aebc-ce1098c7ee41\" (UID: \"7000c6f9-beba-4fda-aebc-ce1098c7ee41\") " Oct 30 16:42:16.121188 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:16.121156 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7000c6f9-beba-4fda-aebc-ce1098c7ee41-plugins-config-volume" pod "7000c6f9-beba-4fda-aebc-ce1098c7ee41" (UID: "7000c6f9-beba-4fda-aebc-ce1098c7ee41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:16.122759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:16.122722 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7000c6f9-beba-4fda-aebc-ce1098c7ee41-kube-api-access-fw7xt" pod "7000c6f9-beba-4fda-aebc-ce1098c7ee41" (UID: "7000c6f9-beba-4fda-aebc-ce1098c7ee41"). InnerVolumeSpecName "kube-api-access-fw7xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:16.222750 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:16.222716 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw7xt\" (UniqueName: \"kubernetes.io/projected/7000c6f9-beba-4fda-aebc-ce1098c7ee41-kube-api-access-fw7xt\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:16.222750 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:16.222746 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7000c6f9-beba-4fda-aebc-ce1098c7ee41-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:16.342903 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:16.342855 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:16.448544 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:16.448501 6202 generic.go:356] "Generic (PLEG): container finished" podID="7000c6f9-beba-4fda-aebc-ce1098c7ee41" containerID="e26b33a04fbc72e3e3799d25661b34ad50f4f5bb9db19f46f5b4213d6365b451" exitCode=0 Oct 30 16:42:16.451394 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:16.451365 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424" Oct 30 16:42:16.453693 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:16.451992 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424" event={"ID":"7000c6f9-beba-4fda-aebc-ce1098c7ee41","Type":"ContainerDied","Data":"e26b33a04fbc72e3e3799d25661b34ad50f4f5bb9db19f46f5b4213d6365b451"} Oct 30 16:42:16.453693 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:16.452043 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424" event={"ID":"7000c6f9-beba-4fda-aebc-ce1098c7ee41","Type":"ContainerDied","Data":"e3d9ae62b32b684f581ae5dea3ce0e6bac7e946b2487478d3e42165ff66d36dd"} Oct 30 16:42:16.453693 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:16.452072 6202 scope.go:122] "RemoveContainer" containerID="e26b33a04fbc72e3e3799d25661b34ad50f4f5bb9db19f46f5b4213d6365b451" Oct 30 16:42:16.467553 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:16.467524 6202 scope.go:122] "RemoveContainer" containerID="e26b33a04fbc72e3e3799d25661b34ad50f4f5bb9db19f46f5b4213d6365b451" Oct 30 16:42:16.467926 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:42:16.467899 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26b33a04fbc72e3e3799d25661b34ad50f4f5bb9db19f46f5b4213d6365b451\": container with ID starting with e26b33a04fbc72e3e3799d25661b34ad50f4f5bb9db19f46f5b4213d6365b451 not found: ID does not exist" containerID="e26b33a04fbc72e3e3799d25661b34ad50f4f5bb9db19f46f5b4213d6365b451" Oct 30 16:42:16.468033 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:16.467946 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26b33a04fbc72e3e3799d25661b34ad50f4f5bb9db19f46f5b4213d6365b451"} err="failed to get container status \"e26b33a04fbc72e3e3799d25661b34ad50f4f5bb9db19f46f5b4213d6365b451\": rpc error: code = NotFound desc = could not find container \"e26b33a04fbc72e3e3799d25661b34ad50f4f5bb9db19f46f5b4213d6365b451\": container with ID starting with e26b33a04fbc72e3e3799d25661b34ad50f4f5bb9db19f46f5b4213d6365b451 not found: ID does not exist" Oct 30 16:42:16.508099 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:16.508061 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424"] Oct 30 16:42:16.516341 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:16.516297 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4db2924d-7c86548495-c8424"] Oct 30 16:42:16.985918 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:42:16.985872 6202 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice/crio-647c759f8fdeb97871011460477612086d143d400bebcaba160fb016d105cf43\": RecentStats: unable to find data in memory cache]" Oct 30 16:42:17.183205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.183168 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl"] Oct 30 16:42:17.183882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.183866 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5ae22304-8037-4bb2-9b6c-35f547407dd8" containerName="revoke-keys" Oct 30 16:42:17.183882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.183881 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae22304-8037-4bb2-9b6c-35f547407dd8" containerName="revoke-keys" Oct 30 16:42:17.184034 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.183905 6202 cpu_manager.go:414] "containerMap: removing container" podUID="7000c6f9-beba-4fda-aebc-ce1098c7ee41" containerName="payload-pre-processing" Oct 30 16:42:17.184034 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.183911 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7000c6f9-beba-4fda-aebc-ce1098c7ee41" containerName="payload-pre-processing" Oct 30 16:42:17.184034 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.183961 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8fa1e96c-f52b-4de4-8949-c22c45d6d842" containerName="istio-proxy" Oct 30 16:42:17.184034 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.183967 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa1e96c-f52b-4de4-8949-c22c45d6d842" containerName="istio-proxy" Oct 30 16:42:17.184230 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.184081 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa1e96c-f52b-4de4-8949-c22c45d6d842" containerName="istio-proxy" Oct 30 16:42:17.184230 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.184094 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae22304-8037-4bb2-9b6c-35f547407dd8" containerName="revoke-keys" Oct 30 16:42:17.184230 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.184103 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="7000c6f9-beba-4fda-aebc-ce1098c7ee41" containerName="payload-pre-processing" Oct 30 16:42:17.188927 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.188908 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl" Oct 30 16:42:17.208034 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.208010 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl"] Oct 30 16:42:17.208303 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.208275 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:42:17.208416 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.208323 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:42:17.213619 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.213597 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:42:17.213738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.213674 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:42:17.246574 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.246484 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xqtn\" (UniqueName: \"kubernetes.io/projected/a97e64ec-670e-4bee-99a1-b681a3ea1328-kube-api-access-2xqtn\") pod \"maas-api-revoke-sub-ca4e901d-cc9zl\" (UID: \"a97e64ec-670e-4bee-99a1-b681a3ea1328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl" Oct 30 16:42:17.246946 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.246921 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97e64ec-670e-4bee-99a1-b681a3ea1328-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca4e901d-cc9zl\" (UID: \"a97e64ec-670e-4bee-99a1-b681a3ea1328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl" Oct 30 16:42:17.247313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.247159 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97e64ec-670e-4bee-99a1-b681a3ea1328-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca4e901d-cc9zl\" (UID: \"a97e64ec-670e-4bee-99a1-b681a3ea1328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl" Oct 30 16:42:17.349022 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.348986 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97e64ec-670e-4bee-99a1-b681a3ea1328-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca4e901d-cc9zl\" (UID: \"a97e64ec-670e-4bee-99a1-b681a3ea1328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl" Oct 30 16:42:17.349475 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.349152 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xqtn\" (UniqueName: \"kubernetes.io/projected/a97e64ec-670e-4bee-99a1-b681a3ea1328-kube-api-access-2xqtn\") pod \"maas-api-revoke-sub-ca4e901d-cc9zl\" (UID: \"a97e64ec-670e-4bee-99a1-b681a3ea1328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl" Oct 30 16:42:17.349475 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.349253 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97e64ec-670e-4bee-99a1-b681a3ea1328-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca4e901d-cc9zl\" (UID: \"a97e64ec-670e-4bee-99a1-b681a3ea1328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl" Oct 30 16:42:17.349724 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.349705 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97e64ec-670e-4bee-99a1-b681a3ea1328-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca4e901d-cc9zl\" (UID: \"a97e64ec-670e-4bee-99a1-b681a3ea1328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl" Oct 30 16:42:17.351975 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.351950 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97e64ec-670e-4bee-99a1-b681a3ea1328-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca4e901d-cc9zl\" (UID: \"a97e64ec-670e-4bee-99a1-b681a3ea1328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl" Oct 30 16:42:17.373362 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.373329 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xqtn\" (UniqueName: \"kubernetes.io/projected/a97e64ec-670e-4bee-99a1-b681a3ea1328-kube-api-access-2xqtn\") pod \"maas-api-revoke-sub-ca4e901d-cc9zl\" (UID: \"a97e64ec-670e-4bee-99a1-b681a3ea1328\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl" Oct 30 16:42:17.503605 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.503523 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl" Oct 30 16:42:17.795460 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.795383 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:17.830172 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:42:17.830141 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97e64ec_670e_4bee_99a1_b681a3ea1328.slice/crio-6ead2dcdffff5bf2d64a78df4114a7407e4d47e6a412324fc8c16be0bbdd6df1 WatchSource:0}: Error finding container 6ead2dcdffff5bf2d64a78df4114a7407e4d47e6a412324fc8c16be0bbdd6df1: Status 404 returned error can't find the container with id 6ead2dcdffff5bf2d64a78df4114a7407e4d47e6a412324fc8c16be0bbdd6df1 Oct 30 16:42:17.832150 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:17.832124 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl"] Oct 30 16:42:18.236232 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:18.236163 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:18.414846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:18.414811 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7000c6f9-beba-4fda-aebc-ce1098c7ee41" path="/var/lib/kubelet/pods/7000c6f9-beba-4fda-aebc-ce1098c7ee41/volumes" Oct 30 16:42:18.503217 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:18.503171 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl" event={"ID":"a97e64ec-670e-4bee-99a1-b681a3ea1328","Type":"ContainerStarted","Data":"1c64539f30147156f9f4d83670100659ebce0eae0423adb486ed74be2468faee"} Oct 30 16:42:18.503428 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:18.503236 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl" event={"ID":"a97e64ec-670e-4bee-99a1-b681a3ea1328","Type":"ContainerStarted","Data":"6ead2dcdffff5bf2d64a78df4114a7407e4d47e6a412324fc8c16be0bbdd6df1"} Oct 30 16:42:18.576825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:18.569643 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl" podStartSLOduration=1.569621554 podStartE2EDuration="1.569621554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:18.546733204 +0000 UTC m=+1252.257187961" watchObservedRunningTime="2026-10-30 16:42:18.569621554 +0000 UTC m=+1252.280076298" Oct 30 16:42:18.638383 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:18.638318 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:18.776693 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:18.776553 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:18.806223 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:18.806189 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8"] Oct 30 16:42:18.810915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:18.810886 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db2924d-z7mb8"] Oct 30 16:42:18.895743 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:18.895710 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:19.014296 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:19.014256 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:19.514933 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:19.514898 6202 generic.go:356] "Generic (PLEG): container finished" podID="a97e64ec-670e-4bee-99a1-b681a3ea1328" containerID="1c64539f30147156f9f4d83670100659ebce0eae0423adb486ed74be2468faee" exitCode=0 Oct 30 16:42:19.515392 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:19.514981 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl" event={"ID":"a97e64ec-670e-4bee-99a1-b681a3ea1328","Type":"ContainerDied","Data":"1c64539f30147156f9f4d83670100659ebce0eae0423adb486ed74be2468faee"} Oct 30 16:42:20.009559 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:20.009518 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:20.127707 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:20.124644 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:20.246643 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:20.246502 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:20.326882 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:42:20.326841 6202 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice/crio-647c759f8fdeb97871011460477612086d143d400bebcaba160fb016d105cf43\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9db09_b582_470c_bcb2_b9a22839fc71.slice\": RecentStats: unable to find data in memory cache]" Oct 30 16:42:20.416575 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:20.416544 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d10705-542d-4993-9657-c51e0da16e31" path="/var/lib/kubelet/pods/b0d10705-542d-4993-9657-c51e0da16e31/volumes" Oct 30 16:42:20.691961 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:20.691925 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl" Oct 30 16:42:20.713889 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:20.713858 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:20.774084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:20.772914 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a97e64ec-670e-4bee-99a1-b681a3ea1328-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97e64ec-670e-4bee-99a1-b681a3ea1328-maas-api-xks-ca\") pod \"a97e64ec-670e-4bee-99a1-b681a3ea1328\" (UID: \"a97e64ec-670e-4bee-99a1-b681a3ea1328\") " Oct 30 16:42:20.774084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:20.773006 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97e64ec-670e-4bee-99a1-b681a3ea1328-kube-api-access-2xqtn\" (UniqueName: \"kubernetes.io/projected/a97e64ec-670e-4bee-99a1-b681a3ea1328-kube-api-access-2xqtn\") pod \"a97e64ec-670e-4bee-99a1-b681a3ea1328\" (UID: \"a97e64ec-670e-4bee-99a1-b681a3ea1328\") " Oct 30 16:42:20.774084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:20.773046 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a97e64ec-670e-4bee-99a1-b681a3ea1328-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97e64ec-670e-4bee-99a1-b681a3ea1328-maas-api-openshift-ca\") pod \"a97e64ec-670e-4bee-99a1-b681a3ea1328\" (UID: \"a97e64ec-670e-4bee-99a1-b681a3ea1328\") " Oct 30 16:42:20.774417 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:20.774220 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a97e64ec-670e-4bee-99a1-b681a3ea1328-maas-api-openshift-ca" pod "a97e64ec-670e-4bee-99a1-b681a3ea1328" (UID: "a97e64ec-670e-4bee-99a1-b681a3ea1328"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:20.802162 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:20.802105 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97e64ec-670e-4bee-99a1-b681a3ea1328-kube-api-access-2xqtn" pod "a97e64ec-670e-4bee-99a1-b681a3ea1328" (UID: "a97e64ec-670e-4bee-99a1-b681a3ea1328"). InnerVolumeSpecName "kube-api-access-2xqtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:20.814274 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:20.814227 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a97e64ec-670e-4bee-99a1-b681a3ea1328-maas-api-xks-ca" pod "a97e64ec-670e-4bee-99a1-b681a3ea1328" (UID: "a97e64ec-670e-4bee-99a1-b681a3ea1328"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:20.875562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:20.875462 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97e64ec-670e-4bee-99a1-b681a3ea1328-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:20.875562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:20.875498 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xqtn\" (UniqueName: \"kubernetes.io/projected/a97e64ec-670e-4bee-99a1-b681a3ea1328-kube-api-access-2xqtn\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:20.875562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:20.875513 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97e64ec-670e-4bee-99a1-b681a3ea1328-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:21.534969 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:21.534936 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl" Oct 30 16:42:21.534969 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:21.534968 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl" event={"ID":"a97e64ec-670e-4bee-99a1-b681a3ea1328","Type":"ContainerDied","Data":"6ead2dcdffff5bf2d64a78df4114a7407e4d47e6a412324fc8c16be0bbdd6df1"} Oct 30 16:42:21.534969 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:21.535019 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ead2dcdffff5bf2d64a78df4114a7407e4d47e6a412324fc8c16be0bbdd6df1" Oct 30 16:42:22.031328 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.031291 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m"] Oct 30 16:42:22.032059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.032032 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a97e64ec-670e-4bee-99a1-b681a3ea1328" containerName="revoke-keys" Oct 30 16:42:22.032059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.032055 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97e64ec-670e-4bee-99a1-b681a3ea1328" containerName="revoke-keys" Oct 30 16:42:22.032226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.032209 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97e64ec-670e-4bee-99a1-b681a3ea1328" containerName="revoke-keys" Oct 30 16:42:22.036320 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.036301 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m" Oct 30 16:42:22.056376 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.056346 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m"] Oct 30 16:42:22.061654 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.061629 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:42:22.061815 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.061629 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:42:22.061815 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.061664 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:42:22.062416 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.062400 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:42:22.096061 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.096020 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71ead75-5cbc-475e-9ade-8406f28427f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5101796f-8847m\" (UID: \"d71ead75-5cbc-475e-9ade-8406f28427f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m" Oct 30 16:42:22.096248 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.096077 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71ead75-5cbc-475e-9ade-8406f28427f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5101796f-8847m\" (UID: \"d71ead75-5cbc-475e-9ade-8406f28427f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m" Oct 30 16:42:22.096248 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.096192 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqddb\" (UniqueName: \"kubernetes.io/projected/d71ead75-5cbc-475e-9ade-8406f28427f7-kube-api-access-jqddb\") pod \"maas-api-revoke-sub-5101796f-8847m\" (UID: \"d71ead75-5cbc-475e-9ade-8406f28427f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m" Oct 30 16:42:22.197272 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.197232 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71ead75-5cbc-475e-9ade-8406f28427f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5101796f-8847m\" (UID: \"d71ead75-5cbc-475e-9ade-8406f28427f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m" Oct 30 16:42:22.197475 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.197297 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71ead75-5cbc-475e-9ade-8406f28427f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5101796f-8847m\" (UID: \"d71ead75-5cbc-475e-9ade-8406f28427f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m" Oct 30 16:42:22.197475 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.197348 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqddb\" (UniqueName: \"kubernetes.io/projected/d71ead75-5cbc-475e-9ade-8406f28427f7-kube-api-access-jqddb\") pod \"maas-api-revoke-sub-5101796f-8847m\" (UID: \"d71ead75-5cbc-475e-9ade-8406f28427f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m" Oct 30 16:42:22.197882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.197861 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71ead75-5cbc-475e-9ade-8406f28427f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5101796f-8847m\" (UID: \"d71ead75-5cbc-475e-9ade-8406f28427f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m" Oct 30 16:42:22.199975 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.199950 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71ead75-5cbc-475e-9ade-8406f28427f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5101796f-8847m\" (UID: \"d71ead75-5cbc-475e-9ade-8406f28427f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m" Oct 30 16:42:22.220682 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.220657 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqddb\" (UniqueName: \"kubernetes.io/projected/d71ead75-5cbc-475e-9ade-8406f28427f7-kube-api-access-jqddb\") pod \"maas-api-revoke-sub-5101796f-8847m\" (UID: \"d71ead75-5cbc-475e-9ade-8406f28427f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m" Oct 30 16:42:22.358885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.358849 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m" Oct 30 16:42:22.360886 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.360857 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:22.707507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:22.707466 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m"] Oct 30 16:42:22.707642 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:42:22.707501 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71ead75_5cbc_475e_9ade_8406f28427f7.slice/crio-19808864137a75c87a3b83ce15aa392253085610da3ab69ce2e62e64d7593687 WatchSource:0}: Error finding container 19808864137a75c87a3b83ce15aa392253085610da3ab69ce2e62e64d7593687: Status 404 returned error can't find the container with id 19808864137a75c87a3b83ce15aa392253085610da3ab69ce2e62e64d7593687 Oct 30 16:42:23.570971 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:23.570925 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m" event={"ID":"d71ead75-5cbc-475e-9ade-8406f28427f7","Type":"ContainerStarted","Data":"23f462464cd1994eadf9e94f43b20228dc12cfd18d66481de09abf9a948ff015"} Oct 30 16:42:23.571440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:23.571009 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m" event={"ID":"d71ead75-5cbc-475e-9ade-8406f28427f7","Type":"ContainerStarted","Data":"19808864137a75c87a3b83ce15aa392253085610da3ab69ce2e62e64d7593687"} Oct 30 16:42:23.616461 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:23.616403 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m" podStartSLOduration=1.616389058 podStartE2EDuration="1.616389058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:23.612097718 +0000 UTC m=+1257.322552461" watchObservedRunningTime="2026-10-30 16:42:23.616389058 +0000 UTC m=+1257.326843802" Oct 30 16:42:24.006428 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:24.006373 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:24.211988 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:24.211895 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:24.583255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:24.583209 6202 generic.go:356] "Generic (PLEG): container finished" podID="d71ead75-5cbc-475e-9ade-8406f28427f7" containerID="23f462464cd1994eadf9e94f43b20228dc12cfd18d66481de09abf9a948ff015" exitCode=0 Oct 30 16:42:24.583726 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:24.583286 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m" event={"ID":"d71ead75-5cbc-475e-9ade-8406f28427f7","Type":"ContainerDied","Data":"23f462464cd1994eadf9e94f43b20228dc12cfd18d66481de09abf9a948ff015"} Oct 30 16:42:25.341626 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:25.341584 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:25.935850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:25.917389 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m" Oct 30 16:42:25.994804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:25.972138 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d71ead75-5cbc-475e-9ade-8406f28427f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71ead75-5cbc-475e-9ade-8406f28427f7-maas-api-xks-ca\") pod \"d71ead75-5cbc-475e-9ade-8406f28427f7\" (UID: \"d71ead75-5cbc-475e-9ade-8406f28427f7\") " Oct 30 16:42:25.994804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:25.972221 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71ead75-5cbc-475e-9ade-8406f28427f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71ead75-5cbc-475e-9ade-8406f28427f7-maas-api-openshift-ca\") pod \"d71ead75-5cbc-475e-9ade-8406f28427f7\" (UID: \"d71ead75-5cbc-475e-9ade-8406f28427f7\") " Oct 30 16:42:25.994804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:25.972263 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71ead75-5cbc-475e-9ade-8406f28427f7-kube-api-access-jqddb\" (UniqueName: \"kubernetes.io/projected/d71ead75-5cbc-475e-9ade-8406f28427f7-kube-api-access-jqddb\") pod \"d71ead75-5cbc-475e-9ade-8406f28427f7\" (UID: \"d71ead75-5cbc-475e-9ade-8406f28427f7\") " Oct 30 16:42:26.000328 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:26.000264 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71ead75-5cbc-475e-9ade-8406f28427f7-maas-api-openshift-ca" pod "d71ead75-5cbc-475e-9ade-8406f28427f7" (UID: "d71ead75-5cbc-475e-9ade-8406f28427f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:26.001150 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:26.001109 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71ead75-5cbc-475e-9ade-8406f28427f7-kube-api-access-jqddb" pod "d71ead75-5cbc-475e-9ade-8406f28427f7" (UID: "d71ead75-5cbc-475e-9ade-8406f28427f7"). InnerVolumeSpecName "kube-api-access-jqddb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:26.012921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:26.011944 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d71ead75-5cbc-475e-9ade-8406f28427f7-maas-api-xks-ca" pod "d71ead75-5cbc-475e-9ade-8406f28427f7" (UID: "d71ead75-5cbc-475e-9ade-8406f28427f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:26.012921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:26.012456 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:26.075789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:26.075301 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71ead75-5cbc-475e-9ade-8406f28427f7-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:26.075789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:26.075339 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71ead75-5cbc-475e-9ade-8406f28427f7-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:26.075789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:26.075354 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqddb\" (UniqueName: \"kubernetes.io/projected/d71ead75-5cbc-475e-9ade-8406f28427f7-kube-api-access-jqddb\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:26.219486 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:26.219407 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:26.302030 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:26.301994 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf"] Oct 30 16:42:26.310869 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:26.310683 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f654570d-wtqlf"] Oct 30 16:42:26.412087 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:26.412055 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="662a08aa-69c4-49cb-9b70-14f59ee316c5" path="/var/lib/kubelet/pods/662a08aa-69c4-49cb-9b70-14f59ee316c5/volumes" Oct 30 16:42:26.425149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:26.425111 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:26.577482 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:26.577447 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:26.614474 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:26.614431 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m" event={"ID":"d71ead75-5cbc-475e-9ade-8406f28427f7","Type":"ContainerDied","Data":"19808864137a75c87a3b83ce15aa392253085610da3ab69ce2e62e64d7593687"} Oct 30 16:42:26.614687 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:26.614491 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19808864137a75c87a3b83ce15aa392253085610da3ab69ce2e62e64d7593687" Oct 30 16:42:26.614687 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:26.614565 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m" Oct 30 16:42:27.185402 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.185361 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:27.591631 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.591591 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw"] Oct 30 16:42:27.592368 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.592350 6202 cpu_manager.go:414] "containerMap: removing container" podUID="d71ead75-5cbc-475e-9ade-8406f28427f7" containerName="revoke-keys" Oct 30 16:42:27.592414 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.592370 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71ead75-5cbc-475e-9ade-8406f28427f7" containerName="revoke-keys" Oct 30 16:42:27.592516 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.592507 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71ead75-5cbc-475e-9ade-8406f28427f7" containerName="revoke-keys" Oct 30 16:42:27.596997 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.596979 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw" Oct 30 16:42:27.612521 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.612496 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:42:27.620018 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.619981 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw"] Oct 30 16:42:27.624662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.624624 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:42:27.624870 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.624809 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:42:27.705551 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.705512 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:27.708824 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.708781 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:42:27.710965 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.710937 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ed0399-456d-4278-bb7d-0265626a65c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd47dcfc-4bjzw\" (UID: \"11ed0399-456d-4278-bb7d-0265626a65c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw" Oct 30 16:42:27.711122 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.711094 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ed0399-456d-4278-bb7d-0265626a65c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd47dcfc-4bjzw\" (UID: \"11ed0399-456d-4278-bb7d-0265626a65c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw" Oct 30 16:42:27.711245 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.711223 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsp56\" (UniqueName: \"kubernetes.io/projected/11ed0399-456d-4278-bb7d-0265626a65c6-kube-api-access-hsp56\") pod \"maas-api-revoke-sub-cd47dcfc-4bjzw\" (UID: \"11ed0399-456d-4278-bb7d-0265626a65c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw" Oct 30 16:42:27.813165 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.813125 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ed0399-456d-4278-bb7d-0265626a65c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd47dcfc-4bjzw\" (UID: \"11ed0399-456d-4278-bb7d-0265626a65c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw" Oct 30 16:42:27.820567 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.813220 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ed0399-456d-4278-bb7d-0265626a65c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd47dcfc-4bjzw\" (UID: \"11ed0399-456d-4278-bb7d-0265626a65c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw" Oct 30 16:42:27.820567 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.813288 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsp56\" (UniqueName: \"kubernetes.io/projected/11ed0399-456d-4278-bb7d-0265626a65c6-kube-api-access-hsp56\") pod \"maas-api-revoke-sub-cd47dcfc-4bjzw\" (UID: \"11ed0399-456d-4278-bb7d-0265626a65c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw" Oct 30 16:42:27.820567 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.814022 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ed0399-456d-4278-bb7d-0265626a65c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd47dcfc-4bjzw\" (UID: \"11ed0399-456d-4278-bb7d-0265626a65c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw" Oct 30 16:42:27.822051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.822014 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ed0399-456d-4278-bb7d-0265626a65c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd47dcfc-4bjzw\" (UID: \"11ed0399-456d-4278-bb7d-0265626a65c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw" Oct 30 16:42:27.839206 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.839135 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsp56\" (UniqueName: \"kubernetes.io/projected/11ed0399-456d-4278-bb7d-0265626a65c6-kube-api-access-hsp56\") pod \"maas-api-revoke-sub-cd47dcfc-4bjzw\" (UID: \"11ed0399-456d-4278-bb7d-0265626a65c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw" Oct 30 16:42:27.915116 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.914760 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw" Oct 30 16:42:27.916132 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:27.915556 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:28.354367 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:42:28.354335 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ed0399_456d_4278_bb7d_0265626a65c6.slice/crio-b726e6dd05082cc3fc815443fb218f26f0bde4a630d9566fe21787463db33896 WatchSource:0}: Error finding container b726e6dd05082cc3fc815443fb218f26f0bde4a630d9566fe21787463db33896: Status 404 returned error can't find the container with id b726e6dd05082cc3fc815443fb218f26f0bde4a630d9566fe21787463db33896 Oct 30 16:42:28.357103 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:28.357075 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw"] Oct 30 16:42:28.565691 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:28.565655 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:28.645065 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:28.644971 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw" event={"ID":"11ed0399-456d-4278-bb7d-0265626a65c6","Type":"ContainerStarted","Data":"b726e6dd05082cc3fc815443fb218f26f0bde4a630d9566fe21787463db33896"} Oct 30 16:42:29.543653 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:29.543563 6202 scope.go:122] "RemoveContainer" containerID="c178c24e1666c68a9bc9a216cecb2f867d591b93259d1ef6c56f546e641a79c1" Oct 30 16:42:29.555506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:29.555404 6202 scope.go:122] "RemoveContainer" containerID="eed38e814845849b028a9145efa899f47b4c1acf2efcd0cb6d127123679e9bb3" Oct 30 16:42:29.568790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:29.568743 6202 scope.go:122] "RemoveContainer" containerID="62b7090a4f9c2201761bd0e3e2fe34ff1ee4ad3161182b1271ee06dd8dec6514" Oct 30 16:42:29.580163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:29.580126 6202 scope.go:122] "RemoveContainer" containerID="5bba2e40d0a4c474f69f51a65492b2852653daff2910d612dab11603b963b638" Oct 30 16:42:29.592154 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:29.592128 6202 scope.go:122] "RemoveContainer" containerID="5314d5936cdb7b4e11a37cfd7ac1ecb1d56f9c7a1e19530e05b5228d2a8af33e" Oct 30 16:42:29.603001 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:29.602980 6202 scope.go:122] "RemoveContainer" containerID="ef2f967928df90f5b291e3357d67f4f9d98e7b5a9b96f79414b0cebcdc0fa2ba" Oct 30 16:42:29.613260 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:29.613236 6202 scope.go:122] "RemoveContainer" containerID="f288e574052a39e70dd5aa1a40ea6cd8ea655bc73446af0ef81c59a07705b1eb" Oct 30 16:42:29.623965 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:29.623929 6202 scope.go:122] "RemoveContainer" containerID="ff1364fd1a559a48a01974304fd15d6eb296c095fff54d67258d29c40bf99260" Oct 30 16:42:29.635166 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:29.635144 6202 scope.go:122] "RemoveContainer" containerID="c4618b80ad59ba60bb3366c5aaff0fea4dbeb28d222a18d2e741964e1495b7ba" Oct 30 16:42:29.689872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:29.689748 6202 generic.go:356] "Generic (PLEG): container finished" podID="11ed0399-456d-4278-bb7d-0265626a65c6" containerID="00ba40e387d45078730fea9b84522c0e099644976b55d1a2295b1b609ca2a3fd" exitCode=0 Oct 30 16:42:29.689948 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:29.689908 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw" event={"ID":"11ed0399-456d-4278-bb7d-0265626a65c6","Type":"ContainerDied","Data":"00ba40e387d45078730fea9b84522c0e099644976b55d1a2295b1b609ca2a3fd"} Oct 30 16:42:30.418420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:30.418375 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:30.892277 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:30.892250 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw" Oct 30 16:42:30.976118 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:30.976007 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11ed0399-456d-4278-bb7d-0265626a65c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ed0399-456d-4278-bb7d-0265626a65c6-maas-api-xks-ca\") pod \"11ed0399-456d-4278-bb7d-0265626a65c6\" (UID: \"11ed0399-456d-4278-bb7d-0265626a65c6\") " Oct 30 16:42:30.976313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:30.976219 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11ed0399-456d-4278-bb7d-0265626a65c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ed0399-456d-4278-bb7d-0265626a65c6-maas-api-openshift-ca\") pod \"11ed0399-456d-4278-bb7d-0265626a65c6\" (UID: \"11ed0399-456d-4278-bb7d-0265626a65c6\") " Oct 30 16:42:30.976377 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:30.976331 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ed0399-456d-4278-bb7d-0265626a65c6-kube-api-access-hsp56\" (UniqueName: \"kubernetes.io/projected/11ed0399-456d-4278-bb7d-0265626a65c6-kube-api-access-hsp56\") pod \"11ed0399-456d-4278-bb7d-0265626a65c6\" (UID: \"11ed0399-456d-4278-bb7d-0265626a65c6\") " Oct 30 16:42:30.977151 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:30.977120 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ed0399-456d-4278-bb7d-0265626a65c6-maas-api-openshift-ca" pod "11ed0399-456d-4278-bb7d-0265626a65c6" (UID: "11ed0399-456d-4278-bb7d-0265626a65c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:30.979907 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:30.979876 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ed0399-456d-4278-bb7d-0265626a65c6-kube-api-access-hsp56" pod "11ed0399-456d-4278-bb7d-0265626a65c6" (UID: "11ed0399-456d-4278-bb7d-0265626a65c6"). InnerVolumeSpecName "kube-api-access-hsp56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:30.980253 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:30.980222 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ed0399-456d-4278-bb7d-0265626a65c6-maas-api-xks-ca" pod "11ed0399-456d-4278-bb7d-0265626a65c6" (UID: "11ed0399-456d-4278-bb7d-0265626a65c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:31.079102 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:31.079064 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ed0399-456d-4278-bb7d-0265626a65c6-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:31.079102 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:31.079102 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsp56\" (UniqueName: \"kubernetes.io/projected/11ed0399-456d-4278-bb7d-0265626a65c6-kube-api-access-hsp56\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:31.079315 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:31.079118 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ed0399-456d-4278-bb7d-0265626a65c6-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:31.347236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:31.347192 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:31.710420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:31.710336 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw" Oct 30 16:42:31.710420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:31.710382 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw" event={"ID":"11ed0399-456d-4278-bb7d-0265626a65c6","Type":"ContainerDied","Data":"b726e6dd05082cc3fc815443fb218f26f0bde4a630d9566fe21787463db33896"} Oct 30 16:42:31.710651 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:31.710430 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b726e6dd05082cc3fc815443fb218f26f0bde4a630d9566fe21787463db33896" Oct 30 16:42:32.240876 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:32.240839 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:32.356021 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:32.355989 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:32.481440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:32.481404 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:32.616209 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:32.616157 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:33.328135 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.328101 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn"] Oct 30 16:42:33.328888 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.328871 6202 cpu_manager.go:414] "containerMap: removing container" podUID="11ed0399-456d-4278-bb7d-0265626a65c6" containerName="revoke-keys" Oct 30 16:42:33.328940 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.328889 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ed0399-456d-4278-bb7d-0265626a65c6" containerName="revoke-keys" Oct 30 16:42:33.329015 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.329005 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ed0399-456d-4278-bb7d-0265626a65c6" containerName="revoke-keys" Oct 30 16:42:33.333068 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.333052 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn" Oct 30 16:42:33.354955 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.354864 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn"] Oct 30 16:42:33.361522 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.361489 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:42:33.362113 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.362076 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:42:33.362291 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.362273 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:42:33.362401 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.362017 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:42:33.416167 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.416131 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aacddea-2a95-4741-88a6-633744acc134-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn\" (UID: \"1aacddea-2a95-4741-88a6-633744acc134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn" Oct 30 16:42:33.416359 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.416302 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aacddea-2a95-4741-88a6-633744acc134-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn\" (UID: \"1aacddea-2a95-4741-88a6-633744acc134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn" Oct 30 16:42:33.416442 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.416422 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vllxn\" (UniqueName: \"kubernetes.io/projected/1aacddea-2a95-4741-88a6-633744acc134-kube-api-access-vllxn\") pod \"maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn\" (UID: \"1aacddea-2a95-4741-88a6-633744acc134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn" Oct 30 16:42:33.518057 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.518020 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aacddea-2a95-4741-88a6-633744acc134-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn\" (UID: \"1aacddea-2a95-4741-88a6-633744acc134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn" Oct 30 16:42:33.518236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.518140 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aacddea-2a95-4741-88a6-633744acc134-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn\" (UID: \"1aacddea-2a95-4741-88a6-633744acc134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn" Oct 30 16:42:33.518236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.518209 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vllxn\" (UniqueName: \"kubernetes.io/projected/1aacddea-2a95-4741-88a6-633744acc134-kube-api-access-vllxn\") pod \"maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn\" (UID: \"1aacddea-2a95-4741-88a6-633744acc134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn" Oct 30 16:42:33.518661 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.518638 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aacddea-2a95-4741-88a6-633744acc134-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn\" (UID: \"1aacddea-2a95-4741-88a6-633744acc134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn" Oct 30 16:42:33.520739 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.520713 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aacddea-2a95-4741-88a6-633744acc134-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn\" (UID: \"1aacddea-2a95-4741-88a6-633744acc134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn" Oct 30 16:42:33.542259 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.542226 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vllxn\" (UniqueName: \"kubernetes.io/projected/1aacddea-2a95-4741-88a6-633744acc134-kube-api-access-vllxn\") pod \"maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn\" (UID: \"1aacddea-2a95-4741-88a6-633744acc134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn" Oct 30 16:42:33.649137 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.649028 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn" Oct 30 16:42:33.982611 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:42:33.982577 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aacddea_2a95_4741_88a6_633744acc134.slice/crio-0f9c4cc22c8a93765bec3fdd82853ce6d0674724a203493d61c7bdeedd281a35 WatchSource:0}: Error finding container 0f9c4cc22c8a93765bec3fdd82853ce6d0674724a203493d61c7bdeedd281a35: Status 404 returned error can't find the container with id 0f9c4cc22c8a93765bec3fdd82853ce6d0674724a203493d61c7bdeedd281a35 Oct 30 16:42:33.984134 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:33.984109 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn"] Oct 30 16:42:34.261121 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:34.236293 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:34.357885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:34.357852 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:34.475119 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:34.475090 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:34.761793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:34.759290 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn" event={"ID":"1aacddea-2a95-4741-88a6-633744acc134","Type":"ContainerStarted","Data":"4e04dae1366e2ee822f5ff3f2683a49c71bf70caeca3ca03a32e02a1ec1cf361"} Oct 30 16:42:34.761793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:34.759356 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn" event={"ID":"1aacddea-2a95-4741-88a6-633744acc134","Type":"ContainerStarted","Data":"0f9c4cc22c8a93765bec3fdd82853ce6d0674724a203493d61c7bdeedd281a35"} Oct 30 16:42:34.807247 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:34.807179 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn" podStartSLOduration=1.807161604 podStartE2EDuration="1.807161604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:34.80467297 +0000 UTC m=+1268.515127720" watchObservedRunningTime="2026-10-30 16:42:34.807161604 +0000 UTC m=+1268.517616349" Oct 30 16:42:35.842324 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:35.842116 6202 generic.go:356] "Generic (PLEG): container finished" podID="1aacddea-2a95-4741-88a6-633744acc134" containerID="4e04dae1366e2ee822f5ff3f2683a49c71bf70caeca3ca03a32e02a1ec1cf361" exitCode=0 Oct 30 16:42:35.842972 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:35.842945 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn" event={"ID":"1aacddea-2a95-4741-88a6-633744acc134","Type":"ContainerDied","Data":"4e04dae1366e2ee822f5ff3f2683a49c71bf70caeca3ca03a32e02a1ec1cf361"} Oct 30 16:42:37.309845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:37.308923 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:37.380792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:37.379860 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:37.479790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:37.471867 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:38.084363 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:38.045436 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 16:42:38.084363 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:38.048645 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:38.084363 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:38.066354 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:38.193907 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:38.085171 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:38.193907 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:38.150095 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:38.385699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:38.356245 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:42:38.385699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:38.364453 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:38.385699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:38.364805 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:38.402799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:38.388985 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:42:38.649896 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:38.509268 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:42:39.027643 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:38.654590 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:39.032197 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:39.032157 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 16:42:39.069783 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:39.069721 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:39.112621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:39.112577 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:39.170867 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:39.151812 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:39.393660 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:39.393615 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:39.394457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:39.393545 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:39.589859 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:39.583171 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:39.839852 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:42:39.820611 6202 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.407s" Oct 30 16:42:39.840690 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:39.840664 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:42:39.840949 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:39.840936 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:42:39.841139 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:39.841126 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:42:39.841321 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:39.841307 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:42:39.841485 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:39.841472 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:42:39.841666 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:39.841652 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:42:39.855750 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:39.855727 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn" Oct 30 16:42:40.051793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.050726 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:42:40.051793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.051376 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:42:40.089006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.063910 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 16:42:40.132321 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.132237 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aacddea-2a95-4741-88a6-633744acc134-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aacddea-2a95-4741-88a6-633744acc134-maas-api-openshift-ca\") pod \"1aacddea-2a95-4741-88a6-633744acc134\" (UID: \"1aacddea-2a95-4741-88a6-633744acc134\") " Oct 30 16:42:40.132744 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.132718 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aacddea-2a95-4741-88a6-633744acc134-kube-api-access-vllxn\" (UniqueName: \"kubernetes.io/projected/1aacddea-2a95-4741-88a6-633744acc134-kube-api-access-vllxn\") pod \"1aacddea-2a95-4741-88a6-633744acc134\" (UID: \"1aacddea-2a95-4741-88a6-633744acc134\") " Oct 30 16:42:40.133028 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.133005 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aacddea-2a95-4741-88a6-633744acc134-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aacddea-2a95-4741-88a6-633744acc134-maas-api-xks-ca\") pod \"1aacddea-2a95-4741-88a6-633744acc134\" (UID: \"1aacddea-2a95-4741-88a6-633744acc134\") " Oct 30 16:42:40.133972 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.133940 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:40.224210 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.136912 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aacddea-2a95-4741-88a6-633744acc134-maas-api-openshift-ca" pod "1aacddea-2a95-4741-88a6-633744acc134" (UID: "1aacddea-2a95-4741-88a6-633744acc134"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:40.370835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.364552 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aacddea-2a95-4741-88a6-633744acc134-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:40.504842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.500223 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aacddea-2a95-4741-88a6-633744acc134-kube-api-access-vllxn" pod "1aacddea-2a95-4741-88a6-633744acc134" (UID: "1aacddea-2a95-4741-88a6-633744acc134"). InnerVolumeSpecName "kube-api-access-vllxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:40.523585 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.523550 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vllxn\" (UniqueName: \"kubernetes.io/projected/1aacddea-2a95-4741-88a6-633744acc134-kube-api-access-vllxn\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:40.597788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.597286 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 16:42:40.734106 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.645283 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 16:42:40.734106 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.729370 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 16:42:40.794878 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.788942 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 16:42:40.794878 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.789001 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:42:40.794878 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.789034 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:42:40.794878 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.789068 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:42:40.829209 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:40.829170 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:41.032491 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:41.032444 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aacddea-2a95-4741-88a6-633744acc134-maas-api-xks-ca" pod "1aacddea-2a95-4741-88a6-633744acc134" (UID: "1aacddea-2a95-4741-88a6-633744acc134"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:41.034331 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:41.034307 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 16:42:41.091475 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:41.091447 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 16:42:41.101792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:41.100861 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:41.114259 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:41.114221 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aacddea-2a95-4741-88a6-633744acc134-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:41.122881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:41.122673 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:42:41.130036 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:41.130014 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 16:42:41.132309 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:41.132285 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn" event={"ID":"1aacddea-2a95-4741-88a6-633744acc134","Type":"ContainerDied","Data":"0f9c4cc22c8a93765bec3fdd82853ce6d0674724a203493d61c7bdeedd281a35"} Oct 30 16:42:41.133605 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:41.133566 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f9c4cc22c8a93765bec3fdd82853ce6d0674724a203493d61c7bdeedd281a35" Oct 30 16:42:41.133736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:41.130370 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn" Oct 30 16:42:41.300426 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:41.300389 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:41.436786 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:41.436685 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:41.553797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:41.549613 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:42.281937 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.281904 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52"] Oct 30 16:42:42.282661 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.282646 6202 cpu_manager.go:414] "containerMap: removing container" podUID="1aacddea-2a95-4741-88a6-633744acc134" containerName="revoke-keys" Oct 30 16:42:42.282706 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.282664 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aacddea-2a95-4741-88a6-633744acc134" containerName="revoke-keys" Oct 30 16:42:42.282792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.282782 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aacddea-2a95-4741-88a6-633744acc134" containerName="revoke-keys" Oct 30 16:42:42.286606 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.286591 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52" Oct 30 16:42:42.307806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.307739 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52"] Oct 30 16:42:42.312091 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.312069 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:42:42.312269 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.312245 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:42:42.312361 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.312346 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:42:42.312834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.312814 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:42:42.368972 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.368933 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx7l6\" (UniqueName: \"kubernetes.io/projected/60339ae6-3e91-4370-9ea1-a7542d77486b-kube-api-access-kx7l6\") pod \"maas-api-revoke-sub-3f747af2-c7k52\" (UID: \"60339ae6-3e91-4370-9ea1-a7542d77486b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52" Oct 30 16:42:42.369170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.369037 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60339ae6-3e91-4370-9ea1-a7542d77486b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f747af2-c7k52\" (UID: \"60339ae6-3e91-4370-9ea1-a7542d77486b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52" Oct 30 16:42:42.369170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.369118 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60339ae6-3e91-4370-9ea1-a7542d77486b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f747af2-c7k52\" (UID: \"60339ae6-3e91-4370-9ea1-a7542d77486b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52" Oct 30 16:42:42.374949 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.374907 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:42.470472 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.470437 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx7l6\" (UniqueName: \"kubernetes.io/projected/60339ae6-3e91-4370-9ea1-a7542d77486b-kube-api-access-kx7l6\") pod \"maas-api-revoke-sub-3f747af2-c7k52\" (UID: \"60339ae6-3e91-4370-9ea1-a7542d77486b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52" Oct 30 16:42:42.470669 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.470531 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60339ae6-3e91-4370-9ea1-a7542d77486b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f747af2-c7k52\" (UID: \"60339ae6-3e91-4370-9ea1-a7542d77486b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52" Oct 30 16:42:42.470669 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.470660 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60339ae6-3e91-4370-9ea1-a7542d77486b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f747af2-c7k52\" (UID: \"60339ae6-3e91-4370-9ea1-a7542d77486b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52" Oct 30 16:42:42.471225 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.471198 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60339ae6-3e91-4370-9ea1-a7542d77486b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f747af2-c7k52\" (UID: \"60339ae6-3e91-4370-9ea1-a7542d77486b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52" Oct 30 16:42:42.473212 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.473192 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60339ae6-3e91-4370-9ea1-a7542d77486b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f747af2-c7k52\" (UID: \"60339ae6-3e91-4370-9ea1-a7542d77486b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52" Oct 30 16:42:42.494228 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.494201 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx7l6\" (UniqueName: \"kubernetes.io/projected/60339ae6-3e91-4370-9ea1-a7542d77486b-kube-api-access-kx7l6\") pod \"maas-api-revoke-sub-3f747af2-c7k52\" (UID: \"60339ae6-3e91-4370-9ea1-a7542d77486b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52" Oct 30 16:42:42.600856 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.600822 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52" Oct 30 16:42:42.934899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:42.934866 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52"] Oct 30 16:42:42.937032 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:42:42.937001 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60339ae6_3e91_4370_9ea1_a7542d77486b.slice/crio-329e1f0dbb75f74b5204bb28ee72840e8fd13e8af6b72d92a8a204438a902adf WatchSource:0}: Error finding container 329e1f0dbb75f74b5204bb28ee72840e8fd13e8af6b72d92a8a204438a902adf: Status 404 returned error can't find the container with id 329e1f0dbb75f74b5204bb28ee72840e8fd13e8af6b72d92a8a204438a902adf Oct 30 16:42:43.175523 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:43.175151 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52" event={"ID":"60339ae6-3e91-4370-9ea1-a7542d77486b","Type":"ContainerStarted","Data":"329e1f0dbb75f74b5204bb28ee72840e8fd13e8af6b72d92a8a204438a902adf"} Oct 30 16:42:43.180885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:43.180801 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:43.769232 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:43.769194 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6"] Oct 30 16:42:43.901514 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:43.901446 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:44.018135 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:44.017349 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:44.194058 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:42:44.188427 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:42:44.196639 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:44.196605 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52" event={"ID":"60339ae6-3e91-4370-9ea1-a7542d77486b","Type":"ContainerStarted","Data":"4183ca445e22e0e5bb78b17f76eeb6b75e3f2fcda0a99274cab8ef05f75114a7"} Oct 30 16:42:45.073214 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:45.073180 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:45.213126 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:45.213082 6202 generic.go:356] "Generic (PLEG): container finished" podID="60339ae6-3e91-4370-9ea1-a7542d77486b" containerID="4183ca445e22e0e5bb78b17f76eeb6b75e3f2fcda0a99274cab8ef05f75114a7" exitCode=0 Oct 30 16:42:45.213450 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:45.213392 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52" event={"ID":"60339ae6-3e91-4370-9ea1-a7542d77486b","Type":"ContainerDied","Data":"4183ca445e22e0e5bb78b17f76eeb6b75e3f2fcda0a99274cab8ef05f75114a7"} Oct 30 16:42:45.430069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:45.430035 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52" Oct 30 16:42:45.540563 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:45.540528 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60339ae6-3e91-4370-9ea1-a7542d77486b-kube-api-access-kx7l6\" (UniqueName: \"kubernetes.io/projected/60339ae6-3e91-4370-9ea1-a7542d77486b-kube-api-access-kx7l6\") pod \"60339ae6-3e91-4370-9ea1-a7542d77486b\" (UID: \"60339ae6-3e91-4370-9ea1-a7542d77486b\") " Oct 30 16:42:45.540755 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:45.540601 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60339ae6-3e91-4370-9ea1-a7542d77486b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60339ae6-3e91-4370-9ea1-a7542d77486b-maas-api-xks-ca\") pod \"60339ae6-3e91-4370-9ea1-a7542d77486b\" (UID: \"60339ae6-3e91-4370-9ea1-a7542d77486b\") " Oct 30 16:42:45.540892 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:45.540872 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60339ae6-3e91-4370-9ea1-a7542d77486b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60339ae6-3e91-4370-9ea1-a7542d77486b-maas-api-openshift-ca\") pod \"60339ae6-3e91-4370-9ea1-a7542d77486b\" (UID: \"60339ae6-3e91-4370-9ea1-a7542d77486b\") " Oct 30 16:42:45.541183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:45.541154 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60339ae6-3e91-4370-9ea1-a7542d77486b-maas-api-openshift-ca" pod "60339ae6-3e91-4370-9ea1-a7542d77486b" (UID: "60339ae6-3e91-4370-9ea1-a7542d77486b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:45.541704 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:45.541684 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60339ae6-3e91-4370-9ea1-a7542d77486b-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:45.543226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:45.543198 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60339ae6-3e91-4370-9ea1-a7542d77486b-kube-api-access-kx7l6" pod "60339ae6-3e91-4370-9ea1-a7542d77486b" (UID: "60339ae6-3e91-4370-9ea1-a7542d77486b"). InnerVolumeSpecName "kube-api-access-kx7l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:45.543934 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:45.543908 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60339ae6-3e91-4370-9ea1-a7542d77486b-maas-api-xks-ca" pod "60339ae6-3e91-4370-9ea1-a7542d77486b" (UID: "60339ae6-3e91-4370-9ea1-a7542d77486b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:45.643063 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:45.642975 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx7l6\" (UniqueName: \"kubernetes.io/projected/60339ae6-3e91-4370-9ea1-a7542d77486b-kube-api-access-kx7l6\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:45.643063 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:45.643004 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60339ae6-3e91-4370-9ea1-a7542d77486b-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:46.231045 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:46.231009 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52" event={"ID":"60339ae6-3e91-4370-9ea1-a7542d77486b","Type":"ContainerDied","Data":"329e1f0dbb75f74b5204bb28ee72840e8fd13e8af6b72d92a8a204438a902adf"} Oct 30 16:42:46.231455 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:46.231068 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="329e1f0dbb75f74b5204bb28ee72840e8fd13e8af6b72d92a8a204438a902adf" Oct 30 16:42:46.231455 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:46.231020 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52" Oct 30 16:42:46.641586 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:46.641531 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:48.110789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.108383 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb"] Oct 30 16:42:48.110789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.109408 6202 cpu_manager.go:414] "containerMap: removing container" podUID="60339ae6-3e91-4370-9ea1-a7542d77486b" containerName="revoke-keys" Oct 30 16:42:48.110789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.109430 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60339ae6-3e91-4370-9ea1-a7542d77486b" containerName="revoke-keys" Oct 30 16:42:48.110789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.109563 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="60339ae6-3e91-4370-9ea1-a7542d77486b" containerName="revoke-keys" Oct 30 16:42:48.126161 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.115467 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.139456 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.138665 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb"] Oct 30 16:42:48.185690 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.185640 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5znlf\" (UniqueName: \"kubernetes.io/projected/45641dd8-8e3a-4a54-872d-c7033e870619-kube-api-access-5znlf\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.185883 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.185714 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45641dd8-8e3a-4a54-872d-c7033e870619-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.185883 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.185859 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.185988 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.185951 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.185988 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.185972 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45641dd8-8e3a-4a54-872d-c7033e870619-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.186082 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.186018 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.186215 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.186198 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.186279 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.186247 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45641dd8-8e3a-4a54-872d-c7033e870619-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.186333 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.186320 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.287584 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.287547 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.298056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.287683 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5znlf\" (UniqueName: \"kubernetes.io/projected/45641dd8-8e3a-4a54-872d-c7033e870619-kube-api-access-5znlf\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.298056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.287744 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45641dd8-8e3a-4a54-872d-c7033e870619-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.298056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.287949 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.298056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.288060 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.298056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.288098 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45641dd8-8e3a-4a54-872d-c7033e870619-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.298056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.288166 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.298056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.288291 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.298056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.288317 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.298056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.288436 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.298056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.288502 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45641dd8-8e3a-4a54-872d-c7033e870619-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.298056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.288550 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.298056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.288685 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.298056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.289399 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45641dd8-8e3a-4a54-872d-c7033e870619-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.298056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.290145 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.298056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.290857 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45641dd8-8e3a-4a54-872d-c7033e870619-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.311688 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.311655 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45641dd8-8e3a-4a54-872d-c7033e870619-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.311875 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.311856 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5znlf\" (UniqueName: \"kubernetes.io/projected/45641dd8-8e3a-4a54-872d-c7033e870619-kube-api-access-5znlf\") pod \"maas-default-gateway-openshift-default-76f4876667-nx9kb\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.369356 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.369327 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:48.450178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.450136 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:48.770465 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.770375 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6" podUID="51ef6485-8b36-4bb2-b643-31b06cdedddd" containerName="payload-processing" containerID="cri-o://521126afbe5eda38b1d1c14944406bf1241d1a0adad558ca533a3e501094096d" gracePeriod=25 Oct 30 16:42:48.782986 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.782946 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:48.806390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.806360 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb"] Oct 30 16:42:48.845227 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:42:48.845190 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45641dd8_8e3a_4a54_872d_c7033e870619.slice/crio-671fc68c4ac352e6e08db6249ffd691ff2be4da305a71b5d29a40aa4f6a4ba43 WatchSource:0}: Error finding container 671fc68c4ac352e6e08db6249ffd691ff2be4da305a71b5d29a40aa4f6a4ba43: Status 404 returned error can't find the container with id 671fc68c4ac352e6e08db6249ffd691ff2be4da305a71b5d29a40aa4f6a4ba43 Oct 30 16:42:48.877591 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.877542 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:48.877706 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.877646 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:48.877706 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:48.877690 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:49.069964 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.069937 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6" Oct 30 16:42:49.203256 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.203214 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ef6485-8b36-4bb2-b643-31b06cdedddd-kube-api-access-k7vfj\" (UniqueName: \"kubernetes.io/projected/51ef6485-8b36-4bb2-b643-31b06cdedddd-kube-api-access-k7vfj\") pod \"51ef6485-8b36-4bb2-b643-31b06cdedddd\" (UID: \"51ef6485-8b36-4bb2-b643-31b06cdedddd\") " Oct 30 16:42:49.203753 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.203723 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51ef6485-8b36-4bb2-b643-31b06cdedddd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ef6485-8b36-4bb2-b643-31b06cdedddd-plugins-config-volume\") pod \"51ef6485-8b36-4bb2-b643-31b06cdedddd\" (UID: \"51ef6485-8b36-4bb2-b643-31b06cdedddd\") " Oct 30 16:42:49.204170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.204141 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ef6485-8b36-4bb2-b643-31b06cdedddd-plugins-config-volume" pod "51ef6485-8b36-4bb2-b643-31b06cdedddd" (UID: "51ef6485-8b36-4bb2-b643-31b06cdedddd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:49.204522 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.204499 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ef6485-8b36-4bb2-b643-31b06cdedddd-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:49.205512 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.205484 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ef6485-8b36-4bb2-b643-31b06cdedddd-kube-api-access-k7vfj" pod "51ef6485-8b36-4bb2-b643-31b06cdedddd" (UID: "51ef6485-8b36-4bb2-b643-31b06cdedddd"). InnerVolumeSpecName "kube-api-access-k7vfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:49.267037 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.267004 6202 generic.go:356] "Generic (PLEG): container finished" podID="e492a729-2008-4c88-a7b9-25caea1a5983" containerID="b3606144e02ad57741bf1d25970e8f45e2e0e8684fd3c8c5a8b5a45359891b8c" exitCode=0 Oct 30 16:42:49.267469 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.267085 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" event={"ID":"e492a729-2008-4c88-a7b9-25caea1a5983","Type":"ContainerDied","Data":"b3606144e02ad57741bf1d25970e8f45e2e0e8684fd3c8c5a8b5a45359891b8c"} Oct 30 16:42:49.268094 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.268066 6202 scope.go:122] "RemoveContainer" containerID="b3606144e02ad57741bf1d25970e8f45e2e0e8684fd3c8c5a8b5a45359891b8c" Oct 30 16:42:49.269897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.269859 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:49.270015 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.269952 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:49.270015 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.270004 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:49.270406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.270382 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" event={"ID":"45641dd8-8e3a-4a54-872d-c7033e870619","Type":"ContainerStarted","Data":"1d33159d34621620b609288c1564981f16c04b7c2d6fb99bef60f461954cdf6d"} Oct 30 16:42:49.270533 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.270429 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" event={"ID":"45641dd8-8e3a-4a54-872d-c7033e870619","Type":"ContainerStarted","Data":"671fc68c4ac352e6e08db6249ffd691ff2be4da305a71b5d29a40aa4f6a4ba43"} Oct 30 16:42:49.273402 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.273374 6202 generic.go:356] "Generic (PLEG): container finished" podID="51ef6485-8b36-4bb2-b643-31b06cdedddd" containerID="521126afbe5eda38b1d1c14944406bf1241d1a0adad558ca533a3e501094096d" exitCode=0 Oct 30 16:42:49.273523 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.273472 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6" Oct 30 16:42:49.275842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.273414 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6" event={"ID":"51ef6485-8b36-4bb2-b643-31b06cdedddd","Type":"ContainerDied","Data":"521126afbe5eda38b1d1c14944406bf1241d1a0adad558ca533a3e501094096d"} Oct 30 16:42:49.276056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.276038 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6" event={"ID":"51ef6485-8b36-4bb2-b643-31b06cdedddd","Type":"ContainerDied","Data":"bf6d178a9215287868f6e03f1238e09c520263df7030d6ed3b15c275a815c04c"} Oct 30 16:42:49.276255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.276220 6202 scope.go:122] "RemoveContainer" containerID="521126afbe5eda38b1d1c14944406bf1241d1a0adad558ca533a3e501094096d" Oct 30 16:42:49.299179 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.299150 6202 scope.go:122] "RemoveContainer" containerID="521126afbe5eda38b1d1c14944406bf1241d1a0adad558ca533a3e501094096d" Oct 30 16:42:49.299537 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:42:49.299509 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521126afbe5eda38b1d1c14944406bf1241d1a0adad558ca533a3e501094096d\": container with ID starting with 521126afbe5eda38b1d1c14944406bf1241d1a0adad558ca533a3e501094096d not found: ID does not exist" containerID="521126afbe5eda38b1d1c14944406bf1241d1a0adad558ca533a3e501094096d" Oct 30 16:42:49.299627 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.299549 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521126afbe5eda38b1d1c14944406bf1241d1a0adad558ca533a3e501094096d"} err="failed to get container status \"521126afbe5eda38b1d1c14944406bf1241d1a0adad558ca533a3e501094096d\": rpc error: code = NotFound desc = could not find container \"521126afbe5eda38b1d1c14944406bf1241d1a0adad558ca533a3e501094096d\": container with ID starting with 521126afbe5eda38b1d1c14944406bf1241d1a0adad558ca533a3e501094096d not found: ID does not exist" Oct 30 16:42:49.306052 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.306029 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7vfj\" (UniqueName: \"kubernetes.io/projected/51ef6485-8b36-4bb2-b643-31b06cdedddd-kube-api-access-k7vfj\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:42:49.436535 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.436416 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" podStartSLOduration=1.436392401 podStartE2EDuration="1.436392401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:49.389487839 +0000 UTC m=+1283.099942584" watchObservedRunningTime="2026-10-30 16:42:49.436392401 +0000 UTC m=+1283.146847146" Oct 30 16:42:49.449943 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.449874 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6"] Oct 30 16:42:49.453920 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.453893 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:49.454875 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.454846 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nx9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 16:42:49.454980 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.454909 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" podUID="45641dd8-8e3a-4a54-872d-c7033e870619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 16:42:49.457818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:49.457791 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a475e0dd-7468bcd444-hpgq6"] Oct 30 16:42:50.087221 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:50.087162 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:50.231271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:50.231240 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:50.335789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:50.332126 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" event={"ID":"e492a729-2008-4c88-a7b9-25caea1a5983","Type":"ContainerStarted","Data":"0b30bf983bdff7bc5bcb62df702c4872ea149466bcb5709c6819c54791fe7489"} Oct 30 16:42:50.354622 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:50.354456 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:50.474394 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:50.474357 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nx9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 16:42:50.474680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:50.474652 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" podUID="45641dd8-8e3a-4a54-872d-c7033e870619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 16:42:50.505816 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:50.498953 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51ef6485-8b36-4bb2-b643-31b06cdedddd" path="/var/lib/kubelet/pods/51ef6485-8b36-4bb2-b643-31b06cdedddd/volumes" Oct 30 16:42:51.208899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:51.208846 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:42:51.209609 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:51.209582 6202 patch_prober.go:29] interesting pod/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 16:42:51.209695 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:51.209647 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" podUID="e492a729-2008-4c88-a7b9-25caea1a5983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 16:42:51.472694 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:51.472600 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nx9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 16:42:51.476909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:51.476856 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" podUID="45641dd8-8e3a-4a54-872d-c7033e870619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 16:42:51.489224 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:51.487388 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:42:52.136599 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:52.136563 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:52.211847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:52.211251 6202 patch_prober.go:29] interesting pod/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 16:42:52.211847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:52.211324 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" podUID="e492a729-2008-4c88-a7b9-25caea1a5983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 16:42:52.460229 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:52.460193 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nx9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 16:42:52.460523 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:52.460494 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" podUID="45641dd8-8e3a-4a54-872d-c7033e870619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 16:42:53.208710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:53.208675 6202 patch_prober.go:29] interesting pod/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 16:42:53.209257 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:53.209224 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" podUID="e492a729-2008-4c88-a7b9-25caea1a5983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 16:42:53.453231 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:53.453193 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nx9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 16:42:53.453552 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:53.453520 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" podUID="45641dd8-8e3a-4a54-872d-c7033e870619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 16:42:54.035786 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:54.035740 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf"] Oct 30 16:42:54.060800 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:54.053464 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:54.115965 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:54.115922 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3655964-nqhmf"] Oct 30 16:42:54.174795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:54.170450 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:54.208340 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:54.208305 6202 patch_prober.go:29] interesting pod/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 16:42:54.208662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:54.208613 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" podUID="e492a729-2008-4c88-a7b9-25caea1a5983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 16:42:54.295853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:54.295461 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:54.413844 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:54.413811 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="006b1304-8f0c-4891-a71d-2447e6f77298" path="/var/lib/kubelet/pods/006b1304-8f0c-4891-a71d-2447e6f77298/volumes" Oct 30 16:42:54.460594 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:54.460558 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nx9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 16:42:54.465149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:54.464736 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" podUID="45641dd8-8e3a-4a54-872d-c7033e870619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 16:42:55.208744 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:55.208700 6202 patch_prober.go:29] interesting pod/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 16:42:55.208995 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:55.208786 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" podUID="e492a729-2008-4c88-a7b9-25caea1a5983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 16:42:55.460964 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:55.460878 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nx9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 16:42:55.461536 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:55.461504 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" podUID="45641dd8-8e3a-4a54-872d-c7033e870619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 16:42:55.994954 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:55.994910 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:56.247970 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:56.247870 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:42:56.252754 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:56.252726 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:42:56.519855 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:56.498657 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nx9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 16:42:56.519855 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:56.498745 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" podUID="45641dd8-8e3a-4a54-872d-c7033e870619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 16:42:57.452139 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:57.452100 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nx9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 16:42:57.452368 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:57.452171 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" podUID="45641dd8-8e3a-4a54-872d-c7033e870619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 16:42:58.149818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:58.149756 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:58.719349 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:58.719298 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:58.719859 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:58.719828 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:58.721180 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:58.721158 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:42:59.988427 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:42:59.988384 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:01.033114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:01.033081 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6"] Oct 30 16:43:01.048029 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:01.047993 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce183d3-sjqm6"] Oct 30 16:43:01.327336 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:01.327309 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:01.816616 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:01.816529 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:01.932806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:01.932754 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn"] Oct 30 16:43:01.935205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:01.935152 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:01.944458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:01.944430 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a475e0dd-8jhmn"] Oct 30 16:43:02.060598 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:02.060560 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:02.138405 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:02.138369 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97"] Oct 30 16:43:02.139022 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:02.138987 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" podUID="e492a729-2008-4c88-a7b9-25caea1a5983" containerName="istio-proxy" containerID="cri-o://0b30bf983bdff7bc5bcb62df702c4872ea149466bcb5709c6819c54791fe7489" gracePeriod=30 Oct 30 16:43:02.182457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:02.182413 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:02.312247 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:02.311900 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:02.416447 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:02.416372 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1943464d-3e38-4d10-ad22-6e4a3a4a47eb" path="/var/lib/kubelet/pods/1943464d-3e38-4d10-ad22-6e4a3a4a47eb/volumes" Oct 30 16:43:02.416998 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:02.416978 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aacddea-2a95-4741-88a6-633744acc134" path="/var/lib/kubelet/pods/1aacddea-2a95-4741-88a6-633744acc134/volumes" Oct 30 16:43:03.183494 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:03.183466 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:03.312945 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:03.312912 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:04.012711 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:04.012673 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:05.876388 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:05.876354 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:06.221166 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:06.221085 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:06.344136 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:06.344100 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:07.418408 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.418317 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:07.429435 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.429411 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:43:07.476865 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.476827 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-istio-data\") pod \"e492a729-2008-4c88-a7b9-25caea1a5983\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " Oct 30 16:43:07.493800 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.477476 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-workload-certs\") pod \"e492a729-2008-4c88-a7b9-25caea1a5983\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " Oct 30 16:43:07.501002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.498043 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-workload-socket\") pod \"e492a729-2008-4c88-a7b9-25caea1a5983\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " Oct 30 16:43:07.501002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.498139 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-credential-socket\") pod \"e492a729-2008-4c88-a7b9-25caea1a5983\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " Oct 30 16:43:07.501002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.498230 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-istio-envoy\") pod \"e492a729-2008-4c88-a7b9-25caea1a5983\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " Oct 30 16:43:07.501002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.498330 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e492a729-2008-4c88-a7b9-25caea1a5983-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e492a729-2008-4c88-a7b9-25caea1a5983-istio-podinfo\") pod \"e492a729-2008-4c88-a7b9-25caea1a5983\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " Oct 30 16:43:07.501002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.498414 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e492a729-2008-4c88-a7b9-25caea1a5983-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e492a729-2008-4c88-a7b9-25caea1a5983-istiod-ca-cert\") pod \"e492a729-2008-4c88-a7b9-25caea1a5983\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " Oct 30 16:43:07.501002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.498472 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e492a729-2008-4c88-a7b9-25caea1a5983-istio-token\" (UniqueName: \"kubernetes.io/projected/e492a729-2008-4c88-a7b9-25caea1a5983-istio-token\") pod \"e492a729-2008-4c88-a7b9-25caea1a5983\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " Oct 30 16:43:07.501002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.498571 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e492a729-2008-4c88-a7b9-25caea1a5983-kube-api-access-glm2v\" (UniqueName: \"kubernetes.io/projected/e492a729-2008-4c88-a7b9-25caea1a5983-kube-api-access-glm2v\") pod \"e492a729-2008-4c88-a7b9-25caea1a5983\" (UID: \"e492a729-2008-4c88-a7b9-25caea1a5983\") " Oct 30 16:43:07.501002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.477393 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-istio-data" pod "e492a729-2008-4c88-a7b9-25caea1a5983" (UID: "e492a729-2008-4c88-a7b9-25caea1a5983"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:07.501002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.488267 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-workload-certs" pod "e492a729-2008-4c88-a7b9-25caea1a5983" (UID: "e492a729-2008-4c88-a7b9-25caea1a5983"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:07.501002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.499246 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-workload-socket" pod "e492a729-2008-4c88-a7b9-25caea1a5983" (UID: "e492a729-2008-4c88-a7b9-25caea1a5983"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:07.501002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.499466 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-credential-socket" pod "e492a729-2008-4c88-a7b9-25caea1a5983" (UID: "e492a729-2008-4c88-a7b9-25caea1a5983"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:07.512833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.502379 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-istio-envoy" pod "e492a729-2008-4c88-a7b9-25caea1a5983" (UID: "e492a729-2008-4c88-a7b9-25caea1a5983"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:07.512833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.508561 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e492a729-2008-4c88-a7b9-25caea1a5983-istio-podinfo" pod "e492a729-2008-4c88-a7b9-25caea1a5983" (UID: "e492a729-2008-4c88-a7b9-25caea1a5983"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:43:07.512833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.508940 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e492a729-2008-4c88-a7b9-25caea1a5983-istiod-ca-cert" pod "e492a729-2008-4c88-a7b9-25caea1a5983" (UID: "e492a729-2008-4c88-a7b9-25caea1a5983"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:43:07.512833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.510063 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e492a729-2008-4c88-a7b9-25caea1a5983-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:43:07.512833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.510084 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:43:07.512833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.510099 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:43:07.512833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.510112 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:43:07.512833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.510127 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:43:07.512833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.510141 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e492a729-2008-4c88-a7b9-25caea1a5983-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:43:07.512833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.510153 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e492a729-2008-4c88-a7b9-25caea1a5983-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:43:07.512833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.512004 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e492a729-2008-4c88-a7b9-25caea1a5983-istio-token" pod "e492a729-2008-4c88-a7b9-25caea1a5983" (UID: "e492a729-2008-4c88-a7b9-25caea1a5983"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:07.519349 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.519314 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e492a729-2008-4c88-a7b9-25caea1a5983-kube-api-access-glm2v" pod "e492a729-2008-4c88-a7b9-25caea1a5983" (UID: "e492a729-2008-4c88-a7b9-25caea1a5983"). InnerVolumeSpecName "kube-api-access-glm2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:07.611421 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.611388 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e492a729-2008-4c88-a7b9-25caea1a5983-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:43:07.611421 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:07.611419 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glm2v\" (UniqueName: \"kubernetes.io/projected/e492a729-2008-4c88-a7b9-25caea1a5983-kube-api-access-glm2v\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:43:08.018040 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:08.017996 6202 generic.go:356] "Generic (PLEG): container finished" podID="e492a729-2008-4c88-a7b9-25caea1a5983" containerID="0b30bf983bdff7bc5bcb62df702c4872ea149466bcb5709c6819c54791fe7489" exitCode=0 Oct 30 16:43:08.018228 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:08.018053 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" Oct 30 16:43:08.018228 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:08.018076 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" event={"ID":"e492a729-2008-4c88-a7b9-25caea1a5983","Type":"ContainerDied","Data":"0b30bf983bdff7bc5bcb62df702c4872ea149466bcb5709c6819c54791fe7489"} Oct 30 16:43:08.018228 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:08.018133 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97" event={"ID":"e492a729-2008-4c88-a7b9-25caea1a5983","Type":"ContainerDied","Data":"dfa0b0239b3c656c8cb05c62f20b268cfd5dfae427238edfe9594c978b9281ce"} Oct 30 16:43:08.018228 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:08.018155 6202 scope.go:122] "RemoveContainer" containerID="0b30bf983bdff7bc5bcb62df702c4872ea149466bcb5709c6819c54791fe7489" Oct 30 16:43:08.029987 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:08.029958 6202 scope.go:122] "RemoveContainer" containerID="b3606144e02ad57741bf1d25970e8f45e2e0e8684fd3c8c5a8b5a45359891b8c" Oct 30 16:43:08.040745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:08.040708 6202 scope.go:122] "RemoveContainer" containerID="0b30bf983bdff7bc5bcb62df702c4872ea149466bcb5709c6819c54791fe7489" Oct 30 16:43:08.041078 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:43:08.041053 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b30bf983bdff7bc5bcb62df702c4872ea149466bcb5709c6819c54791fe7489\": container with ID starting with 0b30bf983bdff7bc5bcb62df702c4872ea149466bcb5709c6819c54791fe7489 not found: ID does not exist" containerID="0b30bf983bdff7bc5bcb62df702c4872ea149466bcb5709c6819c54791fe7489" Oct 30 16:43:08.041167 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:08.041084 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b30bf983bdff7bc5bcb62df702c4872ea149466bcb5709c6819c54791fe7489"} err="failed to get container status \"0b30bf983bdff7bc5bcb62df702c4872ea149466bcb5709c6819c54791fe7489\": rpc error: code = NotFound desc = could not find container \"0b30bf983bdff7bc5bcb62df702c4872ea149466bcb5709c6819c54791fe7489\": container with ID starting with 0b30bf983bdff7bc5bcb62df702c4872ea149466bcb5709c6819c54791fe7489 not found: ID does not exist" Oct 30 16:43:08.041167 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:08.041103 6202 scope.go:122] "RemoveContainer" containerID="b3606144e02ad57741bf1d25970e8f45e2e0e8684fd3c8c5a8b5a45359891b8c" Oct 30 16:43:08.041407 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:43:08.041383 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3606144e02ad57741bf1d25970e8f45e2e0e8684fd3c8c5a8b5a45359891b8c\": container with ID starting with b3606144e02ad57741bf1d25970e8f45e2e0e8684fd3c8c5a8b5a45359891b8c not found: ID does not exist" containerID="b3606144e02ad57741bf1d25970e8f45e2e0e8684fd3c8c5a8b5a45359891b8c" Oct 30 16:43:08.041496 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:08.041414 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3606144e02ad57741bf1d25970e8f45e2e0e8684fd3c8c5a8b5a45359891b8c"} err="failed to get container status \"b3606144e02ad57741bf1d25970e8f45e2e0e8684fd3c8c5a8b5a45359891b8c\": rpc error: code = NotFound desc = could not find container \"b3606144e02ad57741bf1d25970e8f45e2e0e8684fd3c8c5a8b5a45359891b8c\": container with ID starting with b3606144e02ad57741bf1d25970e8f45e2e0e8684fd3c8c5a8b5a45359891b8c not found: ID does not exist" Oct 30 16:43:08.087634 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:08.087597 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97"] Oct 30 16:43:08.097295 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:08.097264 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-a475e0dd-gw-openshift-default-6c9bd6c8d9-dln97"] Oct 30 16:43:08.412087 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:08.412051 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e492a729-2008-4c88-a7b9-25caea1a5983" path="/var/lib/kubelet/pods/e492a729-2008-4c88-a7b9-25caea1a5983/volumes" Oct 30 16:43:09.002190 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:09.002151 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:10.888434 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:10.888403 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:12.617807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.617746 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755"] Oct 30 16:43:12.618457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.618441 6202 cpu_manager.go:414] "containerMap: removing container" podUID="51ef6485-8b36-4bb2-b643-31b06cdedddd" containerName="payload-processing" Oct 30 16:43:12.618507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.618457 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ef6485-8b36-4bb2-b643-31b06cdedddd" containerName="payload-processing" Oct 30 16:43:12.618507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.618474 6202 cpu_manager.go:414] "containerMap: removing container" podUID="e492a729-2008-4c88-a7b9-25caea1a5983" containerName="istio-proxy" Oct 30 16:43:12.618507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.618479 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e492a729-2008-4c88-a7b9-25caea1a5983" containerName="istio-proxy" Oct 30 16:43:12.618603 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.618521 6202 cpu_manager.go:414] "containerMap: removing container" podUID="e492a729-2008-4c88-a7b9-25caea1a5983" containerName="istio-proxy" Oct 30 16:43:12.618603 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.618527 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e492a729-2008-4c88-a7b9-25caea1a5983" containerName="istio-proxy" Oct 30 16:43:12.618668 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.618656 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="e492a729-2008-4c88-a7b9-25caea1a5983" containerName="istio-proxy" Oct 30 16:43:12.618698 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.618670 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ef6485-8b36-4bb2-b643-31b06cdedddd" containerName="payload-processing" Oct 30 16:43:12.618698 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.618678 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="e492a729-2008-4c88-a7b9-25caea1a5983" containerName="istio-proxy" Oct 30 16:43:12.622565 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.622549 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755" Oct 30 16:43:12.638078 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.638052 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:43:12.650788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.650735 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:43:12.658891 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.658864 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:43:12.659086 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.658997 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:43:12.659334 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.659314 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755"] Oct 30 16:43:12.672101 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.672072 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:12.797476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.797429 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8036039-347e-4c99-ae91-af7abb6758a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0777a0c-rv755\" (UID: \"a8036039-347e-4c99-ae91-af7abb6758a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755" Oct 30 16:43:12.797710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.797636 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkqzw\" (UniqueName: \"kubernetes.io/projected/a8036039-347e-4c99-ae91-af7abb6758a6-kube-api-access-dkqzw\") pod \"maas-api-revoke-keys-e2e-ait-a0777a0c-rv755\" (UID: \"a8036039-347e-4c99-ae91-af7abb6758a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755" Oct 30 16:43:12.797710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.797677 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8036039-347e-4c99-ae91-af7abb6758a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0777a0c-rv755\" (UID: \"a8036039-347e-4c99-ae91-af7abb6758a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755" Oct 30 16:43:12.898834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.898693 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8036039-347e-4c99-ae91-af7abb6758a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0777a0c-rv755\" (UID: \"a8036039-347e-4c99-ae91-af7abb6758a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755" Oct 30 16:43:12.899025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.898878 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqzw\" (UniqueName: \"kubernetes.io/projected/a8036039-347e-4c99-ae91-af7abb6758a6-kube-api-access-dkqzw\") pod \"maas-api-revoke-keys-e2e-ait-a0777a0c-rv755\" (UID: \"a8036039-347e-4c99-ae91-af7abb6758a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755" Oct 30 16:43:12.899025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.898919 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8036039-347e-4c99-ae91-af7abb6758a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0777a0c-rv755\" (UID: \"a8036039-347e-4c99-ae91-af7abb6758a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755" Oct 30 16:43:12.899436 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.899412 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8036039-347e-4c99-ae91-af7abb6758a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0777a0c-rv755\" (UID: \"a8036039-347e-4c99-ae91-af7abb6758a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755" Oct 30 16:43:12.901786 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.901740 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8036039-347e-4c99-ae91-af7abb6758a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0777a0c-rv755\" (UID: \"a8036039-347e-4c99-ae91-af7abb6758a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755" Oct 30 16:43:12.922963 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.922938 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkqzw\" (UniqueName: \"kubernetes.io/projected/a8036039-347e-4c99-ae91-af7abb6758a6-kube-api-access-dkqzw\") pod \"maas-api-revoke-keys-e2e-ait-a0777a0c-rv755\" (UID: \"a8036039-347e-4c99-ae91-af7abb6758a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755" Oct 30 16:43:12.936293 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:12.936267 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755" Oct 30 16:43:13.314905 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:43:13.314872 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8036039_347e_4c99_ae91_af7abb6758a6.slice/crio-9658eaf5ae6bece0407d38bfaba167b1c8ff01c6ef870b80e1c798a3abff6c79 WatchSource:0}: Error finding container 9658eaf5ae6bece0407d38bfaba167b1c8ff01c6ef870b80e1c798a3abff6c79: Status 404 returned error can't find the container with id 9658eaf5ae6bece0407d38bfaba167b1c8ff01c6ef870b80e1c798a3abff6c79 Oct 30 16:43:13.317065 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:13.317044 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755"] Oct 30 16:43:14.086405 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:43:14.086363 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:43:14.259837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:14.259799 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755" event={"ID":"a8036039-347e-4c99-ae91-af7abb6758a6","Type":"ContainerStarted","Data":"dbc6035f6fdaccccc0e6f0ea8bfe6161f0fceec635660f3fe6177a739a723655"} Oct 30 16:43:14.260082 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:14.259871 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755" event={"ID":"a8036039-347e-4c99-ae91-af7abb6758a6","Type":"ContainerStarted","Data":"9658eaf5ae6bece0407d38bfaba167b1c8ff01c6ef870b80e1c798a3abff6c79"} Oct 30 16:43:14.301423 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:14.300863 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755" podStartSLOduration=2.30084204 podStartE2EDuration="2.30084204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:14.299005656 +0000 UTC m=+1308.009460401" watchObservedRunningTime="2026-10-30 16:43:14.30084204 +0000 UTC m=+1308.011296785" Oct 30 16:43:15.199628 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:15.199579 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:15.283209 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:15.283173 6202 generic.go:356] "Generic (PLEG): container finished" podID="a8036039-347e-4c99-ae91-af7abb6758a6" containerID="dbc6035f6fdaccccc0e6f0ea8bfe6161f0fceec635660f3fe6177a739a723655" exitCode=0 Oct 30 16:43:15.283437 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:15.283265 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755" event={"ID":"a8036039-347e-4c99-ae91-af7abb6758a6","Type":"ContainerDied","Data":"dbc6035f6fdaccccc0e6f0ea8bfe6161f0fceec635660f3fe6177a739a723655"} Oct 30 16:43:16.468128 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:16.468101 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755" Oct 30 16:43:16.491506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:16.491472 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8036039-347e-4c99-ae91-af7abb6758a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8036039-347e-4c99-ae91-af7abb6758a6-maas-api-openshift-ca\") pod \"a8036039-347e-4c99-ae91-af7abb6758a6\" (UID: \"a8036039-347e-4c99-ae91-af7abb6758a6\") " Oct 30 16:43:16.491673 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:16.491655 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8036039-347e-4c99-ae91-af7abb6758a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8036039-347e-4c99-ae91-af7abb6758a6-maas-api-xks-ca\") pod \"a8036039-347e-4c99-ae91-af7abb6758a6\" (UID: \"a8036039-347e-4c99-ae91-af7abb6758a6\") " Oct 30 16:43:16.491799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:16.491748 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8036039-347e-4c99-ae91-af7abb6758a6-kube-api-access-dkqzw\" (UniqueName: \"kubernetes.io/projected/a8036039-347e-4c99-ae91-af7abb6758a6-kube-api-access-dkqzw\") pod \"a8036039-347e-4c99-ae91-af7abb6758a6\" (UID: \"a8036039-347e-4c99-ae91-af7abb6758a6\") " Oct 30 16:43:16.491888 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:16.491846 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8036039-347e-4c99-ae91-af7abb6758a6-maas-api-openshift-ca" pod "a8036039-347e-4c99-ae91-af7abb6758a6" (UID: "a8036039-347e-4c99-ae91-af7abb6758a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:43:16.492439 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:16.492419 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8036039-347e-4c99-ae91-af7abb6758a6-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:43:16.494079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:16.494055 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8036039-347e-4c99-ae91-af7abb6758a6-maas-api-xks-ca" pod "a8036039-347e-4c99-ae91-af7abb6758a6" (UID: "a8036039-347e-4c99-ae91-af7abb6758a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:43:16.494170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:16.494073 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8036039-347e-4c99-ae91-af7abb6758a6-kube-api-access-dkqzw" pod "a8036039-347e-4c99-ae91-af7abb6758a6" (UID: "a8036039-347e-4c99-ae91-af7abb6758a6"). InnerVolumeSpecName "kube-api-access-dkqzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:16.594251 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:16.594214 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8036039-347e-4c99-ae91-af7abb6758a6-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:43:16.594251 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:16.594249 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkqzw\" (UniqueName: \"kubernetes.io/projected/a8036039-347e-4c99-ae91-af7abb6758a6-kube-api-access-dkqzw\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:43:16.967574 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:16.967491 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:17.036220 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:17.036181 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf"] Oct 30 16:43:17.044655 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:17.044603 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4def6444-97fsf"] Oct 30 16:43:17.301517 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:17.301426 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755" event={"ID":"a8036039-347e-4c99-ae91-af7abb6758a6","Type":"ContainerDied","Data":"9658eaf5ae6bece0407d38bfaba167b1c8ff01c6ef870b80e1c798a3abff6c79"} Oct 30 16:43:17.301517 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:17.301459 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755" Oct 30 16:43:17.301517 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:17.301480 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9658eaf5ae6bece0407d38bfaba167b1c8ff01c6ef870b80e1c798a3abff6c79" Oct 30 16:43:18.013359 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:18.013322 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:18.412518 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:18.412485 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c4264c-1b11-40d8-a62b-ca9928aef5d4" path="/var/lib/kubelet/pods/a0c4264c-1b11-40d8-a62b-ca9928aef5d4/volumes" Oct 30 16:43:18.495865 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:18.495813 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:18.619536 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:18.619490 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:18.984171 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:18.984103 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:19.033633 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:19.033592 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2"] Oct 30 16:43:19.047876 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:19.047836 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806c8e77-tfxt2"] Oct 30 16:43:19.123148 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:19.123116 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:19.673115 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:19.673078 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:19.805864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:19.805827 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:20.189344 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:20.189310 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:20.411480 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:20.411438 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f683a67-3e8a-4d72-99a8-6a3bae293802" path="/var/lib/kubelet/pods/8f683a67-3e8a-4d72-99a8-6a3bae293802/volumes" Oct 30 16:43:21.189647 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.189612 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc"] Oct 30 16:43:21.190348 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.190332 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a8036039-347e-4c99-ae91-af7abb6758a6" containerName="revoke-keys" Oct 30 16:43:21.190394 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.190349 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8036039-347e-4c99-ae91-af7abb6758a6" containerName="revoke-keys" Oct 30 16:43:21.190514 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.190503 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8036039-347e-4c99-ae91-af7abb6758a6" containerName="revoke-keys" Oct 30 16:43:21.195780 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.195748 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc" Oct 30 16:43:21.214462 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.214432 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc"] Oct 30 16:43:21.220236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.220212 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:43:21.220236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.220221 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:43:21.220445 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.220296 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:43:21.220445 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.220406 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:43:21.296211 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.296174 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4fns\" (UniqueName: \"kubernetes.io/projected/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-kube-api-access-g4fns\") pod \"maas-api-revoke-sub-3d6e7ef6-9j7tc\" (UID: \"84dc0c14-6e5e-4c72-90d2-db9ca7d42711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc" Oct 30 16:43:21.296419 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.296398 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d6e7ef6-9j7tc\" (UID: \"84dc0c14-6e5e-4c72-90d2-db9ca7d42711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc" Oct 30 16:43:21.296486 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.296468 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d6e7ef6-9j7tc\" (UID: \"84dc0c14-6e5e-4c72-90d2-db9ca7d42711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc" Oct 30 16:43:21.398419 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.398383 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4fns\" (UniqueName: \"kubernetes.io/projected/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-kube-api-access-g4fns\") pod \"maas-api-revoke-sub-3d6e7ef6-9j7tc\" (UID: \"84dc0c14-6e5e-4c72-90d2-db9ca7d42711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc" Oct 30 16:43:21.398615 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.398529 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d6e7ef6-9j7tc\" (UID: \"84dc0c14-6e5e-4c72-90d2-db9ca7d42711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc" Oct 30 16:43:21.398615 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.398581 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d6e7ef6-9j7tc\" (UID: \"84dc0c14-6e5e-4c72-90d2-db9ca7d42711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc" Oct 30 16:43:21.399418 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.399332 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d6e7ef6-9j7tc\" (UID: \"84dc0c14-6e5e-4c72-90d2-db9ca7d42711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc" Oct 30 16:43:21.401153 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.401131 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d6e7ef6-9j7tc\" (UID: \"84dc0c14-6e5e-4c72-90d2-db9ca7d42711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc" Oct 30 16:43:21.421980 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.421951 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4fns\" (UniqueName: \"kubernetes.io/projected/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-kube-api-access-g4fns\") pod \"maas-api-revoke-sub-3d6e7ef6-9j7tc\" (UID: \"84dc0c14-6e5e-4c72-90d2-db9ca7d42711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc" Oct 30 16:43:21.510697 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.510604 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc" Oct 30 16:43:21.817099 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:43:21.817070 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84dc0c14_6e5e_4c72_90d2_db9ca7d42711.slice/crio-0bd447019f073ecdf76688a07b1da04a235ef8734c252715e2acf822e191d441 WatchSource:0}: Error finding container 0bd447019f073ecdf76688a07b1da04a235ef8734c252715e2acf822e191d441: Status 404 returned error can't find the container with id 0bd447019f073ecdf76688a07b1da04a235ef8734c252715e2acf822e191d441 Oct 30 16:43:21.818013 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.817990 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc"] Oct 30 16:43:21.879792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:21.878999 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:22.348410 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:22.348370 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc" event={"ID":"84dc0c14-6e5e-4c72-90d2-db9ca7d42711","Type":"ContainerStarted","Data":"2c62167b38fee54b8f6e5d7487fedfb8f7a94dfca5314cd66ab76a631909bb3e"} Oct 30 16:43:22.348855 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:22.348467 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc" event={"ID":"84dc0c14-6e5e-4c72-90d2-db9ca7d42711","Type":"ContainerStarted","Data":"0bd447019f073ecdf76688a07b1da04a235ef8734c252715e2acf822e191d441"} Oct 30 16:43:22.388826 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:22.388738 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc" podStartSLOduration=1.388720966 podStartE2EDuration="1.388720966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:22.388254805 +0000 UTC m=+1316.098709550" watchObservedRunningTime="2026-10-30 16:43:22.388720966 +0000 UTC m=+1316.099175710" Oct 30 16:43:23.164533 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:23.164488 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:23.275682 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:23.275638 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:23.363233 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:23.363199 6202 generic.go:356] "Generic (PLEG): container finished" podID="84dc0c14-6e5e-4c72-90d2-db9ca7d42711" containerID="2c62167b38fee54b8f6e5d7487fedfb8f7a94dfca5314cd66ab76a631909bb3e" exitCode=0 Oct 30 16:43:23.363673 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:23.363273 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc" event={"ID":"84dc0c14-6e5e-4c72-90d2-db9ca7d42711","Type":"ContainerDied","Data":"2c62167b38fee54b8f6e5d7487fedfb8f7a94dfca5314cd66ab76a631909bb3e"} Oct 30 16:43:23.576345 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:23.576310 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:24.543134 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:24.543110 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc" Oct 30 16:43:24.662804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:24.662697 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-kube-api-access-g4fns\" (UniqueName: \"kubernetes.io/projected/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-kube-api-access-g4fns\") pod \"84dc0c14-6e5e-4c72-90d2-db9ca7d42711\" (UID: \"84dc0c14-6e5e-4c72-90d2-db9ca7d42711\") " Oct 30 16:43:24.662974 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:24.662822 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-maas-api-xks-ca\") pod \"84dc0c14-6e5e-4c72-90d2-db9ca7d42711\" (UID: \"84dc0c14-6e5e-4c72-90d2-db9ca7d42711\") " Oct 30 16:43:24.662974 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:24.662941 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-maas-api-openshift-ca\") pod \"84dc0c14-6e5e-4c72-90d2-db9ca7d42711\" (UID: \"84dc0c14-6e5e-4c72-90d2-db9ca7d42711\") " Oct 30 16:43:24.663362 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:24.663330 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-maas-api-openshift-ca" pod "84dc0c14-6e5e-4c72-90d2-db9ca7d42711" (UID: "84dc0c14-6e5e-4c72-90d2-db9ca7d42711"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:43:24.663714 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:24.663695 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:43:24.665224 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:24.665202 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-kube-api-access-g4fns" pod "84dc0c14-6e5e-4c72-90d2-db9ca7d42711" (UID: "84dc0c14-6e5e-4c72-90d2-db9ca7d42711"). InnerVolumeSpecName "kube-api-access-g4fns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:24.665359 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:24.665339 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-maas-api-xks-ca" pod "84dc0c14-6e5e-4c72-90d2-db9ca7d42711" (UID: "84dc0c14-6e5e-4c72-90d2-db9ca7d42711"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:43:24.765017 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:24.764990 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:43:24.765017 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:24.765014 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4fns\" (UniqueName: \"kubernetes.io/projected/84dc0c14-6e5e-4c72-90d2-db9ca7d42711-kube-api-access-g4fns\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:43:25.145401 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:25.145361 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:25.383120 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:25.383093 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc" Oct 30 16:43:25.383320 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:25.383132 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc" event={"ID":"84dc0c14-6e5e-4c72-90d2-db9ca7d42711","Type":"ContainerDied","Data":"0bd447019f073ecdf76688a07b1da04a235ef8734c252715e2acf822e191d441"} Oct 30 16:43:25.383320 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:25.383189 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bd447019f073ecdf76688a07b1da04a235ef8734c252715e2acf822e191d441" Oct 30 16:43:25.851465 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:25.851430 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:26.030134 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:26.030099 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq"] Oct 30 16:43:26.044503 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:26.044465 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23ff8ff3-p5kdq"] Oct 30 16:43:26.425172 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:26.425140 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7eee821-a910-467e-9a41-16ed03cfc82e" path="/var/lib/kubelet/pods/d7eee821-a910-467e-9a41-16ed03cfc82e/volumes" Oct 30 16:43:26.564395 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:26.564360 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:26.887824 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:26.887726 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:27.002706 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:27.002661 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:27.134503 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:27.134388 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:27.359180 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:27.359148 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:27.389793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:27.389746 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755"] Oct 30 16:43:27.398894 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:27.398867 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0777a0c-rv755"] Oct 30 16:43:27.476951 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:27.476916 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:28.417027 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:28.416989 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8036039-347e-4c99-ae91-af7abb6758a6" path="/var/lib/kubelet/pods/a8036039-347e-4c99-ae91-af7abb6758a6/volumes" Oct 30 16:43:29.089246 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:29.089209 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:29.754471 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:29.754441 6202 scope.go:122] "RemoveContainer" containerID="e18d37bdd437248fb764b2bd429dc46ee7cebc1225a315dc75de2b246ce88b20" Oct 30 16:43:29.766386 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:29.766365 6202 scope.go:122] "RemoveContainer" containerID="d12859ba897da5a354c27b2424db298bea0dbb14ceec03d31e9580d957abbd89" Oct 30 16:43:29.776740 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:29.776720 6202 scope.go:122] "RemoveContainer" containerID="ae6394a14d9e73813d9287c78d9c7c77ad320d72d56476620809911190694a98" Oct 30 16:43:29.789388 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:29.789363 6202 scope.go:122] "RemoveContainer" containerID="e4911e63163c4346b39332151b898412ea73890812d62e49d18336459a3437f9" Oct 30 16:43:30.930379 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:30.930337 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:31.499041 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.499008 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k"] Oct 30 16:43:31.499855 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.499832 6202 cpu_manager.go:414] "containerMap: removing container" podUID="84dc0c14-6e5e-4c72-90d2-db9ca7d42711" containerName="revoke-keys" Oct 30 16:43:31.499855 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.499854 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84dc0c14-6e5e-4c72-90d2-db9ca7d42711" containerName="revoke-keys" Oct 30 16:43:31.500078 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.500063 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="84dc0c14-6e5e-4c72-90d2-db9ca7d42711" containerName="revoke-keys" Oct 30 16:43:31.505616 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.505596 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k" Oct 30 16:43:31.523441 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.523416 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k"] Oct 30 16:43:31.529603 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.529582 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:43:31.529603 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.529583 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:43:31.531979 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.531957 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:43:31.532074 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.531988 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:43:31.602390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.602348 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a63cc690-xl49k\" (UID: \"294ae9a7-cf8b-46c4-abd3-77b7f2254b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k" Oct 30 16:43:31.602580 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.602517 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a63cc690-xl49k\" (UID: \"294ae9a7-cf8b-46c4-abd3-77b7f2254b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k" Oct 30 16:43:31.602631 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.602594 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqd9l\" (UniqueName: \"kubernetes.io/projected/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-kube-api-access-zqd9l\") pod \"maas-api-revoke-sub-a63cc690-xl49k\" (UID: \"294ae9a7-cf8b-46c4-abd3-77b7f2254b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k" Oct 30 16:43:31.704006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.703954 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a63cc690-xl49k\" (UID: \"294ae9a7-cf8b-46c4-abd3-77b7f2254b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k" Oct 30 16:43:31.704198 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.704079 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqd9l\" (UniqueName: \"kubernetes.io/projected/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-kube-api-access-zqd9l\") pod \"maas-api-revoke-sub-a63cc690-xl49k\" (UID: \"294ae9a7-cf8b-46c4-abd3-77b7f2254b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k" Oct 30 16:43:31.704247 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.704202 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a63cc690-xl49k\" (UID: \"294ae9a7-cf8b-46c4-abd3-77b7f2254b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k" Oct 30 16:43:31.704671 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.704652 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a63cc690-xl49k\" (UID: \"294ae9a7-cf8b-46c4-abd3-77b7f2254b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k" Oct 30 16:43:31.706704 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.706684 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a63cc690-xl49k\" (UID: \"294ae9a7-cf8b-46c4-abd3-77b7f2254b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k" Oct 30 16:43:31.727467 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.727434 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqd9l\" (UniqueName: \"kubernetes.io/projected/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-kube-api-access-zqd9l\") pod \"maas-api-revoke-sub-a63cc690-xl49k\" (UID: \"294ae9a7-cf8b-46c4-abd3-77b7f2254b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k" Oct 30 16:43:31.820149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:31.820068 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k" Oct 30 16:43:32.129019 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:43:32.128984 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294ae9a7_cf8b_46c4_abd3_77b7f2254b0a.slice/crio-5a4e2e8c0b6384bb3fadc752c230c8250a58f400be522802115d81802f8113cb WatchSource:0}: Error finding container 5a4e2e8c0b6384bb3fadc752c230c8250a58f400be522802115d81802f8113cb: Status 404 returned error can't find the container with id 5a4e2e8c0b6384bb3fadc752c230c8250a58f400be522802115d81802f8113cb Oct 30 16:43:32.131393 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:32.131365 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k"] Oct 30 16:43:32.375190 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:32.375116 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:32.462583 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:32.462550 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k" event={"ID":"294ae9a7-cf8b-46c4-abd3-77b7f2254b0a","Type":"ContainerStarted","Data":"5a4e2e8c0b6384bb3fadc752c230c8250a58f400be522802115d81802f8113cb"} Oct 30 16:43:33.032021 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:33.031896 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv"] Oct 30 16:43:33.049943 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:33.049889 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd04269-xzrnv"] Oct 30 16:43:33.479893 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:33.479859 6202 generic.go:356] "Generic (PLEG): container finished" podID="294ae9a7-cf8b-46c4-abd3-77b7f2254b0a" containerID="9f5a6a44502b4bcde7923565e2eb996c22a514eae58bcb07beae428204d57a44" exitCode=0 Oct 30 16:43:33.480304 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:33.479938 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k" event={"ID":"294ae9a7-cf8b-46c4-abd3-77b7f2254b0a","Type":"ContainerDied","Data":"9f5a6a44502b4bcde7923565e2eb996c22a514eae58bcb07beae428204d57a44"} Oct 30 16:43:33.889990 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:33.889956 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:34.411121 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:34.411090 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e083e19b-60cc-4d85-b312-ac6d93ea66ec" path="/var/lib/kubelet/pods/e083e19b-60cc-4d85-b312-ac6d93ea66ec/volumes" Oct 30 16:43:34.674150 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:34.674073 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k" Oct 30 16:43:34.753090 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:34.753054 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-maas-api-xks-ca\") pod \"294ae9a7-cf8b-46c4-abd3-77b7f2254b0a\" (UID: \"294ae9a7-cf8b-46c4-abd3-77b7f2254b0a\") " Oct 30 16:43:34.753308 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:34.753173 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-kube-api-access-zqd9l\" (UniqueName: \"kubernetes.io/projected/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-kube-api-access-zqd9l\") pod \"294ae9a7-cf8b-46c4-abd3-77b7f2254b0a\" (UID: \"294ae9a7-cf8b-46c4-abd3-77b7f2254b0a\") " Oct 30 16:43:34.753308 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:34.753255 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-maas-api-openshift-ca\") pod \"294ae9a7-cf8b-46c4-abd3-77b7f2254b0a\" (UID: \"294ae9a7-cf8b-46c4-abd3-77b7f2254b0a\") " Oct 30 16:43:34.754654 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:34.754625 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-maas-api-openshift-ca" pod "294ae9a7-cf8b-46c4-abd3-77b7f2254b0a" (UID: "294ae9a7-cf8b-46c4-abd3-77b7f2254b0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:43:34.756810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:34.756760 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-maas-api-xks-ca" pod "294ae9a7-cf8b-46c4-abd3-77b7f2254b0a" (UID: "294ae9a7-cf8b-46c4-abd3-77b7f2254b0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:43:34.757083 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:34.757059 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-kube-api-access-zqd9l" pod "294ae9a7-cf8b-46c4-abd3-77b7f2254b0a" (UID: "294ae9a7-cf8b-46c4-abd3-77b7f2254b0a"). InnerVolumeSpecName "kube-api-access-zqd9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:34.855647 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:34.855603 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:43:34.855647 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:34.855641 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqd9l\" (UniqueName: \"kubernetes.io/projected/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-kube-api-access-zqd9l\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:43:34.855647 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:34.855651 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:43:35.500412 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:35.500384 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k" Oct 30 16:43:35.500617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:35.500390 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k" event={"ID":"294ae9a7-cf8b-46c4-abd3-77b7f2254b0a","Type":"ContainerDied","Data":"5a4e2e8c0b6384bb3fadc752c230c8250a58f400be522802115d81802f8113cb"} Oct 30 16:43:35.500617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:35.500494 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a4e2e8c0b6384bb3fadc752c230c8250a58f400be522802115d81802f8113cb" Oct 30 16:43:35.667369 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:35.667333 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:37.256379 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:37.256334 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:38.123484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:38.123450 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:38.785216 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:38.785180 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:39.100792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:39.093834 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb"] Oct 30 16:43:39.164798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:39.154659 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:39.179584 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:39.179413 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d9821a5-6n7sb"] Oct 30 16:43:40.418891 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:40.418851 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c28d5268-f22a-4970-84d6-0bcafaa5ee7f" path="/var/lib/kubelet/pods/c28d5268-f22a-4970-84d6-0bcafaa5ee7f/volumes" Oct 30 16:43:40.910063 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:40.910021 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:42.836677 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:42.836643 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:44.069570 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:43:44.069528 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:43:44.422307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:44.422254 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:46.066467 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:46.066435 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:46.519311 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:46.519272 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:47.050114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:47.050077 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:47.314689 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:47.314595 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:47.957033 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:47.956998 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:49.645006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:49.644966 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:51.345245 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:51.345207 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:51.470690 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:51.470656 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:51.607845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:51.607573 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:52.939526 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:52.939492 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:53.051865 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.051835 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:53.106195 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.106162 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp"] Oct 30 16:43:53.107451 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.107423 6202 cpu_manager.go:414] "containerMap: removing container" podUID="294ae9a7-cf8b-46c4-abd3-77b7f2254b0a" containerName="revoke-keys" Oct 30 16:43:53.107451 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.107449 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294ae9a7-cf8b-46c4-abd3-77b7f2254b0a" containerName="revoke-keys" Oct 30 16:43:53.107677 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.107655 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="294ae9a7-cf8b-46c4-abd3-77b7f2254b0a" containerName="revoke-keys" Oct 30 16:43:53.121346 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.121322 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp" Oct 30 16:43:53.137679 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.136279 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp"] Oct 30 16:43:53.168604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.168572 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:53.172155 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.172134 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-e43400\"" Oct 30 16:43:53.172283 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.172222 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-e43400-dockercfg-kg5gv\"" Oct 30 16:43:53.305689 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.283490 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtc45\" (UniqueName: \"kubernetes.io/projected/854625e6-cc7a-4f5d-9475-ec31b0258eef-kube-api-access-wtc45\") pod \"payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp\" (UID: \"854625e6-cc7a-4f5d-9475-ec31b0258eef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp" Oct 30 16:43:53.305689 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.283606 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854625e6-cc7a-4f5d-9475-ec31b0258eef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp\" (UID: \"854625e6-cc7a-4f5d-9475-ec31b0258eef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp" Oct 30 16:43:53.392789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.389213 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtc45\" (UniqueName: \"kubernetes.io/projected/854625e6-cc7a-4f5d-9475-ec31b0258eef-kube-api-access-wtc45\") pod \"payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp\" (UID: \"854625e6-cc7a-4f5d-9475-ec31b0258eef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp" Oct 30 16:43:53.393558 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.393517 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854625e6-cc7a-4f5d-9475-ec31b0258eef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp\" (UID: \"854625e6-cc7a-4f5d-9475-ec31b0258eef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp" Oct 30 16:43:53.394690 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.394576 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854625e6-cc7a-4f5d-9475-ec31b0258eef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp\" (UID: \"854625e6-cc7a-4f5d-9475-ec31b0258eef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp" Oct 30 16:43:53.422176 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.422147 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtc45\" (UniqueName: \"kubernetes.io/projected/854625e6-cc7a-4f5d-9475-ec31b0258eef-kube-api-access-wtc45\") pod \"payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp\" (UID: \"854625e6-cc7a-4f5d-9475-ec31b0258eef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp" Oct 30 16:43:53.447383 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.447351 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp" Oct 30 16:43:53.469663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.469632 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:53.774702 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:43:53.774668 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854625e6_cc7a_4f5d_9475_ec31b0258eef.slice/crio-d365d139e5966c84a3803a2d5e6f6bae404cf7a3b62c0e93cf4ff1cef8bbcbf1 WatchSource:0}: Error finding container d365d139e5966c84a3803a2d5e6f6bae404cf7a3b62c0e93cf4ff1cef8bbcbf1: Status 404 returned error can't find the container with id d365d139e5966c84a3803a2d5e6f6bae404cf7a3b62c0e93cf4ff1cef8bbcbf1 Oct 30 16:43:53.776877 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:53.776845 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp"] Oct 30 16:43:54.020416 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:54.020362 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:54.189827 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:54.170553 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:54.375397 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:54.375361 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:54.507807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:54.507670 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:54.638967 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:54.638922 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:54.773050 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:54.772958 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp" event={"ID":"854625e6-cc7a-4f5d-9475-ec31b0258eef","Type":"ContainerStarted","Data":"acb2bdc25d36f042b59a9a7b2113959824b2a66f50d848e2ddec6274ef5dd1cd"} Oct 30 16:43:54.773050 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:54.773029 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp" Oct 30 16:43:54.773252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:54.773054 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp" event={"ID":"854625e6-cc7a-4f5d-9475-ec31b0258eef","Type":"ContainerStarted","Data":"d365d139e5966c84a3803a2d5e6f6bae404cf7a3b62c0e93cf4ff1cef8bbcbf1"} Oct 30 16:43:54.820635 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:54.820583 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp" podStartSLOduration=1.820570126 podStartE2EDuration="1.820570126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:54.815038094 +0000 UTC m=+1348.525492837" watchObservedRunningTime="2026-10-30 16:43:54.820570126 +0000 UTC m=+1348.531024869" Oct 30 16:43:55.203682 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:55.203651 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:55.818057 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:55.818023 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:55.982125 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:55.982087 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:56.435677 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:56.435643 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:57.063482 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:57.063451 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:57.173666 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:57.173615 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:58.025455 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.025414 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv"] Oct 30 16:43:58.031479 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.031451 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv" Oct 30 16:43:58.051027 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.051000 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:43:58.051212 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.051000 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:43:58.051334 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.051071 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:43:58.055873 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.055833 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx"] Oct 30 16:43:58.059574 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.059556 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:43:58.060837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.060815 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-090532a8-dlzfx"] Oct 30 16:43:58.067702 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.067656 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv"] Oct 30 16:43:58.087815 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.082982 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49da4384-cdhkv\" (UID: \"ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv" Oct 30 16:43:58.087815 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.083232 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49da4384-cdhkv\" (UID: \"ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv" Oct 30 16:43:58.087815 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.083272 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42nrf\" (UniqueName: \"kubernetes.io/projected/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-kube-api-access-42nrf\") pod \"maas-api-revoke-sub-49da4384-cdhkv\" (UID: \"ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv" Oct 30 16:43:58.108174 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.108145 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:58.185026 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.184994 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49da4384-cdhkv\" (UID: \"ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv" Oct 30 16:43:58.185215 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.185179 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49da4384-cdhkv\" (UID: \"ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv" Oct 30 16:43:58.185215 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.185214 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42nrf\" (UniqueName: \"kubernetes.io/projected/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-kube-api-access-42nrf\") pod \"maas-api-revoke-sub-49da4384-cdhkv\" (UID: \"ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv" Oct 30 16:43:58.185865 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.185839 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49da4384-cdhkv\" (UID: \"ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv" Oct 30 16:43:58.187707 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.187687 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49da4384-cdhkv\" (UID: \"ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv" Oct 30 16:43:58.208861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.208829 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42nrf\" (UniqueName: \"kubernetes.io/projected/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-kube-api-access-42nrf\") pod \"maas-api-revoke-sub-49da4384-cdhkv\" (UID: \"ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv" Oct 30 16:43:58.226299 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.226270 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:58.360378 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.360344 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv" Oct 30 16:43:58.412927 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.412892 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b43d22-8371-40a4-aa48-3baf6e3f96b2" path="/var/lib/kubelet/pods/34b43d22-8371-40a4-aa48-3baf6e3f96b2/volumes" Oct 30 16:43:58.673951 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.673920 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv"] Oct 30 16:43:58.676640 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:43:58.676597 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7fb93a_650e_4c3c_9d27_93f2ac7c5ba3.slice/crio-5c6793fcd20be14aaa41b0ae6fa69eed1d2a03f0e183b117f68f2791983b214a WatchSource:0}: Error finding container 5c6793fcd20be14aaa41b0ae6fa69eed1d2a03f0e183b117f68f2791983b214a: Status 404 returned error can't find the container with id 5c6793fcd20be14aaa41b0ae6fa69eed1d2a03f0e183b117f68f2791983b214a Oct 30 16:43:58.812659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:58.812619 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv" event={"ID":"ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3","Type":"ContainerStarted","Data":"5c6793fcd20be14aaa41b0ae6fa69eed1d2a03f0e183b117f68f2791983b214a"} Oct 30 16:43:59.034605 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:59.034555 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:59.723358 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:59.723311 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:59.823741 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:59.823708 6202 generic.go:356] "Generic (PLEG): container finished" podID="ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3" containerID="d36ae674b90a266add680a7682208c1a55f3c7044f9a18c43bd62aff6471425b" exitCode=0 Oct 30 16:43:59.823969 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:59.823798 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv" event={"ID":"ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3","Type":"ContainerDied","Data":"d36ae674b90a266add680a7682208c1a55f3c7044f9a18c43bd62aff6471425b"} Oct 30 16:43:59.833157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:59.833129 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t"] Oct 30 16:43:59.838095 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:59.838070 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" Oct 30 16:43:59.857808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:59.857783 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-830ba9-dockercfg-pwnfz\"" Oct 30 16:43:59.857808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:59.857806 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-830ba9\"" Oct 30 16:43:59.878678 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:59.878242 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t"] Oct 30 16:43:59.894589 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:59.894494 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:59.912833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:59.912742 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bklg\" (UniqueName: \"kubernetes.io/projected/734b4608-cecf-48ad-961e-31b7a2a22871-kube-api-access-7bklg\") pod \"payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t\" (UID: \"734b4608-cecf-48ad-961e-31b7a2a22871\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" Oct 30 16:43:59.913068 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:43:59.913050 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734b4608-cecf-48ad-961e-31b7a2a22871-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t\" (UID: \"734b4608-cecf-48ad-961e-31b7a2a22871\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" Oct 30 16:44:00.029792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:00.015293 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734b4608-cecf-48ad-961e-31b7a2a22871-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t\" (UID: \"734b4608-cecf-48ad-961e-31b7a2a22871\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" Oct 30 16:44:00.029792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:00.015577 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bklg\" (UniqueName: \"kubernetes.io/projected/734b4608-cecf-48ad-961e-31b7a2a22871-kube-api-access-7bklg\") pod \"payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t\" (UID: \"734b4608-cecf-48ad-961e-31b7a2a22871\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" Oct 30 16:44:00.029792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:00.017080 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734b4608-cecf-48ad-961e-31b7a2a22871-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t\" (UID: \"734b4608-cecf-48ad-961e-31b7a2a22871\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" Oct 30 16:44:00.112843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:00.108557 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bklg\" (UniqueName: \"kubernetes.io/projected/734b4608-cecf-48ad-961e-31b7a2a22871-kube-api-access-7bklg\") pod \"payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t\" (UID: \"734b4608-cecf-48ad-961e-31b7a2a22871\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" Oct 30 16:44:00.176792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:00.169134 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" Oct 30 16:44:00.544459 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:44:00.543700 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734b4608_cecf_48ad_961e_31b7a2a22871.slice/crio-4fc2a5f685c057cdecf19367b32afd6faa20d817fc471555ad140b7ec2d478e0 WatchSource:0}: Error finding container 4fc2a5f685c057cdecf19367b32afd6faa20d817fc471555ad140b7ec2d478e0: Status 404 returned error can't find the container with id 4fc2a5f685c057cdecf19367b32afd6faa20d817fc471555ad140b7ec2d478e0 Oct 30 16:44:00.545626 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:00.545597 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t"] Oct 30 16:44:00.835032 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:00.834998 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" event={"ID":"734b4608-cecf-48ad-961e-31b7a2a22871","Type":"ContainerStarted","Data":"de2550337d35f9b2a850dcc2e3815c2d3cb0604e22bed49e2e07b0a5339b460d"} Oct 30 16:44:00.835256 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:00.835060 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" Oct 30 16:44:00.835256 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:00.835081 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" event={"ID":"734b4608-cecf-48ad-961e-31b7a2a22871","Type":"ContainerStarted","Data":"4fc2a5f685c057cdecf19367b32afd6faa20d817fc471555ad140b7ec2d478e0"} Oct 30 16:44:01.036028 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:01.036001 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv" Oct 30 16:44:01.084821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:01.084734 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" podStartSLOduration=2.084712648 podStartE2EDuration="2.084712648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:00.897203485 +0000 UTC m=+1354.607658230" watchObservedRunningTime="2026-10-30 16:44:01.084712648 +0000 UTC m=+1354.795167396" Oct 30 16:44:01.148098 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:01.147530 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-kube-api-access-42nrf\" (UniqueName: \"kubernetes.io/projected/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-kube-api-access-42nrf\") pod \"ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3\" (UID: \"ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3\") " Oct 30 16:44:01.148098 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:01.147706 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-maas-api-openshift-ca\") pod \"ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3\" (UID: \"ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3\") " Oct 30 16:44:01.148098 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:01.147944 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-maas-api-xks-ca\") pod \"ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3\" (UID: \"ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3\") " Oct 30 16:44:01.149757 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:01.149386 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-maas-api-openshift-ca" pod "ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3" (UID: "ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:44:01.150148 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:01.150121 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:44:01.151198 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:01.151170 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-maas-api-xks-ca" pod "ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3" (UID: "ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:44:01.153458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:01.153433 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-kube-api-access-42nrf" pod "ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3" (UID: "ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3"). InnerVolumeSpecName "kube-api-access-42nrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:01.252056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:01.252018 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:44:01.252056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:01.252056 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42nrf\" (UniqueName: \"kubernetes.io/projected/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3-kube-api-access-42nrf\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:44:01.264045 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:01.264009 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:01.855582 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:01.855436 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv" Oct 30 16:44:01.855582 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:01.855442 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv" event={"ID":"ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3","Type":"ContainerDied","Data":"5c6793fcd20be14aaa41b0ae6fa69eed1d2a03f0e183b117f68f2791983b214a"} Oct 30 16:44:01.855582 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:01.855507 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c6793fcd20be14aaa41b0ae6fa69eed1d2a03f0e183b117f68f2791983b214a" Oct 30 16:44:02.996743 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:02.996712 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:04.573394 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:04.573339 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:05.789342 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:05.789260 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp" Oct 30 16:44:06.189453 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:06.189410 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:07.032243 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:07.032210 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc"] Oct 30 16:44:07.046518 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:07.046482 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d35981fb-cpcdc"] Oct 30 16:44:07.707895 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:07.707845 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:07.834799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:07.834264 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:08.348099 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:08.348053 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:08.412659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:08.412622 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbad445a-ffd5-427b-be3c-59225c098c21" path="/var/lib/kubelet/pods/cbad445a-ffd5-427b-be3c-59225c098c21/volumes" Oct 30 16:44:09.573809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:09.573751 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:11.100830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:11.100752 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:11.859296 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:11.859267 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" Oct 30 16:44:12.548955 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:12.548921 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:12.672322 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:12.672275 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:12.885230 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:12.885197 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:14.072021 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:44:14.071975 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:44:14.337988 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:14.337898 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:14.463245 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:14.463207 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:14.616116 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:14.616040 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:15.463960 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:15.463921 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:16.259689 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:16.259647 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:17.181434 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:17.181402 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:17.293647 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:17.293613 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:17.849972 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:17.849935 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:19.031450 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:19.031420 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:19.168687 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:19.168631 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:19.761538 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:19.761508 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:20.034913 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:20.034826 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp"] Oct 30 16:44:20.089933 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:20.089853 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e68bae0-gl6gp"] Oct 30 16:44:20.414429 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:20.414397 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d394a5a-344d-428e-b024-0597f6ce1bcb" path="/var/lib/kubelet/pods/9d394a5a-344d-428e-b024-0597f6ce1bcb/volumes" Oct 30 16:44:22.114047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:22.113998 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:24.676518 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:24.676465 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:26.759546 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:26.759503 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:28.351348 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:28.351306 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:29.871514 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:29.871482 6202 scope.go:122] "RemoveContainer" containerID="c47f770bd16becbd313d7d36bac6f2c5c0d2588c21c890c2fb2fcf593b2ea069" Oct 30 16:44:29.930043 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:29.930020 6202 scope.go:122] "RemoveContainer" containerID="cc0853ee64225f3e2b2490e75d6f311e6fcb1b6bb52b17bfa3ecfeae4b1b5db1" Oct 30 16:44:29.978121 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:29.976974 6202 scope.go:122] "RemoveContainer" containerID="2f1c52f03b06e8ebb5cb716abaee394e77d0d01afcc0cc567effd0b9703fc9c1" Oct 30 16:44:29.994071 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:29.994042 6202 scope.go:122] "RemoveContainer" containerID="3efd4ceb5d9a5308129739f3bb69d0c6b7db66579599957de15bb5ee5d4fcefb" Oct 30 16:44:30.013861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.013821 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:30.016557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.016537 6202 scope.go:122] "RemoveContainer" containerID="5e9c2713b13fbb17eefe44f8fb1e4571ad3ff4e9fd9f61a3a0d1437c9f134901" Oct 30 16:44:30.027206 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.027186 6202 scope.go:122] "RemoveContainer" containerID="75b922eab2b94e3b57256d964bbdec6216f617c45a4cb9b3e97e7b06c3178472" Oct 30 16:44:30.032736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.032702 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q"] Oct 30 16:44:30.056223 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.056195 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6"] Oct 30 16:44:30.065138 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.065107 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0cb704d-rsp2q"] Oct 30 16:44:30.072978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.072956 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d67e98bf-cq4v6"] Oct 30 16:44:30.134990 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.134959 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:30.412828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.412744 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5114a2a6-9142-4b3e-a820-fb75b364e628" path="/var/lib/kubelet/pods/5114a2a6-9142-4b3e-a820-fb75b364e628/volumes" Oct 30 16:44:30.413272 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.413258 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5" path="/var/lib/kubelet/pods/aac7cc92-21b3-46c1-bbf6-6f4fa4c6c7f5/volumes" Oct 30 16:44:30.509481 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.509435 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2"] Oct 30 16:44:30.510601 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.510574 6202 cpu_manager.go:414] "containerMap: removing container" podUID="ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3" containerName="revoke-keys" Oct 30 16:44:30.510601 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.510602 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3" containerName="revoke-keys" Oct 30 16:44:30.510885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.510868 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3" containerName="revoke-keys" Oct 30 16:44:30.517705 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.517688 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.533110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.533082 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2"] Oct 30 16:44:30.540617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.539665 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b8af660d-openshift-default-dockercfg-vhvv6\"" Oct 30 16:44:30.582858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.582817 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb03379b-4139-4f06-a67b-73249017ccb8-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.583041 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.582877 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4fws\" (UniqueName: \"kubernetes.io/projected/cb03379b-4139-4f06-a67b-73249017ccb8-kube-api-access-g4fws\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.583122 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.583093 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.583297 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.583272 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.583415 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.583362 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb03379b-4139-4f06-a67b-73249017ccb8-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.583415 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.583394 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.583522 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.583474 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb03379b-4139-4f06-a67b-73249017ccb8-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.583522 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.583509 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.583629 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.583611 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.584461 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.584431 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:30.693837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.689325 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.693837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.689405 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb03379b-4139-4f06-a67b-73249017ccb8-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.693837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.689452 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4fws\" (UniqueName: \"kubernetes.io/projected/cb03379b-4139-4f06-a67b-73249017ccb8-kube-api-access-g4fws\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.693837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.689534 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.693837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.689645 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.693837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.689728 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb03379b-4139-4f06-a67b-73249017ccb8-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.693837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.689756 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.693837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.689853 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb03379b-4139-4f06-a67b-73249017ccb8-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.693837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.689888 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.693837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.690378 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.693837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.691644 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb03379b-4139-4f06-a67b-73249017ccb8-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.693837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.692594 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.693837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.692850 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.693837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.693405 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.736098 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.733696 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb03379b-4139-4f06-a67b-73249017ccb8-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.736098 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.735287 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:30.758128 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.757116 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.758128 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.757592 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb03379b-4139-4f06-a67b-73249017ccb8-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.783374 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.783287 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4fws\" (UniqueName: \"kubernetes.io/projected/cb03379b-4139-4f06-a67b-73249017ccb8-kube-api-access-g4fws\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:30.844169 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:30.844121 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:31.223347 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:44:31.223318 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb03379b_4139_4f06_a67b_73249017ccb8.slice/crio-923122bd868eb831cd21d68c804da1eac1b4d3f33307d342283ff954c8c65b0d WatchSource:0}: Error finding container 923122bd868eb831cd21d68c804da1eac1b4d3f33307d342283ff954c8c65b0d: Status 404 returned error can't find the container with id 923122bd868eb831cd21d68c804da1eac1b4d3f33307d342283ff954c8c65b0d Oct 30 16:44:31.224868 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:31.224844 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2"] Oct 30 16:44:31.226275 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:31.226238 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:44:31.226375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:31.226308 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:44:31.226375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:31.226337 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:44:31.250440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:31.250410 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" event={"ID":"cb03379b-4139-4f06-a67b-73249017ccb8","Type":"ContainerStarted","Data":"923122bd868eb831cd21d68c804da1eac1b4d3f33307d342283ff954c8c65b0d"} Oct 30 16:44:31.978149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:31.978103 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:32.265037 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:32.264946 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" event={"ID":"cb03379b-4139-4f06-a67b-73249017ccb8","Type":"ContainerStarted","Data":"8e9e9bdaa8de63e518d25dcfcf0cb2cfe66445e6a3ea6b22fc0c517903e46b4e"} Oct 30 16:44:32.324480 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:32.324421 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" podStartSLOduration=2.324403391 podStartE2EDuration="2.324403391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:32.32225787 +0000 UTC m=+1386.032712611" watchObservedRunningTime="2026-10-30 16:44:32.324403391 +0000 UTC m=+1386.034858135" Oct 30 16:44:32.844861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:32.844825 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:32.851870 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:32.851841 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:33.273475 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:33.273377 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:33.274172 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:33.274152 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:44:33.733668 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:33.733633 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:34.958801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:34.958723 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:35.083590 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:35.083558 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:35.672189 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:35.672149 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:36.034604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:36.034504 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67"] Oct 30 16:44:36.041008 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:36.040982 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv"] Oct 30 16:44:36.045476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:36.045450 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03450b79-xtp67"] Oct 30 16:44:36.049311 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:36.049288 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e04a6d5-f9qrv"] Oct 30 16:44:36.412334 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:36.412302 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e7280c2-6b04-4d04-9edc-81323992d117" path="/var/lib/kubelet/pods/2e7280c2-6b04-4d04-9edc-81323992d117/volumes" Oct 30 16:44:36.412811 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:36.412792 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a72768-8289-4a5d-8e6d-599ccfec9502" path="/var/lib/kubelet/pods/72a72768-8289-4a5d-8e6d-599ccfec9502/volumes" Oct 30 16:44:36.516624 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:36.516589 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:36.644479 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:36.644419 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:36.689117 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:36.689025 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk"] Oct 30 16:44:36.696572 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:36.696542 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" Oct 30 16:44:36.723423 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:36.723392 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk"] Oct 30 16:44:36.744563 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:36.744530 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b8af660d-dockercfg-64sg8\"" Oct 30 16:44:36.773793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:36.745364 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b8af660d\"" Oct 30 16:44:36.919837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:36.895210 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/082aec51-71a4-48f7-994b-bd6cc7e05e49-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk\" (UID: \"082aec51-71a4-48f7-994b-bd6cc7e05e49\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" Oct 30 16:44:36.920607 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:36.920569 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95b28\" (UniqueName: \"kubernetes.io/projected/082aec51-71a4-48f7-994b-bd6cc7e05e49-kube-api-access-95b28\") pod \"payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk\" (UID: \"082aec51-71a4-48f7-994b-bd6cc7e05e49\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" Oct 30 16:44:37.024305 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:37.024203 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/082aec51-71a4-48f7-994b-bd6cc7e05e49-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk\" (UID: \"082aec51-71a4-48f7-994b-bd6cc7e05e49\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" Oct 30 16:44:37.025603 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:37.025374 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/082aec51-71a4-48f7-994b-bd6cc7e05e49-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk\" (UID: \"082aec51-71a4-48f7-994b-bd6cc7e05e49\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" Oct 30 16:44:37.026226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:37.026205 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95b28\" (UniqueName: \"kubernetes.io/projected/082aec51-71a4-48f7-994b-bd6cc7e05e49-kube-api-access-95b28\") pod \"payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk\" (UID: \"082aec51-71a4-48f7-994b-bd6cc7e05e49\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" Oct 30 16:44:37.064898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:37.064863 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95b28\" (UniqueName: \"kubernetes.io/projected/082aec51-71a4-48f7-994b-bd6cc7e05e49-kube-api-access-95b28\") pod \"payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk\" (UID: \"082aec51-71a4-48f7-994b-bd6cc7e05e49\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" Oct 30 16:44:37.142574 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:37.142543 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" Oct 30 16:44:37.501147 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:37.497856 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk"] Oct 30 16:44:38.057239 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:38.057208 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:38.317248 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:38.317146 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" event={"ID":"082aec51-71a4-48f7-994b-bd6cc7e05e49","Type":"ContainerStarted","Data":"13b333e57b4d0b695a71b2f305149001ba6a71e90d696c494a5aef8ea53a4b3b"} Oct 30 16:44:38.317631 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:38.317253 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" event={"ID":"082aec51-71a4-48f7-994b-bd6cc7e05e49","Type":"ContainerStarted","Data":"39275d4c7c62a4fd5d4c609cb4efb908164656c7ce096bec6911a1423f23d003"} Oct 30 16:44:38.317631 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:38.317284 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" Oct 30 16:44:38.341596 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:38.341563 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:38.363213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:38.363154 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" podStartSLOduration=2.363139844 podStartE2EDuration="2.363139844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:38.361084739 +0000 UTC m=+1392.071539482" watchObservedRunningTime="2026-10-30 16:44:38.363139844 +0000 UTC m=+1392.073594588" Oct 30 16:44:38.933307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:38.933276 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:39.440667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:39.440633 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:39.897436 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:39.897405 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:40.030126 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:40.030093 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:40.047739 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:40.047703 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs"] Oct 30 16:44:40.058591 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:40.058556 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bacd9d6-rrfxs"] Oct 30 16:44:40.418173 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:40.418143 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b2b662-ec40-4014-879a-c60a860da839" path="/var/lib/kubelet/pods/f6b2b662-ec40-4014-879a-c60a860da839/volumes" Oct 30 16:44:40.494362 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:40.494326 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:41.604265 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:41.604230 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:41.968794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:41.968694 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:43.033332 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:43.033296 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6"] Oct 30 16:44:43.050121 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:43.050088 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d5dd50-rvpn6"] Oct 30 16:44:43.349785 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:43.349736 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:44.070558 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:44:44.070515 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:44:44.411838 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:44.411801 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b73814b8-e973-48ff-891c-f4c1ed64c28a" path="/var/lib/kubelet/pods/b73814b8-e973-48ff-891c-f4c1ed64c28a/volumes" Oct 30 16:44:44.856130 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:44.856091 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:46.444340 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:46.444300 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:48.091431 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:48.091395 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:49.031182 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:49.031148 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:49.328957 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:49.328929 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" Oct 30 16:44:49.631846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:49.618698 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:50.198856 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.195499 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:50.198856 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.196097 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p"] Oct 30 16:44:50.237135 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.237101 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p" Oct 30 16:44:50.246618 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.246574 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p"] Oct 30 16:44:50.256002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.255205 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:44:50.256002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.255421 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:44:50.266930 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.265923 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:44:50.266930 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.266371 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:44:50.321548 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.317894 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:50.373379 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.373341 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p\" (UID: \"3a58bf98-a76b-45f1-8be3-e1b7711a86b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p" Oct 30 16:44:50.373996 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.373522 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbt4b\" (UniqueName: \"kubernetes.io/projected/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-kube-api-access-jbt4b\") pod \"maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p\" (UID: \"3a58bf98-a76b-45f1-8be3-e1b7711a86b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p" Oct 30 16:44:50.373996 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.373672 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p\" (UID: \"3a58bf98-a76b-45f1-8be3-e1b7711a86b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p" Oct 30 16:44:50.475229 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.475130 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p\" (UID: \"3a58bf98-a76b-45f1-8be3-e1b7711a86b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p" Oct 30 16:44:50.475717 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.475293 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p\" (UID: \"3a58bf98-a76b-45f1-8be3-e1b7711a86b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p" Oct 30 16:44:50.475717 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.475461 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbt4b\" (UniqueName: \"kubernetes.io/projected/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-kube-api-access-jbt4b\") pod \"maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p\" (UID: \"3a58bf98-a76b-45f1-8be3-e1b7711a86b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p" Oct 30 16:44:50.476309 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.476147 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p\" (UID: \"3a58bf98-a76b-45f1-8be3-e1b7711a86b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p" Oct 30 16:44:50.478891 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.478862 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p\" (UID: \"3a58bf98-a76b-45f1-8be3-e1b7711a86b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p" Oct 30 16:44:50.499581 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.499551 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbt4b\" (UniqueName: \"kubernetes.io/projected/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-kube-api-access-jbt4b\") pod \"maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p\" (UID: \"3a58bf98-a76b-45f1-8be3-e1b7711a86b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p" Oct 30 16:44:50.565399 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:50.565363 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p" Oct 30 16:44:51.082324 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:44:51.082279 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a58bf98_a76b_45f1_8be3_e1b7711a86b1.slice/crio-74a16aca53b76a4c1698b33a9816ea3a98d1b833f7b0b52c2e32374c866eb302 WatchSource:0}: Error finding container 74a16aca53b76a4c1698b33a9816ea3a98d1b833f7b0b52c2e32374c866eb302: Status 404 returned error can't find the container with id 74a16aca53b76a4c1698b33a9816ea3a98d1b833f7b0b52c2e32374c866eb302 Oct 30 16:44:51.082532 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:51.082510 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p"] Oct 30 16:44:51.323308 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:51.323251 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:51.485009 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:51.484848 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p" event={"ID":"3a58bf98-a76b-45f1-8be3-e1b7711a86b1","Type":"ContainerStarted","Data":"74a16aca53b76a4c1698b33a9816ea3a98d1b833f7b0b52c2e32374c866eb302"} Oct 30 16:44:51.556193 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:51.556154 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:52.491106 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:52.491070 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:52.498605 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:52.498575 6202 generic.go:356] "Generic (PLEG): container finished" podID="3a58bf98-a76b-45f1-8be3-e1b7711a86b1" containerID="a8fd6463c01f9eabcc795014051d6e64bea794bf171ab752e34004775cd5513b" exitCode=0 Oct 30 16:44:52.498750 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:52.498627 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p" event={"ID":"3a58bf98-a76b-45f1-8be3-e1b7711a86b1","Type":"ContainerDied","Data":"a8fd6463c01f9eabcc795014051d6e64bea794bf171ab752e34004775cd5513b"} Oct 30 16:44:52.696761 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:52.696729 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:53.036497 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:53.036462 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:53.211027 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:53.209830 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:53.714077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:53.714051 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p" Oct 30 16:44:53.783509 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:53.783424 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-kube-api-access-jbt4b\" (UniqueName: \"kubernetes.io/projected/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-kube-api-access-jbt4b\") pod \"3a58bf98-a76b-45f1-8be3-e1b7711a86b1\" (UID: \"3a58bf98-a76b-45f1-8be3-e1b7711a86b1\") " Oct 30 16:44:53.784564 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:53.784055 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-maas-api-xks-ca\") pod \"3a58bf98-a76b-45f1-8be3-e1b7711a86b1\" (UID: \"3a58bf98-a76b-45f1-8be3-e1b7711a86b1\") " Oct 30 16:44:53.784679 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:53.784582 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-maas-api-openshift-ca\") pod \"3a58bf98-a76b-45f1-8be3-e1b7711a86b1\" (UID: \"3a58bf98-a76b-45f1-8be3-e1b7711a86b1\") " Oct 30 16:44:53.785682 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:53.785653 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-maas-api-openshift-ca" pod "3a58bf98-a76b-45f1-8be3-e1b7711a86b1" (UID: "3a58bf98-a76b-45f1-8be3-e1b7711a86b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:44:53.786416 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:53.786388 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-kube-api-access-jbt4b" pod "3a58bf98-a76b-45f1-8be3-e1b7711a86b1" (UID: "3a58bf98-a76b-45f1-8be3-e1b7711a86b1"). InnerVolumeSpecName "kube-api-access-jbt4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:53.788345 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:53.788320 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-maas-api-xks-ca" pod "3a58bf98-a76b-45f1-8be3-e1b7711a86b1" (UID: "3a58bf98-a76b-45f1-8be3-e1b7711a86b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:44:53.841525 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:53.841492 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:53.886922 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:53.886890 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:44:53.886922 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:53.886924 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:44:53.887101 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:53.886940 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbt4b\" (UniqueName: \"kubernetes.io/projected/3a58bf98-a76b-45f1-8be3-e1b7711a86b1-kube-api-access-jbt4b\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:44:53.984868 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:53.984836 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:54.518277 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:54.518246 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p" Oct 30 16:44:54.518431 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:54.518252 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p" event={"ID":"3a58bf98-a76b-45f1-8be3-e1b7711a86b1","Type":"ContainerDied","Data":"74a16aca53b76a4c1698b33a9816ea3a98d1b833f7b0b52c2e32374c866eb302"} Oct 30 16:44:54.518431 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:54.518371 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74a16aca53b76a4c1698b33a9816ea3a98d1b833f7b0b52c2e32374c866eb302" Oct 30 16:44:54.797558 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:54.797451 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:54.951737 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:54.951706 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:55.095793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:55.087633 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:55.926335 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:55.926301 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:56.042015 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.041982 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6"] Oct 30 16:44:56.052651 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.052622 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fe0f7d-55rb6"] Oct 30 16:44:56.074191 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.074163 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn"] Oct 30 16:44:56.075091 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.075073 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3a58bf98-a76b-45f1-8be3-e1b7711a86b1" containerName="revoke-keys" Oct 30 16:44:56.075183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.075092 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a58bf98-a76b-45f1-8be3-e1b7711a86b1" containerName="revoke-keys" Oct 30 16:44:56.075254 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.075243 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a58bf98-a76b-45f1-8be3-e1b7711a86b1" containerName="revoke-keys" Oct 30 16:44:56.075471 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.075447 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:56.078957 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.078938 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" Oct 30 16:44:56.095562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.095531 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-05810b\"" Oct 30 16:44:56.095680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.095572 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-05810b-dockercfg-vm497\"" Oct 30 16:44:56.103200 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.103170 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn"] Oct 30 16:44:56.146791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.136749 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a9b980-5908-4440-b609-20cddc8c2327-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn\" (UID: \"71a9b980-5908-4440-b609-20cddc8c2327\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" Oct 30 16:44:56.146791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.136838 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7z92\" (UniqueName: \"kubernetes.io/projected/71a9b980-5908-4440-b609-20cddc8c2327-kube-api-access-j7z92\") pod \"payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn\" (UID: \"71a9b980-5908-4440-b609-20cddc8c2327\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" Oct 30 16:44:56.246846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.246296 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a9b980-5908-4440-b609-20cddc8c2327-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn\" (UID: \"71a9b980-5908-4440-b609-20cddc8c2327\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" Oct 30 16:44:56.246846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.246357 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7z92\" (UniqueName: \"kubernetes.io/projected/71a9b980-5908-4440-b609-20cddc8c2327-kube-api-access-j7z92\") pod \"payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn\" (UID: \"71a9b980-5908-4440-b609-20cddc8c2327\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" Oct 30 16:44:56.253791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.247601 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a9b980-5908-4440-b609-20cddc8c2327-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn\" (UID: \"71a9b980-5908-4440-b609-20cddc8c2327\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" Oct 30 16:44:56.412685 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.412647 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7z92\" (UniqueName: \"kubernetes.io/projected/71a9b980-5908-4440-b609-20cddc8c2327-kube-api-access-j7z92\") pod \"payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn\" (UID: \"71a9b980-5908-4440-b609-20cddc8c2327\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" Oct 30 16:44:56.424004 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.423969 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd" path="/var/lib/kubelet/pods/87cbeb5b-5973-45ec-a9e4-a5ebd55eb8bd/volumes" Oct 30 16:44:56.713790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:56.713050 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" Oct 30 16:44:57.157322 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:57.157293 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn"] Oct 30 16:44:57.203220 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:57.203188 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:57.325250 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:57.325207 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:57.548689 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:57.548585 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" event={"ID":"71a9b980-5908-4440-b609-20cddc8c2327","Type":"ContainerStarted","Data":"f41e2acabaaef5dc30f711a57be593767e8cc5992c6323b04a73deca2f41274c"} Oct 30 16:44:57.548689 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:57.548646 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" event={"ID":"71a9b980-5908-4440-b609-20cddc8c2327","Type":"ContainerStarted","Data":"cb1424763aef090c1ed0546b9fe936712c980f608a8ad47e8af67b8517cc03a4"} Oct 30 16:44:57.548947 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:57.548685 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" Oct 30 16:44:57.598249 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:57.598189 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" podStartSLOduration=1.598172385 podStartE2EDuration="1.598172385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:57.594124383 +0000 UTC m=+1411.304579146" watchObservedRunningTime="2026-10-30 16:44:57.598172385 +0000 UTC m=+1411.308627128" Oct 30 16:44:57.676168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:57.676128 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:58.327457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.327427 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:58.395533 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.395501 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz"] Oct 30 16:44:58.400305 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.400281 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" Oct 30 16:44:58.418222 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.418177 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-dc74d5\"" Oct 30 16:44:58.435098 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.435070 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz"] Oct 30 16:44:58.455294 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.454491 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122dfb4c-4008-4ee9-881f-d0e8a4525e05-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz\" (UID: \"122dfb4c-4008-4ee9-881f-d0e8a4525e05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" Oct 30 16:44:58.455294 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.454567 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4zbl\" (UniqueName: \"kubernetes.io/projected/122dfb4c-4008-4ee9-881f-d0e8a4525e05-kube-api-access-f4zbl\") pod \"payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz\" (UID: \"122dfb4c-4008-4ee9-881f-d0e8a4525e05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" Oct 30 16:44:58.475209 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.475168 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:58.582846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.578194 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122dfb4c-4008-4ee9-881f-d0e8a4525e05-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz\" (UID: \"122dfb4c-4008-4ee9-881f-d0e8a4525e05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" Oct 30 16:44:58.582846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.578281 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4zbl\" (UniqueName: \"kubernetes.io/projected/122dfb4c-4008-4ee9-881f-d0e8a4525e05-kube-api-access-f4zbl\") pod \"payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz\" (UID: \"122dfb4c-4008-4ee9-881f-d0e8a4525e05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" Oct 30 16:44:58.582846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.579467 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122dfb4c-4008-4ee9-881f-d0e8a4525e05-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz\" (UID: \"122dfb4c-4008-4ee9-881f-d0e8a4525e05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" Oct 30 16:44:58.634815 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.627403 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4zbl\" (UniqueName: \"kubernetes.io/projected/122dfb4c-4008-4ee9-881f-d0e8a4525e05-kube-api-access-f4zbl\") pod \"payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz\" (UID: \"122dfb4c-4008-4ee9-881f-d0e8a4525e05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" Oct 30 16:44:58.649893 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.649503 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb"] Oct 30 16:44:58.681791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.680369 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" Oct 30 16:44:58.737807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.725149 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-dc74d5-dockercfg-vhnjw\"" Oct 30 16:44:58.737807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.726520 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" Oct 30 16:44:58.766176 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.766095 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb"] Oct 30 16:44:58.796864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.789962 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pjwj\" (UniqueName: \"kubernetes.io/projected/5af8f435-0196-487a-b4ce-017b4adc6f50-kube-api-access-8pjwj\") pod \"payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb\" (UID: \"5af8f435-0196-487a-b4ce-017b4adc6f50\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" Oct 30 16:44:58.796864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.790044 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af8f435-0196-487a-b4ce-017b4adc6f50-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb\" (UID: \"5af8f435-0196-487a-b4ce-017b4adc6f50\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" Oct 30 16:44:58.891586 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.891547 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pjwj\" (UniqueName: \"kubernetes.io/projected/5af8f435-0196-487a-b4ce-017b4adc6f50-kube-api-access-8pjwj\") pod \"payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb\" (UID: \"5af8f435-0196-487a-b4ce-017b4adc6f50\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" Oct 30 16:44:58.891758 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.891604 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af8f435-0196-487a-b4ce-017b4adc6f50-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb\" (UID: \"5af8f435-0196-487a-b4ce-017b4adc6f50\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" Oct 30 16:44:58.892365 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.892343 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af8f435-0196-487a-b4ce-017b4adc6f50-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb\" (UID: \"5af8f435-0196-487a-b4ce-017b4adc6f50\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" Oct 30 16:44:58.920096 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:58.917358 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pjwj\" (UniqueName: \"kubernetes.io/projected/5af8f435-0196-487a-b4ce-017b4adc6f50-kube-api-access-8pjwj\") pod \"payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb\" (UID: \"5af8f435-0196-487a-b4ce-017b4adc6f50\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" Oct 30 16:44:59.052064 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.052035 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" Oct 30 16:44:59.154631 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.154603 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz"] Oct 30 16:44:59.157294 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:44:59.157268 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122dfb4c_4008_4ee9_881f_d0e8a4525e05.slice/crio-6b5a24f75b8f79051d124c4d31ecf3be78cc13d344a9f34ef3695cfdc9bd3955 WatchSource:0}: Error finding container 6b5a24f75b8f79051d124c4d31ecf3be78cc13d344a9f34ef3695cfdc9bd3955: Status 404 returned error can't find the container with id 6b5a24f75b8f79051d124c4d31ecf3be78cc13d344a9f34ef3695cfdc9bd3955 Oct 30 16:44:59.333535 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.333500 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:59.370545 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:44:59.370518 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af8f435_0196_487a_b4ce_017b4adc6f50.slice/crio-d471bc778172d0d7af213f66e4f5bf3248d77b2410c42a899cb4c156ca0c1d46 WatchSource:0}: Error finding container d471bc778172d0d7af213f66e4f5bf3248d77b2410c42a899cb4c156ca0c1d46: Status 404 returned error can't find the container with id d471bc778172d0d7af213f66e4f5bf3248d77b2410c42a899cb4c156ca0c1d46 Oct 30 16:44:59.371315 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.371288 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb"] Oct 30 16:44:59.421969 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.421945 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6"] Oct 30 16:44:59.428238 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.428218 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6" Oct 30 16:44:59.446521 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.446482 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-1b7bb3-dockercfg-s7vqs\"" Oct 30 16:44:59.455310 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.455281 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-1b7bb3\"" Oct 30 16:44:59.472838 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.472809 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6"] Oct 30 16:44:59.480221 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.480121 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:59.500176 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.500115 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f9bd673-68e5-4f3a-9c34-ffc286feaf04-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6\" (UID: \"1f9bd673-68e5-4f3a-9c34-ffc286feaf04\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6" Oct 30 16:44:59.500327 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.500273 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvq4h\" (UniqueName: \"kubernetes.io/projected/1f9bd673-68e5-4f3a-9c34-ffc286feaf04-kube-api-access-kvq4h\") pod \"payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6\" (UID: \"1f9bd673-68e5-4f3a-9c34-ffc286feaf04\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6" Oct 30 16:44:59.615550 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.614045 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f9bd673-68e5-4f3a-9c34-ffc286feaf04-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6\" (UID: \"1f9bd673-68e5-4f3a-9c34-ffc286feaf04\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6" Oct 30 16:44:59.615550 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.614131 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvq4h\" (UniqueName: \"kubernetes.io/projected/1f9bd673-68e5-4f3a-9c34-ffc286feaf04-kube-api-access-kvq4h\") pod \"payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6\" (UID: \"1f9bd673-68e5-4f3a-9c34-ffc286feaf04\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6" Oct 30 16:44:59.615550 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.615475 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f9bd673-68e5-4f3a-9c34-ffc286feaf04-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6\" (UID: \"1f9bd673-68e5-4f3a-9c34-ffc286feaf04\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6" Oct 30 16:44:59.630931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.616374 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:59.754376 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.754331 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" event={"ID":"5af8f435-0196-487a-b4ce-017b4adc6f50","Type":"ContainerStarted","Data":"035d3b02e7f92fb3ec0e67c3fd016ad8d112db23e218c95bdae212a47ec49576"} Oct 30 16:44:59.754677 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.754651 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" event={"ID":"5af8f435-0196-487a-b4ce-017b4adc6f50","Type":"ContainerStarted","Data":"d471bc778172d0d7af213f66e4f5bf3248d77b2410c42a899cb4c156ca0c1d46"} Oct 30 16:44:59.755820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.755796 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" Oct 30 16:44:59.854488 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.854455 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" event={"ID":"122dfb4c-4008-4ee9-881f-d0e8a4525e05","Type":"ContainerStarted","Data":"d706f7da72290bcdb59666c58cfbd2b11a15091f76b61f22c8338f1c24489981"} Oct 30 16:44:59.854756 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.854734 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" Oct 30 16:44:59.855320 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.855295 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" event={"ID":"122dfb4c-4008-4ee9-881f-d0e8a4525e05","Type":"ContainerStarted","Data":"6b5a24f75b8f79051d124c4d31ecf3be78cc13d344a9f34ef3695cfdc9bd3955"} Oct 30 16:44:59.855483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.854374 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvq4h\" (UniqueName: \"kubernetes.io/projected/1f9bd673-68e5-4f3a-9c34-ffc286feaf04-kube-api-access-kvq4h\") pod \"payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6\" (UID: \"1f9bd673-68e5-4f3a-9c34-ffc286feaf04\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6" Oct 30 16:44:59.878824 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.871864 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" podStartSLOduration=1.8718443809999998 podStartE2EDuration="1.871844381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:59.82755241 +0000 UTC m=+1413.538007167" watchObservedRunningTime="2026-10-30 16:44:59.871844381 +0000 UTC m=+1413.582299126" Oct 30 16:44:59.887201 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:44:59.887111 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" podStartSLOduration=1.8870972350000002 podStartE2EDuration="1.887097235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:59.868650694 +0000 UTC m=+1413.579105439" watchObservedRunningTime="2026-10-30 16:44:59.887097235 +0000 UTC m=+1413.597552014" Oct 30 16:45:00.047373 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.047335 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6" Oct 30 16:45:00.186169 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.181892 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw"] Oct 30 16:45:00.211410 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.198575 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v"] Oct 30 16:45:00.211410 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.199471 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" Oct 30 16:45:00.219909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.216006 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782"] Oct 30 16:45:00.219909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.216107 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" Oct 30 16:45:00.219909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.219185 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:45:00.219909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.219192 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:45:00.219909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.219246 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:45:00.247451 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.247425 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7"] Oct 30 16:45:00.259504 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.248069 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" Oct 30 16:45:00.263125 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.263100 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9"] Oct 30 16:45:00.264003 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.263986 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" Oct 30 16:45:00.269551 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.269534 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw"] Oct 30 16:45:00.269828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.269811 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7"] Oct 30 16:45:00.270030 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.270016 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" Oct 30 16:45:00.275633 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.275613 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v"] Oct 30 16:45:00.285651 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.285596 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9"] Oct 30 16:45:00.288480 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.288455 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdwvn\" (UniqueName: \"kubernetes.io/projected/89525800-714a-4a94-a37a-170e7b6da7d0-kube-api-access-tdwvn\") pod \"maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw\" (UID: \"89525800-714a-4a94-a37a-170e7b6da7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" Oct 30 16:45:00.288607 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.288582 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5812171-b69d-461b-8cfb-d23a17f7c64b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v\" (UID: \"f5812171-b69d-461b-8cfb-d23a17f7c64b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" Oct 30 16:45:00.288736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.288714 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89525800-714a-4a94-a37a-170e7b6da7d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw\" (UID: \"89525800-714a-4a94-a37a-170e7b6da7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" Oct 30 16:45:00.288922 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.288900 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzx5b\" (UniqueName: \"kubernetes.io/projected/f5812171-b69d-461b-8cfb-d23a17f7c64b-kube-api-access-pzx5b\") pod \"maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v\" (UID: \"f5812171-b69d-461b-8cfb-d23a17f7c64b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" Oct 30 16:45:00.290805 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.290742 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782"] Oct 30 16:45:00.295681 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.295655 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph"] Oct 30 16:45:00.301952 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.301933 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl"] Oct 30 16:45:00.304027 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.302862 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" Oct 30 16:45:00.313365 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.313343 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc"] Oct 30 16:45:00.313470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.313460 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" Oct 30 16:45:00.318574 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.318555 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph"] Oct 30 16:45:00.318674 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.318648 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" Oct 30 16:45:00.322993 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.322976 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl"] Oct 30 16:45:00.327065 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.327047 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc"] Oct 30 16:45:00.389878 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.389844 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa8cd066-6dc6-4480-9e56-b7a45efaf112-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph\" (UID: \"aa8cd066-6dc6-4480-9e56-b7a45efaf112\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" Oct 30 16:45:00.390316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.389902 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55faa818-b86f-4438-974e-f753872814d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc\" (UID: \"55faa818-b86f-4438-974e-f753872814d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" Oct 30 16:45:00.390316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.389996 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5812171-b69d-461b-8cfb-d23a17f7c64b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v\" (UID: \"f5812171-b69d-461b-8cfb-d23a17f7c64b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" Oct 30 16:45:00.390316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.390041 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h22h\" (UniqueName: \"kubernetes.io/projected/de229d2f-d50d-4469-b884-fe682c4e457b-kube-api-access-9h22h\") pod \"maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9\" (UID: \"de229d2f-d50d-4469-b884-fe682c4e457b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" Oct 30 16:45:00.390316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.390069 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtsq6\" (UniqueName: \"kubernetes.io/projected/8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc-kube-api-access-jtsq6\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl\" (UID: \"8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" Oct 30 16:45:00.390316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.390099 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de229d2f-d50d-4469-b884-fe682c4e457b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9\" (UID: \"de229d2f-d50d-4469-b884-fe682c4e457b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" Oct 30 16:45:00.390316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.390220 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2c59\" (UniqueName: \"kubernetes.io/projected/aa8cd066-6dc6-4480-9e56-b7a45efaf112-kube-api-access-r2c59\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph\" (UID: \"aa8cd066-6dc6-4480-9e56-b7a45efaf112\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" Oct 30 16:45:00.390316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.390268 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl\" (UID: \"8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" Oct 30 16:45:00.390316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.390321 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89525800-714a-4a94-a37a-170e7b6da7d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw\" (UID: \"89525800-714a-4a94-a37a-170e7b6da7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" Oct 30 16:45:00.390731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.390361 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44941116-b5e2-4ed0-81d5-e5bb1e8f7678-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782\" (UID: \"44941116-b5e2-4ed0-81d5-e5bb1e8f7678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" Oct 30 16:45:00.390731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.390616 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcsqb\" (UniqueName: \"kubernetes.io/projected/55faa818-b86f-4438-974e-f753872814d1-kube-api-access-mcsqb\") pod \"maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc\" (UID: \"55faa818-b86f-4438-974e-f753872814d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" Oct 30 16:45:00.390731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.390669 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzx5b\" (UniqueName: \"kubernetes.io/projected/f5812171-b69d-461b-8cfb-d23a17f7c64b-kube-api-access-pzx5b\") pod \"maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v\" (UID: \"f5812171-b69d-461b-8cfb-d23a17f7c64b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" Oct 30 16:45:00.390731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.390693 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca0e07b-d937-48e9-9261-9b5182ffba5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7\" (UID: \"fca0e07b-d937-48e9-9261-9b5182ffba5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" Oct 30 16:45:00.390995 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.390735 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gqcq\" (UniqueName: \"kubernetes.io/projected/fca0e07b-d937-48e9-9261-9b5182ffba5b-kube-api-access-6gqcq\") pod \"maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7\" (UID: \"fca0e07b-d937-48e9-9261-9b5182ffba5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" Oct 30 16:45:00.390995 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.390804 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5812171-b69d-461b-8cfb-d23a17f7c64b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v\" (UID: \"f5812171-b69d-461b-8cfb-d23a17f7c64b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" Oct 30 16:45:00.390995 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.390840 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tppbc\" (UniqueName: \"kubernetes.io/projected/44941116-b5e2-4ed0-81d5-e5bb1e8f7678-kube-api-access-tppbc\") pod \"maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782\" (UID: \"44941116-b5e2-4ed0-81d5-e5bb1e8f7678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" Oct 30 16:45:00.390995 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.390896 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdwvn\" (UniqueName: \"kubernetes.io/projected/89525800-714a-4a94-a37a-170e7b6da7d0-kube-api-access-tdwvn\") pod \"maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw\" (UID: \"89525800-714a-4a94-a37a-170e7b6da7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" Oct 30 16:45:00.391408 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.391390 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89525800-714a-4a94-a37a-170e7b6da7d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw\" (UID: \"89525800-714a-4a94-a37a-170e7b6da7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" Oct 30 16:45:00.396433 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:45:00.396367 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9bd673_68e5_4f3a_9c34_ffc286feaf04.slice/crio-7bef02e01565b31b231706e64d95de51429308e0636eb36eb26928c2ebbf2427 WatchSource:0}: Error finding container 7bef02e01565b31b231706e64d95de51429308e0636eb36eb26928c2ebbf2427: Status 404 returned error can't find the container with id 7bef02e01565b31b231706e64d95de51429308e0636eb36eb26928c2ebbf2427 Oct 30 16:45:00.396938 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.396916 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6"] Oct 30 16:45:00.414959 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.414903 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzx5b\" (UniqueName: \"kubernetes.io/projected/f5812171-b69d-461b-8cfb-d23a17f7c64b-kube-api-access-pzx5b\") pod \"maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v\" (UID: \"f5812171-b69d-461b-8cfb-d23a17f7c64b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" Oct 30 16:45:00.425402 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.425373 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdwvn\" (UniqueName: \"kubernetes.io/projected/89525800-714a-4a94-a37a-170e7b6da7d0-kube-api-access-tdwvn\") pod \"maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw\" (UID: \"89525800-714a-4a94-a37a-170e7b6da7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" Oct 30 16:45:00.492745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.492710 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa8cd066-6dc6-4480-9e56-b7a45efaf112-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph\" (UID: \"aa8cd066-6dc6-4480-9e56-b7a45efaf112\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" Oct 30 16:45:00.492899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.492753 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55faa818-b86f-4438-974e-f753872814d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc\" (UID: \"55faa818-b86f-4438-974e-f753872814d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" Oct 30 16:45:00.492899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.492793 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h22h\" (UniqueName: \"kubernetes.io/projected/de229d2f-d50d-4469-b884-fe682c4e457b-kube-api-access-9h22h\") pod \"maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9\" (UID: \"de229d2f-d50d-4469-b884-fe682c4e457b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" Oct 30 16:45:00.492899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.492811 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtsq6\" (UniqueName: \"kubernetes.io/projected/8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc-kube-api-access-jtsq6\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl\" (UID: \"8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" Oct 30 16:45:00.492899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.492834 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de229d2f-d50d-4469-b884-fe682c4e457b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9\" (UID: \"de229d2f-d50d-4469-b884-fe682c4e457b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" Oct 30 16:45:00.492899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.492883 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2c59\" (UniqueName: \"kubernetes.io/projected/aa8cd066-6dc6-4480-9e56-b7a45efaf112-kube-api-access-r2c59\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph\" (UID: \"aa8cd066-6dc6-4480-9e56-b7a45efaf112\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" Oct 30 16:45:00.493182 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.492917 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl\" (UID: \"8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" Oct 30 16:45:00.493182 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.492976 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44941116-b5e2-4ed0-81d5-e5bb1e8f7678-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782\" (UID: \"44941116-b5e2-4ed0-81d5-e5bb1e8f7678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" Oct 30 16:45:00.493182 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.493098 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcsqb\" (UniqueName: \"kubernetes.io/projected/55faa818-b86f-4438-974e-f753872814d1-kube-api-access-mcsqb\") pod \"maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc\" (UID: \"55faa818-b86f-4438-974e-f753872814d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" Oct 30 16:45:00.493182 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.493167 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca0e07b-d937-48e9-9261-9b5182ffba5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7\" (UID: \"fca0e07b-d937-48e9-9261-9b5182ffba5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" Oct 30 16:45:00.493378 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.493226 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gqcq\" (UniqueName: \"kubernetes.io/projected/fca0e07b-d937-48e9-9261-9b5182ffba5b-kube-api-access-6gqcq\") pod \"maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7\" (UID: \"fca0e07b-d937-48e9-9261-9b5182ffba5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" Oct 30 16:45:00.493378 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.493310 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tppbc\" (UniqueName: \"kubernetes.io/projected/44941116-b5e2-4ed0-81d5-e5bb1e8f7678-kube-api-access-tppbc\") pod \"maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782\" (UID: \"44941116-b5e2-4ed0-81d5-e5bb1e8f7678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" Oct 30 16:45:00.493548 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.493502 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa8cd066-6dc6-4480-9e56-b7a45efaf112-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph\" (UID: \"aa8cd066-6dc6-4480-9e56-b7a45efaf112\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" Oct 30 16:45:00.493687 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.493662 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55faa818-b86f-4438-974e-f753872814d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc\" (UID: \"55faa818-b86f-4438-974e-f753872814d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" Oct 30 16:45:00.493785 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.493696 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de229d2f-d50d-4469-b884-fe682c4e457b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9\" (UID: \"de229d2f-d50d-4469-b884-fe682c4e457b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" Oct 30 16:45:00.494066 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.494041 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44941116-b5e2-4ed0-81d5-e5bb1e8f7678-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782\" (UID: \"44941116-b5e2-4ed0-81d5-e5bb1e8f7678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" Oct 30 16:45:00.494158 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.494068 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca0e07b-d937-48e9-9261-9b5182ffba5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7\" (UID: \"fca0e07b-d937-48e9-9261-9b5182ffba5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" Oct 30 16:45:00.494251 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.494230 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl\" (UID: \"8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" Oct 30 16:45:00.516562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.516536 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtsq6\" (UniqueName: \"kubernetes.io/projected/8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc-kube-api-access-jtsq6\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl\" (UID: \"8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" Oct 30 16:45:00.517985 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.517915 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcsqb\" (UniqueName: \"kubernetes.io/projected/55faa818-b86f-4438-974e-f753872814d1-kube-api-access-mcsqb\") pod \"maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc\" (UID: \"55faa818-b86f-4438-974e-f753872814d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" Oct 30 16:45:00.518131 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.518109 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h22h\" (UniqueName: \"kubernetes.io/projected/de229d2f-d50d-4469-b884-fe682c4e457b-kube-api-access-9h22h\") pod \"maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9\" (UID: \"de229d2f-d50d-4469-b884-fe682c4e457b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" Oct 30 16:45:00.518131 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.518121 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2c59\" (UniqueName: \"kubernetes.io/projected/aa8cd066-6dc6-4480-9e56-b7a45efaf112-kube-api-access-r2c59\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph\" (UID: \"aa8cd066-6dc6-4480-9e56-b7a45efaf112\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" Oct 30 16:45:00.518528 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.518510 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gqcq\" (UniqueName: \"kubernetes.io/projected/fca0e07b-d937-48e9-9261-9b5182ffba5b-kube-api-access-6gqcq\") pod \"maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7\" (UID: \"fca0e07b-d937-48e9-9261-9b5182ffba5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" Oct 30 16:45:00.518646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.518624 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tppbc\" (UniqueName: \"kubernetes.io/projected/44941116-b5e2-4ed0-81d5-e5bb1e8f7678-kube-api-access-tppbc\") pod \"maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782\" (UID: \"44941116-b5e2-4ed0-81d5-e5bb1e8f7678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" Oct 30 16:45:00.548451 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.548427 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" Oct 30 16:45:00.589853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.589825 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" Oct 30 16:45:00.599976 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.599937 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" Oct 30 16:45:00.612173 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.612142 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" Oct 30 16:45:00.630648 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.630031 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" Oct 30 16:45:00.640461 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.640041 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" Oct 30 16:45:00.650638 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.649112 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" Oct 30 16:45:00.660428 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.656687 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" Oct 30 16:45:00.787036 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.787002 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:00.804899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.803547 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6" event={"ID":"1f9bd673-68e5-4f3a-9c34-ffc286feaf04","Type":"ContainerStarted","Data":"3a34f7c58155893ec8d2abf7b923b6aa1ec8098b1ecaee276f07537553dab5b6"} Oct 30 16:45:00.804899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.803611 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6" event={"ID":"1f9bd673-68e5-4f3a-9c34-ffc286feaf04","Type":"ContainerStarted","Data":"7bef02e01565b31b231706e64d95de51429308e0636eb36eb26928c2ebbf2427"} Oct 30 16:45:00.804899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.804364 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6" Oct 30 16:45:00.936258 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.935248 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:00.947749 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:00.947688 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6" podStartSLOduration=1.947667779 podStartE2EDuration="1.947667779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:00.858317376 +0000 UTC m=+1414.568772143" watchObservedRunningTime="2026-10-30 16:45:00.947667779 +0000 UTC m=+1414.658122523" Oct 30 16:45:01.120715 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:01.120684 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr"] Oct 30 16:45:01.126274 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:01.126051 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5dc9043-h2vsr"] Oct 30 16:45:01.267063 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:01.267032 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw"] Oct 30 16:45:01.457970 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:01.442329 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v"] Oct 30 16:45:01.662091 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:01.662055 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc"] Oct 30 16:45:01.686985 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:01.686961 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782"] Oct 30 16:45:01.729895 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:01.729869 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl"] Oct 30 16:45:01.743811 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:45:01.743379 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55faa818_b86f_4438_974e_f753872814d1.slice/crio-ee5cc65c0b4e518e96c2ef7b7e2248334ae23c7e1167f683ef1672e912b0517a WatchSource:0}: Error finding container ee5cc65c0b4e518e96c2ef7b7e2248334ae23c7e1167f683ef1672e912b0517a: Status 404 returned error can't find the container with id ee5cc65c0b4e518e96c2ef7b7e2248334ae23c7e1167f683ef1672e912b0517a Oct 30 16:45:01.744071 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:45:01.744031 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44941116_b5e2_4ed0_81d5_e5bb1e8f7678.slice/crio-8bc1f33c54411a3d28ea83965417f85bc45ec50bf0f748a4953c1e21a61b78f1 WatchSource:0}: Error finding container 8bc1f33c54411a3d28ea83965417f85bc45ec50bf0f748a4953c1e21a61b78f1: Status 404 returned error can't find the container with id 8bc1f33c54411a3d28ea83965417f85bc45ec50bf0f748a4953c1e21a61b78f1 Oct 30 16:45:01.745637 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:45:01.745606 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbe8ab5_ebfd_42cf_acd5_a2a279c874cc.slice/crio-64394b3d190ff71870ee0ec54023bc63794f8de4f973cb8a7cc6e5de0264bfd1 WatchSource:0}: Error finding container 64394b3d190ff71870ee0ec54023bc63794f8de4f973cb8a7cc6e5de0264bfd1: Status 404 returned error can't find the container with id 64394b3d190ff71870ee0ec54023bc63794f8de4f973cb8a7cc6e5de0264bfd1 Oct 30 16:45:01.786377 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:01.786276 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:01.797131 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:01.797107 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7"] Oct 30 16:45:01.797470 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:45:01.797439 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca0e07b_d937_48e9_9261_9b5182ffba5b.slice/crio-9c381656b5548c50fa5fbd486486d0a794bef193312096a1f035503972167a1b WatchSource:0}: Error finding container 9c381656b5548c50fa5fbd486486d0a794bef193312096a1f035503972167a1b: Status 404 returned error can't find the container with id 9c381656b5548c50fa5fbd486486d0a794bef193312096a1f035503972167a1b Oct 30 16:45:01.803916 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:45:01.802074 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde229d2f_d50d_4469_b884_fe682c4e457b.slice/crio-300f803edfc8776c4937311b9934cbb1b4c29d742da1044670b7b21fc6cdf1b8 WatchSource:0}: Error finding container 300f803edfc8776c4937311b9934cbb1b4c29d742da1044670b7b21fc6cdf1b8: Status 404 returned error can't find the container with id 300f803edfc8776c4937311b9934cbb1b4c29d742da1044670b7b21fc6cdf1b8 Oct 30 16:45:01.804088 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:01.803980 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9"] Oct 30 16:45:01.821503 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:01.821452 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" event={"ID":"f5812171-b69d-461b-8cfb-d23a17f7c64b","Type":"ContainerStarted","Data":"ad58a51b2a41315bebd7d15eb31965e4ed3ea446abf680f5d98a8285070d915b"} Oct 30 16:45:01.828549 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:01.828509 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" event={"ID":"89525800-714a-4a94-a37a-170e7b6da7d0","Type":"ContainerStarted","Data":"f8a1c04a0727c1e58c58ab5976db738473678a84fa53918754254ed5d9ea7dd5"} Oct 30 16:45:01.878673 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:01.876876 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph"] Oct 30 16:45:01.885795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:01.885724 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" event={"ID":"8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc","Type":"ContainerStarted","Data":"64394b3d190ff71870ee0ec54023bc63794f8de4f973cb8a7cc6e5de0264bfd1"} Oct 30 16:45:01.918501 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:01.917553 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" event={"ID":"44941116-b5e2-4ed0-81d5-e5bb1e8f7678","Type":"ContainerStarted","Data":"8bc1f33c54411a3d28ea83965417f85bc45ec50bf0f748a4953c1e21a61b78f1"} Oct 30 16:45:01.928118 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:01.925863 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" event={"ID":"fca0e07b-d937-48e9-9261-9b5182ffba5b","Type":"ContainerStarted","Data":"9c381656b5548c50fa5fbd486486d0a794bef193312096a1f035503972167a1b"} Oct 30 16:45:01.930814 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:01.929160 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" event={"ID":"55faa818-b86f-4438-974e-f753872814d1","Type":"ContainerStarted","Data":"ee5cc65c0b4e518e96c2ef7b7e2248334ae23c7e1167f683ef1672e912b0517a"} Oct 30 16:45:02.031614 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.031517 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4"] Oct 30 16:45:02.037746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.037717 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00adcb0a-prxz4"] Oct 30 16:45:02.414237 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.414201 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e89a950-1f55-4aa9-8e35-59472d90d654" path="/var/lib/kubelet/pods/0e89a950-1f55-4aa9-8e35-59472d90d654/volumes" Oct 30 16:45:02.414742 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.414723 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49f93fed-b0a8-4c0b-9bb9-16526a966a13" path="/var/lib/kubelet/pods/49f93fed-b0a8-4c0b-9bb9-16526a966a13/volumes" Oct 30 16:45:02.946451 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.946407 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" event={"ID":"fca0e07b-d937-48e9-9261-9b5182ffba5b","Type":"ContainerStarted","Data":"debb1d61d1b0166b9edba12090c4ce06b26d0d2cf9ae4cbd35bd480ac519dbf8"} Oct 30 16:45:02.949819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.949789 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" event={"ID":"aa8cd066-6dc6-4480-9e56-b7a45efaf112","Type":"ContainerStarted","Data":"d7dc8303110e7c714aecddc4fa6da41f56d770e269aa4e6126726b7f1d651824"} Oct 30 16:45:02.949946 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.949844 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" event={"ID":"aa8cd066-6dc6-4480-9e56-b7a45efaf112","Type":"ContainerStarted","Data":"aabf87dd15c9f2cbb010e1ca78e46217cd11107c4fc598f4ea32563ea9219c0d"} Oct 30 16:45:02.954347 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.954318 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" event={"ID":"55faa818-b86f-4438-974e-f753872814d1","Type":"ContainerStarted","Data":"045b6700dce4d078b401a9222098e42518190c363628657e962ac51d7ae56af3"} Oct 30 16:45:02.956057 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.956028 6202 generic.go:356] "Generic (PLEG): container finished" podID="f5812171-b69d-461b-8cfb-d23a17f7c64b" containerID="34e148a76b98aa8245c2f1ee3328fe17942aa7094e874d97dbfaadf0c904025e" exitCode=60 Oct 30 16:45:02.956160 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.956113 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" event={"ID":"f5812171-b69d-461b-8cfb-d23a17f7c64b","Type":"ContainerDied","Data":"34e148a76b98aa8245c2f1ee3328fe17942aa7094e874d97dbfaadf0c904025e"} Oct 30 16:45:02.956380 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.956360 6202 scope.go:122] "RemoveContainer" containerID="34e148a76b98aa8245c2f1ee3328fe17942aa7094e874d97dbfaadf0c904025e" Oct 30 16:45:02.958357 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.958332 6202 generic.go:356] "Generic (PLEG): container finished" podID="89525800-714a-4a94-a37a-170e7b6da7d0" containerID="427523df455e582e37b0f7cf19c2d12aeb346708413e4131a167ffc27e0b6a9b" exitCode=60 Oct 30 16:45:02.958506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.958486 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" event={"ID":"89525800-714a-4a94-a37a-170e7b6da7d0","Type":"ContainerDied","Data":"427523df455e582e37b0f7cf19c2d12aeb346708413e4131a167ffc27e0b6a9b"} Oct 30 16:45:02.958695 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.958642 6202 scope.go:122] "RemoveContainer" containerID="427523df455e582e37b0f7cf19c2d12aeb346708413e4131a167ffc27e0b6a9b" Oct 30 16:45:02.960786 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.960565 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" event={"ID":"de229d2f-d50d-4469-b884-fe682c4e457b","Type":"ContainerStarted","Data":"d66133b1386fffa6f99d6e0217690434980f360862142b49a9127d86345a0f4a"} Oct 30 16:45:02.960786 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.960609 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" event={"ID":"de229d2f-d50d-4469-b884-fe682c4e457b","Type":"ContainerStarted","Data":"300f803edfc8776c4937311b9934cbb1b4c29d742da1044670b7b21fc6cdf1b8"} Oct 30 16:45:02.969409 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.969381 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" event={"ID":"8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc","Type":"ContainerStarted","Data":"fdd9e2a191ea9d9c4c2216daf719838f202afe3115365cc1a23074be0a64b095"} Oct 30 16:45:02.977872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.977800 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" event={"ID":"44941116-b5e2-4ed0-81d5-e5bb1e8f7678","Type":"ContainerStarted","Data":"99aefeb92c890248f65b6233a9d8c221b334b194c36f75cd191c5055952c8214"} Oct 30 16:45:02.987089 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:02.987044 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" podStartSLOduration=2.987032464 podStartE2EDuration="2.987032464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:02.985781522 +0000 UTC m=+1416.696236259" watchObservedRunningTime="2026-10-30 16:45:02.987032464 +0000 UTC m=+1416.697487208" Oct 30 16:45:03.026343 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:03.026290 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" podStartSLOduration=3.026275043 podStartE2EDuration="3.026275043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:03.024299556 +0000 UTC m=+1416.734754300" watchObservedRunningTime="2026-10-30 16:45:03.026275043 +0000 UTC m=+1416.736729827" Oct 30 16:45:03.069275 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:03.069224 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" podStartSLOduration=3.06920937 podStartE2EDuration="3.06920937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:03.066953507 +0000 UTC m=+1416.777408252" watchObservedRunningTime="2026-10-30 16:45:03.06920937 +0000 UTC m=+1416.779664113" Oct 30 16:45:03.152962 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:03.152860 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" podStartSLOduration=3.152841336 podStartE2EDuration="3.152841336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:03.148950686 +0000 UTC m=+1416.859405451" watchObservedRunningTime="2026-10-30 16:45:03.152841336 +0000 UTC m=+1416.863296082" Oct 30 16:45:03.232023 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:03.231973 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" podStartSLOduration=3.231948831 podStartE2EDuration="3.231948831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:03.227041024 +0000 UTC m=+1416.937495769" watchObservedRunningTime="2026-10-30 16:45:03.231948831 +0000 UTC m=+1416.942403574" Oct 30 16:45:03.269750 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:03.269676 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" podStartSLOduration=3.269657233 podStartE2EDuration="3.269657233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:03.266006616 +0000 UTC m=+1416.976461359" watchObservedRunningTime="2026-10-30 16:45:03.269657233 +0000 UTC m=+1416.980111978" Oct 30 16:45:03.376135 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:03.376098 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:03.991834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:03.991788 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" event={"ID":"f5812171-b69d-461b-8cfb-d23a17f7c64b","Type":"ContainerStarted","Data":"a7e15b2c83c65c05a54e2149d4a87a38058b095d83b3fd5271578ff06f860432"} Oct 30 16:45:03.993929 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:03.993899 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" event={"ID":"89525800-714a-4a94-a37a-170e7b6da7d0","Type":"ContainerStarted","Data":"12703ee883335c4245e3c43b1e09cb43c49986b19e919005691609652ea5ed1a"} Oct 30 16:45:03.995479 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:03.995455 6202 generic.go:356] "Generic (PLEG): container finished" podID="de229d2f-d50d-4469-b884-fe682c4e457b" containerID="d66133b1386fffa6f99d6e0217690434980f360862142b49a9127d86345a0f4a" exitCode=60 Oct 30 16:45:03.995591 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:03.995535 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" event={"ID":"de229d2f-d50d-4469-b884-fe682c4e457b","Type":"ContainerDied","Data":"d66133b1386fffa6f99d6e0217690434980f360862142b49a9127d86345a0f4a"} Oct 30 16:45:03.995859 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:03.995839 6202 scope.go:122] "RemoveContainer" containerID="d66133b1386fffa6f99d6e0217690434980f360862142b49a9127d86345a0f4a" Oct 30 16:45:03.997299 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:03.997274 6202 generic.go:356] "Generic (PLEG): container finished" podID="8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc" containerID="fdd9e2a191ea9d9c4c2216daf719838f202afe3115365cc1a23074be0a64b095" exitCode=60 Oct 30 16:45:03.997401 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:03.997373 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" event={"ID":"8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc","Type":"ContainerDied","Data":"fdd9e2a191ea9d9c4c2216daf719838f202afe3115365cc1a23074be0a64b095"} Oct 30 16:45:03.997604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:03.997587 6202 scope.go:122] "RemoveContainer" containerID="fdd9e2a191ea9d9c4c2216daf719838f202afe3115365cc1a23074be0a64b095" Oct 30 16:45:03.999179 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:03.999159 6202 generic.go:356] "Generic (PLEG): container finished" podID="44941116-b5e2-4ed0-81d5-e5bb1e8f7678" containerID="99aefeb92c890248f65b6233a9d8c221b334b194c36f75cd191c5055952c8214" exitCode=60 Oct 30 16:45:03.999286 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:03.999259 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" event={"ID":"44941116-b5e2-4ed0-81d5-e5bb1e8f7678","Type":"ContainerDied","Data":"99aefeb92c890248f65b6233a9d8c221b334b194c36f75cd191c5055952c8214"} Oct 30 16:45:03.999497 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:03.999473 6202 scope.go:122] "RemoveContainer" containerID="99aefeb92c890248f65b6233a9d8c221b334b194c36f75cd191c5055952c8214" Oct 30 16:45:04.001264 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:04.001202 6202 generic.go:356] "Generic (PLEG): container finished" podID="fca0e07b-d937-48e9-9261-9b5182ffba5b" containerID="debb1d61d1b0166b9edba12090c4ce06b26d0d2cf9ae4cbd35bd480ac519dbf8" exitCode=60 Oct 30 16:45:04.001326 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:04.001279 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" event={"ID":"fca0e07b-d937-48e9-9261-9b5182ffba5b","Type":"ContainerDied","Data":"debb1d61d1b0166b9edba12090c4ce06b26d0d2cf9ae4cbd35bd480ac519dbf8"} Oct 30 16:45:04.001535 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:04.001518 6202 scope.go:122] "RemoveContainer" containerID="debb1d61d1b0166b9edba12090c4ce06b26d0d2cf9ae4cbd35bd480ac519dbf8" Oct 30 16:45:04.003047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:04.003025 6202 generic.go:356] "Generic (PLEG): container finished" podID="aa8cd066-6dc6-4480-9e56-b7a45efaf112" containerID="d7dc8303110e7c714aecddc4fa6da41f56d770e269aa4e6126726b7f1d651824" exitCode=60 Oct 30 16:45:04.003128 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:04.003105 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" event={"ID":"aa8cd066-6dc6-4480-9e56-b7a45efaf112","Type":"ContainerDied","Data":"d7dc8303110e7c714aecddc4fa6da41f56d770e269aa4e6126726b7f1d651824"} Oct 30 16:45:04.003291 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:04.003278 6202 scope.go:122] "RemoveContainer" containerID="d7dc8303110e7c714aecddc4fa6da41f56d770e269aa4e6126726b7f1d651824" Oct 30 16:45:04.004713 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:04.004687 6202 generic.go:356] "Generic (PLEG): container finished" podID="55faa818-b86f-4438-974e-f753872814d1" containerID="045b6700dce4d078b401a9222098e42518190c363628657e962ac51d7ae56af3" exitCode=7 Oct 30 16:45:04.004828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:04.004796 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" event={"ID":"55faa818-b86f-4438-974e-f753872814d1","Type":"ContainerDied","Data":"045b6700dce4d078b401a9222098e42518190c363628657e962ac51d7ae56af3"} Oct 30 16:45:04.004954 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:04.004938 6202 scope.go:122] "RemoveContainer" containerID="045b6700dce4d078b401a9222098e42518190c363628657e962ac51d7ae56af3" Oct 30 16:45:04.033996 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:04.033944 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" podStartSLOduration=4.033926958 podStartE2EDuration="4.033926958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:04.030247772 +0000 UTC m=+1417.740702517" watchObservedRunningTime="2026-10-30 16:45:04.033926958 +0000 UTC m=+1417.744381703" Oct 30 16:45:04.313794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:04.313729 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" podStartSLOduration=4.313714356 podStartE2EDuration="4.313714356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:04.309955694 +0000 UTC m=+1418.020410438" watchObservedRunningTime="2026-10-30 16:45:04.313714356 +0000 UTC m=+1418.024169101" Oct 30 16:45:04.774919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:04.773748 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:05.057887 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.056550 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" event={"ID":"55faa818-b86f-4438-974e-f753872814d1","Type":"ContainerStarted","Data":"426bde1f9d35116bc1ed36655ef4666fedfaff1a7b883c8d6151dc1180e9108b"} Oct 30 16:45:05.077573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.074919 6202 generic.go:356] "Generic (PLEG): container finished" podID="f5812171-b69d-461b-8cfb-d23a17f7c64b" containerID="a7e15b2c83c65c05a54e2149d4a87a38058b095d83b3fd5271578ff06f860432" exitCode=60 Oct 30 16:45:05.077573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.075057 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" event={"ID":"f5812171-b69d-461b-8cfb-d23a17f7c64b","Type":"ContainerDied","Data":"a7e15b2c83c65c05a54e2149d4a87a38058b095d83b3fd5271578ff06f860432"} Oct 30 16:45:05.077573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.075115 6202 scope.go:122] "RemoveContainer" containerID="34e148a76b98aa8245c2f1ee3328fe17942aa7094e874d97dbfaadf0c904025e" Oct 30 16:45:05.077573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.075673 6202 scope.go:122] "RemoveContainer" containerID="a7e15b2c83c65c05a54e2149d4a87a38058b095d83b3fd5271578ff06f860432" Oct 30 16:45:05.077573 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:05.076228 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v_odh-ai-gateway-infra(f5812171-b69d-461b-8cfb-d23a17f7c64b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" podUID="f5812171-b69d-461b-8cfb-d23a17f7c64b" Oct 30 16:45:05.112964 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.110574 6202 generic.go:356] "Generic (PLEG): container finished" podID="89525800-714a-4a94-a37a-170e7b6da7d0" containerID="12703ee883335c4245e3c43b1e09cb43c49986b19e919005691609652ea5ed1a" exitCode=60 Oct 30 16:45:05.112964 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.110721 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" event={"ID":"89525800-714a-4a94-a37a-170e7b6da7d0","Type":"ContainerDied","Data":"12703ee883335c4245e3c43b1e09cb43c49986b19e919005691609652ea5ed1a"} Oct 30 16:45:05.112964 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.111279 6202 scope.go:122] "RemoveContainer" containerID="12703ee883335c4245e3c43b1e09cb43c49986b19e919005691609652ea5ed1a" Oct 30 16:45:05.112964 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:05.111717 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw_odh-ai-gateway-infra(89525800-714a-4a94-a37a-170e7b6da7d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" podUID="89525800-714a-4a94-a37a-170e7b6da7d0" Oct 30 16:45:05.130457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.130154 6202 scope.go:122] "RemoveContainer" containerID="427523df455e582e37b0f7cf19c2d12aeb346708413e4131a167ffc27e0b6a9b" Oct 30 16:45:05.132188 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.131709 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" event={"ID":"de229d2f-d50d-4469-b884-fe682c4e457b","Type":"ContainerStarted","Data":"8eb76756405781b88c81e94d2c02ab6db8d081b683d3b87207f9c32871b0d33b"} Oct 30 16:45:05.140789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.137958 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" event={"ID":"8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc","Type":"ContainerStarted","Data":"85a25d0ffdd9a6d2bbd4ee07f22fdb06ab72ad5a31feabe0bbb45df2c476f098"} Oct 30 16:45:05.153178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.144996 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" event={"ID":"44941116-b5e2-4ed0-81d5-e5bb1e8f7678","Type":"ContainerStarted","Data":"a861c7fb3770fbf51282a52a2c52800e4ff8a7be9f37757bfa8f8a3d50767c63"} Oct 30 16:45:05.153178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.148887 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" event={"ID":"fca0e07b-d937-48e9-9261-9b5182ffba5b","Type":"ContainerStarted","Data":"7596671e95fb9b18c83cc9d2b72d0e0bbde6485ca791fb9efe933d141eea8849"} Oct 30 16:45:05.172079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.172035 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" event={"ID":"aa8cd066-6dc6-4480-9e56-b7a45efaf112","Type":"ContainerStarted","Data":"56f08e77b799d24866b534b1fd5694aec5983da3320469dde4fc027431742110"} Oct 30 16:45:05.302909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.302871 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:05.473866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.473830 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:05.523751 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.523720 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p"] Oct 30 16:45:05.527510 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.527484 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e4aca49c-8sc6p"] Oct 30 16:45:05.619139 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:05.619103 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:06.046293 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.046184 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:06.188565 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.188511 6202 generic.go:356] "Generic (PLEG): container finished" podID="fca0e07b-d937-48e9-9261-9b5182ffba5b" containerID="7596671e95fb9b18c83cc9d2b72d0e0bbde6485ca791fb9efe933d141eea8849" exitCode=60 Oct 30 16:45:06.189244 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.188967 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" event={"ID":"fca0e07b-d937-48e9-9261-9b5182ffba5b","Type":"ContainerDied","Data":"7596671e95fb9b18c83cc9d2b72d0e0bbde6485ca791fb9efe933d141eea8849"} Oct 30 16:45:06.189402 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.189289 6202 scope.go:122] "RemoveContainer" containerID="debb1d61d1b0166b9edba12090c4ce06b26d0d2cf9ae4cbd35bd480ac519dbf8" Oct 30 16:45:06.189402 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.189334 6202 scope.go:122] "RemoveContainer" containerID="7596671e95fb9b18c83cc9d2b72d0e0bbde6485ca791fb9efe933d141eea8849" Oct 30 16:45:06.189841 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:06.189814 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7_odh-ai-gateway-infra(fca0e07b-d937-48e9-9261-9b5182ffba5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" podUID="fca0e07b-d937-48e9-9261-9b5182ffba5b" Oct 30 16:45:06.192908 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.192883 6202 generic.go:356] "Generic (PLEG): container finished" podID="aa8cd066-6dc6-4480-9e56-b7a45efaf112" containerID="56f08e77b799d24866b534b1fd5694aec5983da3320469dde4fc027431742110" exitCode=60 Oct 30 16:45:06.193047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.193023 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" event={"ID":"aa8cd066-6dc6-4480-9e56-b7a45efaf112","Type":"ContainerDied","Data":"56f08e77b799d24866b534b1fd5694aec5983da3320469dde4fc027431742110"} Oct 30 16:45:06.193332 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.193312 6202 scope.go:122] "RemoveContainer" containerID="56f08e77b799d24866b534b1fd5694aec5983da3320469dde4fc027431742110" Oct 30 16:45:06.193755 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:06.193732 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph_odh-ai-gateway-infra(aa8cd066-6dc6-4480-9e56-b7a45efaf112)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" podUID="aa8cd066-6dc6-4480-9e56-b7a45efaf112" Oct 30 16:45:06.195245 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.195223 6202 generic.go:356] "Generic (PLEG): container finished" podID="55faa818-b86f-4438-974e-f753872814d1" containerID="426bde1f9d35116bc1ed36655ef4666fedfaff1a7b883c8d6151dc1180e9108b" exitCode=60 Oct 30 16:45:06.195365 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.195313 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" event={"ID":"55faa818-b86f-4438-974e-f753872814d1","Type":"ContainerDied","Data":"426bde1f9d35116bc1ed36655ef4666fedfaff1a7b883c8d6151dc1180e9108b"} Oct 30 16:45:06.195628 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.195609 6202 scope.go:122] "RemoveContainer" containerID="426bde1f9d35116bc1ed36655ef4666fedfaff1a7b883c8d6151dc1180e9108b" Oct 30 16:45:06.196072 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:06.196045 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc_odh-ai-gateway-infra(55faa818-b86f-4438-974e-f753872814d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" podUID="55faa818-b86f-4438-974e-f753872814d1" Oct 30 16:45:06.197345 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.197325 6202 scope.go:122] "RemoveContainer" containerID="a7e15b2c83c65c05a54e2149d4a87a38058b095d83b3fd5271578ff06f860432" Oct 30 16:45:06.197647 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:06.197616 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v_odh-ai-gateway-infra(f5812171-b69d-461b-8cfb-d23a17f7c64b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" podUID="f5812171-b69d-461b-8cfb-d23a17f7c64b" Oct 30 16:45:06.199161 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.199142 6202 scope.go:122] "RemoveContainer" containerID="12703ee883335c4245e3c43b1e09cb43c49986b19e919005691609652ea5ed1a" Oct 30 16:45:06.199440 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:06.199420 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw_odh-ai-gateway-infra(89525800-714a-4a94-a37a-170e7b6da7d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" podUID="89525800-714a-4a94-a37a-170e7b6da7d0" Oct 30 16:45:06.200617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.200584 6202 generic.go:356] "Generic (PLEG): container finished" podID="de229d2f-d50d-4469-b884-fe682c4e457b" containerID="8eb76756405781b88c81e94d2c02ab6db8d081b683d3b87207f9c32871b0d33b" exitCode=60 Oct 30 16:45:06.200701 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.200687 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" event={"ID":"de229d2f-d50d-4469-b884-fe682c4e457b","Type":"ContainerDied","Data":"8eb76756405781b88c81e94d2c02ab6db8d081b683d3b87207f9c32871b0d33b"} Oct 30 16:45:06.200983 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.200969 6202 scope.go:122] "RemoveContainer" containerID="8eb76756405781b88c81e94d2c02ab6db8d081b683d3b87207f9c32871b0d33b" Oct 30 16:45:06.201215 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:06.201200 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9_odh-ai-gateway-infra(de229d2f-d50d-4469-b884-fe682c4e457b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" podUID="de229d2f-d50d-4469-b884-fe682c4e457b" Oct 30 16:45:06.202978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.202912 6202 generic.go:356] "Generic (PLEG): container finished" podID="8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc" containerID="85a25d0ffdd9a6d2bbd4ee07f22fdb06ab72ad5a31feabe0bbb45df2c476f098" exitCode=60 Oct 30 16:45:06.203080 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.202989 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" event={"ID":"8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc","Type":"ContainerDied","Data":"85a25d0ffdd9a6d2bbd4ee07f22fdb06ab72ad5a31feabe0bbb45df2c476f098"} Oct 30 16:45:06.203492 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.203475 6202 scope.go:122] "RemoveContainer" containerID="85a25d0ffdd9a6d2bbd4ee07f22fdb06ab72ad5a31feabe0bbb45df2c476f098" Oct 30 16:45:06.203952 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:06.203930 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl_odh-ai-gateway-infra(8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" podUID="8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc" Oct 30 16:45:06.205026 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.205006 6202 generic.go:356] "Generic (PLEG): container finished" podID="44941116-b5e2-4ed0-81d5-e5bb1e8f7678" containerID="a861c7fb3770fbf51282a52a2c52800e4ff8a7be9f37757bfa8f8a3d50767c63" exitCode=60 Oct 30 16:45:06.205153 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.205114 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" event={"ID":"44941116-b5e2-4ed0-81d5-e5bb1e8f7678","Type":"ContainerDied","Data":"a861c7fb3770fbf51282a52a2c52800e4ff8a7be9f37757bfa8f8a3d50767c63"} Oct 30 16:45:06.205408 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.205394 6202 scope.go:122] "RemoveContainer" containerID="a861c7fb3770fbf51282a52a2c52800e4ff8a7be9f37757bfa8f8a3d50767c63" Oct 30 16:45:06.205624 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:06.205607 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782_odh-ai-gateway-infra(44941116-b5e2-4ed0-81d5-e5bb1e8f7678)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" podUID="44941116-b5e2-4ed0-81d5-e5bb1e8f7678" Oct 30 16:45:06.208278 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.208259 6202 scope.go:122] "RemoveContainer" containerID="d7dc8303110e7c714aecddc4fa6da41f56d770e269aa4e6126726b7f1d651824" Oct 30 16:45:06.232918 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.232781 6202 scope.go:122] "RemoveContainer" containerID="045b6700dce4d078b401a9222098e42518190c363628657e962ac51d7ae56af3" Oct 30 16:45:06.250372 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.250346 6202 scope.go:122] "RemoveContainer" containerID="d66133b1386fffa6f99d6e0217690434980f360862142b49a9127d86345a0f4a" Oct 30 16:45:06.266585 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.266564 6202 scope.go:122] "RemoveContainer" containerID="fdd9e2a191ea9d9c4c2216daf719838f202afe3115365cc1a23074be0a64b095" Oct 30 16:45:06.284242 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.284207 6202 scope.go:122] "RemoveContainer" containerID="99aefeb92c890248f65b6233a9d8c221b334b194c36f75cd191c5055952c8214" Oct 30 16:45:06.413316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:06.413276 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a58bf98-a76b-45f1-8be3-e1b7711a86b1" path="/var/lib/kubelet/pods/3a58bf98-a76b-45f1-8be3-e1b7711a86b1/volumes" Oct 30 16:45:07.217097 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:07.217068 6202 scope.go:122] "RemoveContainer" containerID="426bde1f9d35116bc1ed36655ef4666fedfaff1a7b883c8d6151dc1180e9108b" Oct 30 16:45:07.217576 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:07.217512 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc_odh-ai-gateway-infra(55faa818-b86f-4438-974e-f753872814d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" podUID="55faa818-b86f-4438-974e-f753872814d1" Oct 30 16:45:07.218959 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:07.218936 6202 scope.go:122] "RemoveContainer" containerID="8eb76756405781b88c81e94d2c02ab6db8d081b683d3b87207f9c32871b0d33b" Oct 30 16:45:07.219297 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:07.219278 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9_odh-ai-gateway-infra(de229d2f-d50d-4469-b884-fe682c4e457b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" podUID="de229d2f-d50d-4469-b884-fe682c4e457b" Oct 30 16:45:07.220600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:07.220574 6202 scope.go:122] "RemoveContainer" containerID="85a25d0ffdd9a6d2bbd4ee07f22fdb06ab72ad5a31feabe0bbb45df2c476f098" Oct 30 16:45:07.220922 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:07.220903 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl_odh-ai-gateway-infra(8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" podUID="8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc" Oct 30 16:45:07.222097 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:07.222079 6202 scope.go:122] "RemoveContainer" containerID="a861c7fb3770fbf51282a52a2c52800e4ff8a7be9f37757bfa8f8a3d50767c63" Oct 30 16:45:07.222352 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:07.222330 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782_odh-ai-gateway-infra(44941116-b5e2-4ed0-81d5-e5bb1e8f7678)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" podUID="44941116-b5e2-4ed0-81d5-e5bb1e8f7678" Oct 30 16:45:07.223702 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:07.223676 6202 scope.go:122] "RemoveContainer" containerID="7596671e95fb9b18c83cc9d2b72d0e0bbde6485ca791fb9efe933d141eea8849" Oct 30 16:45:07.223986 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:07.223963 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7_odh-ai-gateway-infra(fca0e07b-d937-48e9-9261-9b5182ffba5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" podUID="fca0e07b-d937-48e9-9261-9b5182ffba5b" Oct 30 16:45:07.225241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:07.225225 6202 scope.go:122] "RemoveContainer" containerID="56f08e77b799d24866b534b1fd5694aec5983da3320469dde4fc027431742110" Oct 30 16:45:07.225449 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:07.225432 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph_odh-ai-gateway-infra(aa8cd066-6dc6-4480-9e56-b7a45efaf112)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" podUID="aa8cd066-6dc6-4480-9e56-b7a45efaf112" Oct 30 16:45:07.597320 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:07.597282 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:08.622397 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:08.622367 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" Oct 30 16:45:09.273130 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:09.273092 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:10.948481 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:10.948442 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:11.933137 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:11.933111 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" Oct 30 16:45:11.933537 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:11.933510 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" Oct 30 16:45:12.416998 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:12.416966 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:12.623064 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:12.623028 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:12.743328 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:12.743225 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:12.982570 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:12.982543 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6" Oct 30 16:45:13.132483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:13.132424 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:13.641685 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:13.641647 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:13.937291 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:13.937150 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:14.066892 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:14.066829 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:14.072862 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:14.072751 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:45:14.197837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:14.197735 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:14.440794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:14.413608 6202 scope.go:122] "RemoveContainer" containerID="a7e15b2c83c65c05a54e2149d4a87a38058b095d83b3fd5271578ff06f860432" Oct 30 16:45:14.496400 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:14.484413 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:15.031399 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:15.031322 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq"] Oct 30 16:45:15.061600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:15.061562 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4fc01e-f9zbq"] Oct 30 16:45:15.354850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:15.354809 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" event={"ID":"f5812171-b69d-461b-8cfb-d23a17f7c64b","Type":"ContainerStarted","Data":"d45c5a2904a2e529599507eb34b98c8c596bcb0bbd4f4c8005d23ad8c699cc13"} Oct 30 16:45:15.442496 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:15.442461 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:15.565222 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:15.563172 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:15.836729 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:15.836697 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:16.367972 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:16.367921 6202 generic.go:356] "Generic (PLEG): container finished" podID="f5812171-b69d-461b-8cfb-d23a17f7c64b" containerID="d45c5a2904a2e529599507eb34b98c8c596bcb0bbd4f4c8005d23ad8c699cc13" exitCode=60 Oct 30 16:45:16.368462 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:16.367997 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" event={"ID":"f5812171-b69d-461b-8cfb-d23a17f7c64b","Type":"ContainerDied","Data":"d45c5a2904a2e529599507eb34b98c8c596bcb0bbd4f4c8005d23ad8c699cc13"} Oct 30 16:45:16.368462 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:16.368077 6202 scope.go:122] "RemoveContainer" containerID="a7e15b2c83c65c05a54e2149d4a87a38058b095d83b3fd5271578ff06f860432" Oct 30 16:45:16.368568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:16.368553 6202 scope.go:122] "RemoveContainer" containerID="d45c5a2904a2e529599507eb34b98c8c596bcb0bbd4f4c8005d23ad8c699cc13" Oct 30 16:45:16.369003 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:16.368980 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v_odh-ai-gateway-infra(f5812171-b69d-461b-8cfb-d23a17f7c64b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" podUID="f5812171-b69d-461b-8cfb-d23a17f7c64b" Oct 30 16:45:16.404035 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:16.404006 6202 scope.go:122] "RemoveContainer" containerID="85a25d0ffdd9a6d2bbd4ee07f22fdb06ab72ad5a31feabe0bbb45df2c476f098" Oct 30 16:45:16.404184 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:16.404101 6202 scope.go:122] "RemoveContainer" containerID="426bde1f9d35116bc1ed36655ef4666fedfaff1a7b883c8d6151dc1180e9108b" Oct 30 16:45:16.414354 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:16.414330 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a14f5f6-8af9-43d7-8775-26852b873c72" path="/var/lib/kubelet/pods/9a14f5f6-8af9-43d7-8775-26852b873c72/volumes" Oct 30 16:45:16.414696 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:16.414681 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v"] Oct 30 16:45:17.384083 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:17.384038 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" event={"ID":"8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc","Type":"ContainerStarted","Data":"3f352e66db697b7cd00cfb4b735ca57d5e3d647b1684acbabf975c15195ae713"} Oct 30 16:45:17.384556 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:17.384529 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:17.386588 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:17.386563 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" event={"ID":"55faa818-b86f-4438-974e-f753872814d1","Type":"ContainerStarted","Data":"3405ba74fcf1f2d3da60a1d95a00d4a6db9d31f13b5360690f9b3cbc23520a0c"} Oct 30 16:45:17.405955 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:17.405843 6202 scope.go:122] "RemoveContainer" containerID="56f08e77b799d24866b534b1fd5694aec5983da3320469dde4fc027431742110" Oct 30 16:45:17.451203 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:17.408238 6202 scope.go:122] "RemoveContainer" containerID="a861c7fb3770fbf51282a52a2c52800e4ff8a7be9f37757bfa8f8a3d50767c63" Oct 30 16:45:17.474801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:17.472751 6202 scope.go:122] "RemoveContainer" containerID="12703ee883335c4245e3c43b1e09cb43c49986b19e919005691609652ea5ed1a" Oct 30 16:45:17.474801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:17.473159 6202 scope.go:122] "RemoveContainer" containerID="7596671e95fb9b18c83cc9d2b72d0e0bbde6485ca791fb9efe933d141eea8849" Oct 30 16:45:17.878314 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:17.878283 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" Oct 30 16:45:17.991324 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:17.991290 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5812171-b69d-461b-8cfb-d23a17f7c64b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5812171-b69d-461b-8cfb-d23a17f7c64b-maas-api-service-ca\") pod \"f5812171-b69d-461b-8cfb-d23a17f7c64b\" (UID: \"f5812171-b69d-461b-8cfb-d23a17f7c64b\") " Oct 30 16:45:17.991518 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:17.991468 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5812171-b69d-461b-8cfb-d23a17f7c64b-kube-api-access-pzx5b\" (UniqueName: \"kubernetes.io/projected/f5812171-b69d-461b-8cfb-d23a17f7c64b-kube-api-access-pzx5b\") pod \"f5812171-b69d-461b-8cfb-d23a17f7c64b\" (UID: \"f5812171-b69d-461b-8cfb-d23a17f7c64b\") " Oct 30 16:45:17.991712 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:17.991685 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5812171-b69d-461b-8cfb-d23a17f7c64b-maas-api-service-ca" pod "f5812171-b69d-461b-8cfb-d23a17f7c64b" (UID: "f5812171-b69d-461b-8cfb-d23a17f7c64b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:17.992068 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:17.992047 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5812171-b69d-461b-8cfb-d23a17f7c64b-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.064195 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.064155 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5812171-b69d-461b-8cfb-d23a17f7c64b-kube-api-access-pzx5b" pod "f5812171-b69d-461b-8cfb-d23a17f7c64b" (UID: "f5812171-b69d-461b-8cfb-d23a17f7c64b"). InnerVolumeSpecName "kube-api-access-pzx5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:18.100243 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.095751 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzx5b\" (UniqueName: \"kubernetes.io/projected/f5812171-b69d-461b-8cfb-d23a17f7c64b-kube-api-access-pzx5b\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.397140 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.397104 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" Oct 30 16:45:18.397140 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.397132 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v" event={"ID":"f5812171-b69d-461b-8cfb-d23a17f7c64b","Type":"ContainerDied","Data":"ad58a51b2a41315bebd7d15eb31965e4ed3ea446abf680f5d98a8285070d915b"} Oct 30 16:45:18.397714 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.397231 6202 scope.go:122] "RemoveContainer" containerID="d45c5a2904a2e529599507eb34b98c8c596bcb0bbd4f4c8005d23ad8c699cc13" Oct 30 16:45:18.399818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.399743 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" event={"ID":"89525800-714a-4a94-a37a-170e7b6da7d0","Type":"ContainerStarted","Data":"733fa48f9037cae3a226fac2eb8186464f8c29a7d6202584a323c5078aa32f6e"} Oct 30 16:45:18.402844 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.402821 6202 scope.go:122] "RemoveContainer" containerID="8eb76756405781b88c81e94d2c02ab6db8d081b683d3b87207f9c32871b0d33b" Oct 30 16:45:18.406931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.403326 6202 generic.go:356] "Generic (PLEG): container finished" podID="8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc" containerID="3f352e66db697b7cd00cfb4b735ca57d5e3d647b1684acbabf975c15195ae713" exitCode=60 Oct 30 16:45:18.412404 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.412377 6202 generic.go:356] "Generic (PLEG): container finished" podID="55faa818-b86f-4438-974e-f753872814d1" containerID="3405ba74fcf1f2d3da60a1d95a00d4a6db9d31f13b5360690f9b3cbc23520a0c" exitCode=60 Oct 30 16:45:18.415371 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.415349 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" event={"ID":"8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc","Type":"ContainerDied","Data":"3f352e66db697b7cd00cfb4b735ca57d5e3d647b1684acbabf975c15195ae713"} Oct 30 16:45:18.415465 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.415387 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" event={"ID":"44941116-b5e2-4ed0-81d5-e5bb1e8f7678","Type":"ContainerStarted","Data":"28efe78132d4647ac89f1a2faec0fa8e3ad8bf3d580df664eaf0db3d0b7d0355"} Oct 30 16:45:18.415465 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.415407 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" event={"ID":"fca0e07b-d937-48e9-9261-9b5182ffba5b","Type":"ContainerStarted","Data":"6a973449814be0b36544651e426a66970bc17e16f2f84eb67097409527e52069"} Oct 30 16:45:18.415465 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.415424 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" event={"ID":"aa8cd066-6dc6-4480-9e56-b7a45efaf112","Type":"ContainerStarted","Data":"d2099f66e2295e25fdae75db37e2f126abec086bfb4d6065d0f87cc8274ef8fe"} Oct 30 16:45:18.415465 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.415429 6202 scope.go:122] "RemoveContainer" containerID="85a25d0ffdd9a6d2bbd4ee07f22fdb06ab72ad5a31feabe0bbb45df2c476f098" Oct 30 16:45:18.415465 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.415446 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" event={"ID":"55faa818-b86f-4438-974e-f753872814d1","Type":"ContainerDied","Data":"3405ba74fcf1f2d3da60a1d95a00d4a6db9d31f13b5360690f9b3cbc23520a0c"} Oct 30 16:45:18.415793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.415757 6202 scope.go:122] "RemoveContainer" containerID="3f352e66db697b7cd00cfb4b735ca57d5e3d647b1684acbabf975c15195ae713" Oct 30 16:45:18.416150 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:18.416125 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl_odh-ai-gateway-infra(8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" podUID="8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc" Oct 30 16:45:18.416248 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.416166 6202 scope.go:122] "RemoveContainer" containerID="3405ba74fcf1f2d3da60a1d95a00d4a6db9d31f13b5360690f9b3cbc23520a0c" Oct 30 16:45:18.416501 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:18.416477 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc_odh-ai-gateway-infra(55faa818-b86f-4438-974e-f753872814d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" podUID="55faa818-b86f-4438-974e-f753872814d1" Oct 30 16:45:18.435470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.435446 6202 scope.go:122] "RemoveContainer" containerID="426bde1f9d35116bc1ed36655ef4666fedfaff1a7b883c8d6151dc1180e9108b" Oct 30 16:45:18.501594 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.501501 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v"] Oct 30 16:45:18.507692 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.507651 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889645-nml2v"] Oct 30 16:45:18.512001 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.511977 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc"] Oct 30 16:45:18.570894 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:18.570858 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl"] Oct 30 16:45:19.073477 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.073441 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:19.184121 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.184022 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:19.311277 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.311237 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:19.427094 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.427062 6202 generic.go:356] "Generic (PLEG): container finished" podID="89525800-714a-4a94-a37a-170e7b6da7d0" containerID="733fa48f9037cae3a226fac2eb8186464f8c29a7d6202584a323c5078aa32f6e" exitCode=60 Oct 30 16:45:19.427516 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.427124 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" event={"ID":"89525800-714a-4a94-a37a-170e7b6da7d0","Type":"ContainerDied","Data":"733fa48f9037cae3a226fac2eb8186464f8c29a7d6202584a323c5078aa32f6e"} Oct 30 16:45:19.427516 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.427176 6202 scope.go:122] "RemoveContainer" containerID="12703ee883335c4245e3c43b1e09cb43c49986b19e919005691609652ea5ed1a" Oct 30 16:45:19.427633 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.427604 6202 scope.go:122] "RemoveContainer" containerID="733fa48f9037cae3a226fac2eb8186464f8c29a7d6202584a323c5078aa32f6e" Oct 30 16:45:19.428069 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:19.428042 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw_odh-ai-gateway-infra(89525800-714a-4a94-a37a-170e7b6da7d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" podUID="89525800-714a-4a94-a37a-170e7b6da7d0" Oct 30 16:45:19.429529 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.429508 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" event={"ID":"de229d2f-d50d-4469-b884-fe682c4e457b","Type":"ContainerStarted","Data":"6f67bf6c5941887211fb5e77042e886002852287757ebb15d94862cee2d7f330"} Oct 30 16:45:19.434539 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.434485 6202 generic.go:356] "Generic (PLEG): container finished" podID="44941116-b5e2-4ed0-81d5-e5bb1e8f7678" containerID="28efe78132d4647ac89f1a2faec0fa8e3ad8bf3d580df664eaf0db3d0b7d0355" exitCode=60 Oct 30 16:45:19.434655 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.434570 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" event={"ID":"44941116-b5e2-4ed0-81d5-e5bb1e8f7678","Type":"ContainerDied","Data":"28efe78132d4647ac89f1a2faec0fa8e3ad8bf3d580df664eaf0db3d0b7d0355"} Oct 30 16:45:19.435117 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.435096 6202 scope.go:122] "RemoveContainer" containerID="28efe78132d4647ac89f1a2faec0fa8e3ad8bf3d580df664eaf0db3d0b7d0355" Oct 30 16:45:19.435522 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:19.435495 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782_odh-ai-gateway-infra(44941116-b5e2-4ed0-81d5-e5bb1e8f7678)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" podUID="44941116-b5e2-4ed0-81d5-e5bb1e8f7678" Oct 30 16:45:19.437127 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.437106 6202 generic.go:356] "Generic (PLEG): container finished" podID="fca0e07b-d937-48e9-9261-9b5182ffba5b" containerID="6a973449814be0b36544651e426a66970bc17e16f2f84eb67097409527e52069" exitCode=60 Oct 30 16:45:19.437237 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.437180 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" event={"ID":"fca0e07b-d937-48e9-9261-9b5182ffba5b","Type":"ContainerDied","Data":"6a973449814be0b36544651e426a66970bc17e16f2f84eb67097409527e52069"} Oct 30 16:45:19.437555 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.437536 6202 scope.go:122] "RemoveContainer" containerID="6a973449814be0b36544651e426a66970bc17e16f2f84eb67097409527e52069" Oct 30 16:45:19.437955 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:19.437930 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7_odh-ai-gateway-infra(fca0e07b-d937-48e9-9261-9b5182ffba5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" podUID="fca0e07b-d937-48e9-9261-9b5182ffba5b" Oct 30 16:45:19.447809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.447733 6202 generic.go:356] "Generic (PLEG): container finished" podID="aa8cd066-6dc6-4480-9e56-b7a45efaf112" containerID="d2099f66e2295e25fdae75db37e2f126abec086bfb4d6065d0f87cc8274ef8fe" exitCode=60 Oct 30 16:45:19.447975 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.447921 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" event={"ID":"aa8cd066-6dc6-4480-9e56-b7a45efaf112","Type":"ContainerDied","Data":"d2099f66e2295e25fdae75db37e2f126abec086bfb4d6065d0f87cc8274ef8fe"} Oct 30 16:45:19.448500 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.448478 6202 scope.go:122] "RemoveContainer" containerID="d2099f66e2295e25fdae75db37e2f126abec086bfb4d6065d0f87cc8274ef8fe" Oct 30 16:45:19.449012 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:19.448984 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph_odh-ai-gateway-infra(aa8cd066-6dc6-4480-9e56-b7a45efaf112)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" podUID="aa8cd066-6dc6-4480-9e56-b7a45efaf112" Oct 30 16:45:19.453956 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.453905 6202 scope.go:122] "RemoveContainer" containerID="a861c7fb3770fbf51282a52a2c52800e4ff8a7be9f37757bfa8f8a3d50767c63" Oct 30 16:45:19.475806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.475720 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw"] Oct 30 16:45:19.494452 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.494427 6202 scope.go:122] "RemoveContainer" containerID="7596671e95fb9b18c83cc9d2b72d0e0bbde6485ca791fb9efe933d141eea8849" Oct 30 16:45:19.513955 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.513930 6202 scope.go:122] "RemoveContainer" containerID="56f08e77b799d24866b534b1fd5694aec5983da3320469dde4fc027431742110" Oct 30 16:45:19.558261 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.557663 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782"] Oct 30 16:45:19.602489 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.602434 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph"] Oct 30 16:45:19.659130 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.658875 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7"] Oct 30 16:45:19.757394 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.757368 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" Oct 30 16:45:19.761945 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.761919 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" Oct 30 16:45:19.854381 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.854350 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc-kube-api-access-jtsq6\" (UniqueName: \"kubernetes.io/projected/8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc-kube-api-access-jtsq6\") pod \"8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc\" (UID: \"8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc\") " Oct 30 16:45:19.854602 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.854460 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55faa818-b86f-4438-974e-f753872814d1-kube-api-access-mcsqb\" (UniqueName: \"kubernetes.io/projected/55faa818-b86f-4438-974e-f753872814d1-kube-api-access-mcsqb\") pod \"55faa818-b86f-4438-974e-f753872814d1\" (UID: \"55faa818-b86f-4438-974e-f753872814d1\") " Oct 30 16:45:19.854794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.854748 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc-maas-api-service-ca\") pod \"8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc\" (UID: \"8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc\") " Oct 30 16:45:19.855035 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.855010 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55faa818-b86f-4438-974e-f753872814d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55faa818-b86f-4438-974e-f753872814d1-maas-api-service-ca\") pod \"55faa818-b86f-4438-974e-f753872814d1\" (UID: \"55faa818-b86f-4438-974e-f753872814d1\") " Oct 30 16:45:19.855220 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.855190 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc-maas-api-service-ca" pod "8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc" (UID: "8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:19.855572 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.855542 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55faa818-b86f-4438-974e-f753872814d1-maas-api-service-ca" pod "55faa818-b86f-4438-974e-f753872814d1" (UID: "55faa818-b86f-4438-974e-f753872814d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:19.856126 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.856101 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:19.856223 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.856130 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55faa818-b86f-4438-974e-f753872814d1-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:19.857016 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.856989 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55faa818-b86f-4438-974e-f753872814d1-kube-api-access-mcsqb" pod "55faa818-b86f-4438-974e-f753872814d1" (UID: "55faa818-b86f-4438-974e-f753872814d1"). InnerVolumeSpecName "kube-api-access-mcsqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:19.857217 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.857194 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc-kube-api-access-jtsq6" pod "8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc" (UID: "8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc"). InnerVolumeSpecName "kube-api-access-jtsq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:19.957815 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.957676 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtsq6\" (UniqueName: \"kubernetes.io/projected/8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc-kube-api-access-jtsq6\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:19.957815 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:19.957712 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcsqb\" (UniqueName: \"kubernetes.io/projected/55faa818-b86f-4438-974e-f753872814d1-kube-api-access-mcsqb\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:20.445043 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.444998 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5812171-b69d-461b-8cfb-d23a17f7c64b" path="/var/lib/kubelet/pods/f5812171-b69d-461b-8cfb-d23a17f7c64b/volumes" Oct 30 16:45:20.557854 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.556990 6202 generic.go:356] "Generic (PLEG): container finished" podID="de229d2f-d50d-4469-b884-fe682c4e457b" containerID="6f67bf6c5941887211fb5e77042e886002852287757ebb15d94862cee2d7f330" exitCode=60 Oct 30 16:45:20.557854 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.557121 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" event={"ID":"de229d2f-d50d-4469-b884-fe682c4e457b","Type":"ContainerDied","Data":"6f67bf6c5941887211fb5e77042e886002852287757ebb15d94862cee2d7f330"} Oct 30 16:45:20.557854 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.557177 6202 scope.go:122] "RemoveContainer" containerID="8eb76756405781b88c81e94d2c02ab6db8d081b683d3b87207f9c32871b0d33b" Oct 30 16:45:20.557854 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.557739 6202 scope.go:122] "RemoveContainer" containerID="6f67bf6c5941887211fb5e77042e886002852287757ebb15d94862cee2d7f330" Oct 30 16:45:20.584130 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:20.558219 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9_odh-ai-gateway-infra(de229d2f-d50d-4469-b884-fe682c4e457b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" podUID="de229d2f-d50d-4469-b884-fe682c4e457b" Oct 30 16:45:20.584130 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.561121 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" event={"ID":"8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc","Type":"ContainerDied","Data":"64394b3d190ff71870ee0ec54023bc63794f8de4f973cb8a7cc6e5de0264bfd1"} Oct 30 16:45:20.584130 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.561232 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl" Oct 30 16:45:20.591566 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.591531 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" event={"ID":"55faa818-b86f-4438-974e-f753872814d1","Type":"ContainerDied","Data":"ee5cc65c0b4e518e96c2ef7b7e2248334ae23c7e1167f683ef1672e912b0517a"} Oct 30 16:45:20.591865 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.591846 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc" Oct 30 16:45:20.713118 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.713034 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm"] Oct 30 16:45:20.716026 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.715976 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" podUID="860b7b50-ea1d-4832-80e1-f842e1e9f74c" containerName="istio-proxy" containerID="cri-o://57243f94287d5b04ab6205bc73b886b182f6e71cf968b8720f0ad7a12f87eb85" gracePeriod=30 Oct 30 16:45:20.750881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.736436 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk"] Oct 30 16:45:20.750881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.736690 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" podUID="92906137-4350-4921-813c-cdcaf92a2143" containerName="istio-proxy" containerID="cri-o://778e36bef68f6300f8f67641050430f9b2aa17650583af9a7334eaa26422cde3" gracePeriod=30 Oct 30 16:45:20.768566 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.768122 6202 scope.go:122] "RemoveContainer" containerID="3f352e66db697b7cd00cfb4b735ca57d5e3d647b1684acbabf975c15195ae713" Oct 30 16:45:20.779577 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.779045 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm"] Oct 30 16:45:20.779577 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.779261 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" podUID="3afba1ef-8cb5-456c-8725-10ef761e3524" containerName="istio-proxy" containerID="cri-o://b72b30c3979e5948306ce9748d3d383821063e8e09947f363aee1be8d293c72b" gracePeriod=30 Oct 30 16:45:20.823071 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.822645 6202 scope.go:122] "RemoveContainer" containerID="3405ba74fcf1f2d3da60a1d95a00d4a6db9d31f13b5360690f9b3cbc23520a0c" Oct 30 16:45:20.826818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.826662 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk"] Oct 30 16:45:20.827637 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.827112 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" podUID="10786690-fb43-4289-826d-1bbe86793662" containerName="istio-proxy" containerID="cri-o://f3955af3bb8f925971b694a0658de920514c8aacfca451863deb645d971950d6" gracePeriod=30 Oct 30 16:45:20.845020 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.842755 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9"] Oct 30 16:45:20.974115 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.970076 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc"] Oct 30 16:45:20.983222 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:20.983189 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-b8af660d-29889645-lfgrc"] Oct 30 16:45:21.025515 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.025494 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" Oct 30 16:45:21.032185 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.032136 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f"] Oct 30 16:45:21.041238 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.041206 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe0871e-c8p7f"] Oct 30 16:45:21.100859 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.100738 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" Oct 30 16:45:21.125622 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.124981 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl"] Oct 30 16:45:21.125622 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.125573 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8cd066-6dc6-4480-9e56-b7a45efaf112-kube-api-access-r2c59\" (UniqueName: \"kubernetes.io/projected/aa8cd066-6dc6-4480-9e56-b7a45efaf112-kube-api-access-r2c59\") pod \"aa8cd066-6dc6-4480-9e56-b7a45efaf112\" (UID: \"aa8cd066-6dc6-4480-9e56-b7a45efaf112\") " Oct 30 16:45:21.125895 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.125715 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa8cd066-6dc6-4480-9e56-b7a45efaf112-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa8cd066-6dc6-4480-9e56-b7a45efaf112-maas-api-service-ca\") pod \"aa8cd066-6dc6-4480-9e56-b7a45efaf112\" (UID: \"aa8cd066-6dc6-4480-9e56-b7a45efaf112\") " Oct 30 16:45:21.127169 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.127084 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa8cd066-6dc6-4480-9e56-b7a45efaf112-maas-api-service-ca" pod "aa8cd066-6dc6-4480-9e56-b7a45efaf112" (UID: "aa8cd066-6dc6-4480-9e56-b7a45efaf112"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:21.128038 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.128011 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa8cd066-6dc6-4480-9e56-b7a45efaf112-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.132248 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.132175 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889645-dcrfl"] Oct 30 16:45:21.134883 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.134859 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8cd066-6dc6-4480-9e56-b7a45efaf112-kube-api-access-r2c59" pod "aa8cd066-6dc6-4480-9e56-b7a45efaf112" (UID: "aa8cd066-6dc6-4480-9e56-b7a45efaf112"). InnerVolumeSpecName "kube-api-access-r2c59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:21.175081 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.175058 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" Oct 30 16:45:21.180821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.180798 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" Oct 30 16:45:21.229850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.229719 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89525800-714a-4a94-a37a-170e7b6da7d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89525800-714a-4a94-a37a-170e7b6da7d0-maas-api-service-ca\") pod \"89525800-714a-4a94-a37a-170e7b6da7d0\" (UID: \"89525800-714a-4a94-a37a-170e7b6da7d0\") " Oct 30 16:45:21.230345 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.230304 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89525800-714a-4a94-a37a-170e7b6da7d0-kube-api-access-tdwvn\" (UniqueName: \"kubernetes.io/projected/89525800-714a-4a94-a37a-170e7b6da7d0-kube-api-access-tdwvn\") pod \"89525800-714a-4a94-a37a-170e7b6da7d0\" (UID: \"89525800-714a-4a94-a37a-170e7b6da7d0\") " Oct 30 16:45:21.242225 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.242188 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2c59\" (UniqueName: \"kubernetes.io/projected/aa8cd066-6dc6-4480-9e56-b7a45efaf112-kube-api-access-r2c59\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.242446 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.242280 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89525800-714a-4a94-a37a-170e7b6da7d0-maas-api-service-ca" pod "89525800-714a-4a94-a37a-170e7b6da7d0" (UID: "89525800-714a-4a94-a37a-170e7b6da7d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:21.253650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.253605 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89525800-714a-4a94-a37a-170e7b6da7d0-kube-api-access-tdwvn" pod "89525800-714a-4a94-a37a-170e7b6da7d0" (UID: "89525800-714a-4a94-a37a-170e7b6da7d0"). InnerVolumeSpecName "kube-api-access-tdwvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:21.350120 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.350080 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca0e07b-d937-48e9-9261-9b5182ffba5b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca0e07b-d937-48e9-9261-9b5182ffba5b-maas-api-service-ca\") pod \"fca0e07b-d937-48e9-9261-9b5182ffba5b\" (UID: \"fca0e07b-d937-48e9-9261-9b5182ffba5b\") " Oct 30 16:45:21.350505 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.350425 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44941116-b5e2-4ed0-81d5-e5bb1e8f7678-kube-api-access-tppbc\" (UniqueName: \"kubernetes.io/projected/44941116-b5e2-4ed0-81d5-e5bb1e8f7678-kube-api-access-tppbc\") pod \"44941116-b5e2-4ed0-81d5-e5bb1e8f7678\" (UID: \"44941116-b5e2-4ed0-81d5-e5bb1e8f7678\") " Oct 30 16:45:21.350648 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.350587 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca0e07b-d937-48e9-9261-9b5182ffba5b-kube-api-access-6gqcq\" (UniqueName: \"kubernetes.io/projected/fca0e07b-d937-48e9-9261-9b5182ffba5b-kube-api-access-6gqcq\") pod \"fca0e07b-d937-48e9-9261-9b5182ffba5b\" (UID: \"fca0e07b-d937-48e9-9261-9b5182ffba5b\") " Oct 30 16:45:21.350842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.350826 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44941116-b5e2-4ed0-81d5-e5bb1e8f7678-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44941116-b5e2-4ed0-81d5-e5bb1e8f7678-maas-api-service-ca\") pod \"44941116-b5e2-4ed0-81d5-e5bb1e8f7678\" (UID: \"44941116-b5e2-4ed0-81d5-e5bb1e8f7678\") " Oct 30 16:45:21.351591 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.351565 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdwvn\" (UniqueName: \"kubernetes.io/projected/89525800-714a-4a94-a37a-170e7b6da7d0-kube-api-access-tdwvn\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.352229 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.352210 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89525800-714a-4a94-a37a-170e7b6da7d0-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.352645 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.352617 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca0e07b-d937-48e9-9261-9b5182ffba5b-maas-api-service-ca" pod "fca0e07b-d937-48e9-9261-9b5182ffba5b" (UID: "fca0e07b-d937-48e9-9261-9b5182ffba5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:21.352886 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.352860 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44941116-b5e2-4ed0-81d5-e5bb1e8f7678-maas-api-service-ca" pod "44941116-b5e2-4ed0-81d5-e5bb1e8f7678" (UID: "44941116-b5e2-4ed0-81d5-e5bb1e8f7678"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:21.358752 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.358712 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca0e07b-d937-48e9-9261-9b5182ffba5b-kube-api-access-6gqcq" pod "fca0e07b-d937-48e9-9261-9b5182ffba5b" (UID: "fca0e07b-d937-48e9-9261-9b5182ffba5b"). InnerVolumeSpecName "kube-api-access-6gqcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:21.361049 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.361015 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44941116-b5e2-4ed0-81d5-e5bb1e8f7678-kube-api-access-tppbc" pod "44941116-b5e2-4ed0-81d5-e5bb1e8f7678" (UID: "44941116-b5e2-4ed0-81d5-e5bb1e8f7678"). InnerVolumeSpecName "kube-api-access-tppbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:21.454513 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.454449 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tppbc\" (UniqueName: \"kubernetes.io/projected/44941116-b5e2-4ed0-81d5-e5bb1e8f7678-kube-api-access-tppbc\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.455587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.455565 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gqcq\" (UniqueName: \"kubernetes.io/projected/fca0e07b-d937-48e9-9261-9b5182ffba5b-kube-api-access-6gqcq\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.455963 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.455938 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44941116-b5e2-4ed0-81d5-e5bb1e8f7678-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.456067 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.456054 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fca0e07b-d937-48e9-9261-9b5182ffba5b-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.492912 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.492703 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:21.605828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.605758 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" Oct 30 16:45:21.605828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.605803 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw" event={"ID":"89525800-714a-4a94-a37a-170e7b6da7d0","Type":"ContainerDied","Data":"f8a1c04a0727c1e58c58ab5976db738473678a84fa53918754254ed5d9ea7dd5"} Oct 30 16:45:21.606094 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.605861 6202 scope.go:122] "RemoveContainer" containerID="733fa48f9037cae3a226fac2eb8186464f8c29a7d6202584a323c5078aa32f6e" Oct 30 16:45:21.612880 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.612854 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" Oct 30 16:45:21.612998 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.612910 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782" event={"ID":"44941116-b5e2-4ed0-81d5-e5bb1e8f7678","Type":"ContainerDied","Data":"8bc1f33c54411a3d28ea83965417f85bc45ec50bf0f748a4953c1e21a61b78f1"} Oct 30 16:45:21.617040 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.615041 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" Oct 30 16:45:21.617040 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.615058 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7" event={"ID":"fca0e07b-d937-48e9-9261-9b5182ffba5b","Type":"ContainerDied","Data":"9c381656b5548c50fa5fbd486486d0a794bef193312096a1f035503972167a1b"} Oct 30 16:45:21.619532 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.619499 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" event={"ID":"aa8cd066-6dc6-4480-9e56-b7a45efaf112","Type":"ContainerDied","Data":"aabf87dd15c9f2cbb010e1ca78e46217cd11107c4fc598f4ea32563ea9219c0d"} Oct 30 16:45:21.619664 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.619642 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph" Oct 30 16:45:21.621154 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.621133 6202 scope.go:122] "RemoveContainer" containerID="28efe78132d4647ac89f1a2faec0fa8e3ad8bf3d580df664eaf0db3d0b7d0355" Oct 30 16:45:21.663734 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.663709 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw"] Oct 30 16:45:21.670789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.670432 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889645-k62vw"] Oct 30 16:45:21.673164 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.672920 6202 scope.go:122] "RemoveContainer" containerID="6a973449814be0b36544651e426a66970bc17e16f2f84eb67097409527e52069" Oct 30 16:45:21.685759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.685737 6202 scope.go:122] "RemoveContainer" containerID="d2099f66e2295e25fdae75db37e2f126abec086bfb4d6065d0f87cc8274ef8fe" Oct 30 16:45:21.784456 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.784380 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7"] Oct 30 16:45:21.791364 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.791339 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889645-wr9v7"] Oct 30 16:45:21.826002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.825968 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782"] Oct 30 16:45:21.833167 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.833138 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889645-h6782"] Oct 30 16:45:21.840032 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.840008 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" Oct 30 16:45:21.864441 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.864402 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph"] Oct 30 16:45:21.872211 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.872179 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889645-srvph"] Oct 30 16:45:21.978861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.978824 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de229d2f-d50d-4469-b884-fe682c4e457b-kube-api-access-9h22h\" (UniqueName: \"kubernetes.io/projected/de229d2f-d50d-4469-b884-fe682c4e457b-kube-api-access-9h22h\") pod \"de229d2f-d50d-4469-b884-fe682c4e457b\" (UID: \"de229d2f-d50d-4469-b884-fe682c4e457b\") " Oct 30 16:45:21.979158 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.979140 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de229d2f-d50d-4469-b884-fe682c4e457b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de229d2f-d50d-4469-b884-fe682c4e457b-maas-api-service-ca\") pod \"de229d2f-d50d-4469-b884-fe682c4e457b\" (UID: \"de229d2f-d50d-4469-b884-fe682c4e457b\") " Oct 30 16:45:21.979558 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.979494 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de229d2f-d50d-4469-b884-fe682c4e457b-maas-api-service-ca" pod "de229d2f-d50d-4469-b884-fe682c4e457b" (UID: "de229d2f-d50d-4469-b884-fe682c4e457b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:21.979893 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.979872 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de229d2f-d50d-4469-b884-fe682c4e457b-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.981219 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:21.981194 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de229d2f-d50d-4469-b884-fe682c4e457b-kube-api-access-9h22h" pod "de229d2f-d50d-4469-b884-fe682c4e457b" (UID: "de229d2f-d50d-4469-b884-fe682c4e457b"). InnerVolumeSpecName "kube-api-access-9h22h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:22.081288 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:22.081254 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h22h\" (UniqueName: \"kubernetes.io/projected/de229d2f-d50d-4469-b884-fe682c4e457b-kube-api-access-9h22h\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:22.325235 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:22.325204 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:22.414151 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:22.414075 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dafed04-7d9e-40e7-b28e-42b4607598ba" path="/var/lib/kubelet/pods/0dafed04-7d9e-40e7-b28e-42b4607598ba/volumes" Oct 30 16:45:22.414559 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:22.414545 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44941116-b5e2-4ed0-81d5-e5bb1e8f7678" path="/var/lib/kubelet/pods/44941116-b5e2-4ed0-81d5-e5bb1e8f7678/volumes" Oct 30 16:45:22.415009 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:22.414994 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55faa818-b86f-4438-974e-f753872814d1" path="/var/lib/kubelet/pods/55faa818-b86f-4438-974e-f753872814d1/volumes" Oct 30 16:45:22.415778 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:22.415747 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89525800-714a-4a94-a37a-170e7b6da7d0" path="/var/lib/kubelet/pods/89525800-714a-4a94-a37a-170e7b6da7d0/volumes" Oct 30 16:45:22.416201 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:22.416186 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc" path="/var/lib/kubelet/pods/8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc/volumes" Oct 30 16:45:22.416600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:22.416582 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa8cd066-6dc6-4480-9e56-b7a45efaf112" path="/var/lib/kubelet/pods/aa8cd066-6dc6-4480-9e56-b7a45efaf112/volumes" Oct 30 16:45:22.417237 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:22.417220 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca0e07b-d937-48e9-9261-9b5182ffba5b" path="/var/lib/kubelet/pods/fca0e07b-d937-48e9-9261-9b5182ffba5b/volumes" Oct 30 16:45:22.491523 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:22.491494 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:22.643280 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:22.643250 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" Oct 30 16:45:22.643454 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:22.643284 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9" event={"ID":"de229d2f-d50d-4469-b884-fe682c4e457b","Type":"ContainerDied","Data":"300f803edfc8776c4937311b9934cbb1b4c29d742da1044670b7b21fc6cdf1b8"} Oct 30 16:45:22.643454 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:22.643343 6202 scope.go:122] "RemoveContainer" containerID="6f67bf6c5941887211fb5e77042e886002852287757ebb15d94862cee2d7f330" Oct 30 16:45:22.698528 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:22.698444 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9"] Oct 30 16:45:22.702808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:22.702786 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889645-ssbx9"] Oct 30 16:45:22.830482 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:22.830446 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:23.287306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.287264 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:23.302625 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.302593 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl"] Oct 30 16:45:23.303430 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303411 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc" containerName="cleanup" Oct 30 16:45:23.303484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303432 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc" containerName="cleanup" Oct 30 16:45:23.303484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303449 6202 cpu_manager.go:414] "containerMap: removing container" podUID="89525800-714a-4a94-a37a-170e7b6da7d0" containerName="cleanup" Oct 30 16:45:23.303484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303455 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89525800-714a-4a94-a37a-170e7b6da7d0" containerName="cleanup" Oct 30 16:45:23.303484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303461 6202 cpu_manager.go:414] "containerMap: removing container" podUID="44941116-b5e2-4ed0-81d5-e5bb1e8f7678" containerName="cleanup" Oct 30 16:45:23.303484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303465 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44941116-b5e2-4ed0-81d5-e5bb1e8f7678" containerName="cleanup" Oct 30 16:45:23.303484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303472 6202 cpu_manager.go:414] "containerMap: removing container" podUID="89525800-714a-4a94-a37a-170e7b6da7d0" containerName="cleanup" Oct 30 16:45:23.303484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303477 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89525800-714a-4a94-a37a-170e7b6da7d0" containerName="cleanup" Oct 30 16:45:23.303484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303483 6202 cpu_manager.go:414] "containerMap: removing container" podUID="fca0e07b-d937-48e9-9261-9b5182ffba5b" containerName="cleanup" Oct 30 16:45:23.303484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303487 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca0e07b-d937-48e9-9261-9b5182ffba5b" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303500 6202 cpu_manager.go:414] "containerMap: removing container" podUID="89525800-714a-4a94-a37a-170e7b6da7d0" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303504 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89525800-714a-4a94-a37a-170e7b6da7d0" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303521 6202 cpu_manager.go:414] "containerMap: removing container" podUID="de229d2f-d50d-4469-b884-fe682c4e457b" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303525 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de229d2f-d50d-4469-b884-fe682c4e457b" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303539 6202 cpu_manager.go:414] "containerMap: removing container" podUID="55faa818-b86f-4438-974e-f753872814d1" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303543 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55faa818-b86f-4438-974e-f753872814d1" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303552 6202 cpu_manager.go:414] "containerMap: removing container" podUID="aa8cd066-6dc6-4480-9e56-b7a45efaf112" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303557 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8cd066-6dc6-4480-9e56-b7a45efaf112" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303568 6202 cpu_manager.go:414] "containerMap: removing container" podUID="f5812171-b69d-461b-8cfb-d23a17f7c64b" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303572 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5812171-b69d-461b-8cfb-d23a17f7c64b" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303583 6202 cpu_manager.go:414] "containerMap: removing container" podUID="de229d2f-d50d-4469-b884-fe682c4e457b" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303587 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de229d2f-d50d-4469-b884-fe682c4e457b" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303594 6202 cpu_manager.go:414] "containerMap: removing container" podUID="44941116-b5e2-4ed0-81d5-e5bb1e8f7678" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303598 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44941116-b5e2-4ed0-81d5-e5bb1e8f7678" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303604 6202 cpu_manager.go:414] "containerMap: removing container" podUID="f5812171-b69d-461b-8cfb-d23a17f7c64b" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303609 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5812171-b69d-461b-8cfb-d23a17f7c64b" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303616 6202 cpu_manager.go:414] "containerMap: removing container" podUID="55faa818-b86f-4438-974e-f753872814d1" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303620 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55faa818-b86f-4438-974e-f753872814d1" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303625 6202 cpu_manager.go:414] "containerMap: removing container" podUID="aa8cd066-6dc6-4480-9e56-b7a45efaf112" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303629 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8cd066-6dc6-4480-9e56-b7a45efaf112" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303634 6202 cpu_manager.go:414] "containerMap: removing container" podUID="de229d2f-d50d-4469-b884-fe682c4e457b" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303638 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de229d2f-d50d-4469-b884-fe682c4e457b" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303648 6202 cpu_manager.go:414] "containerMap: removing container" podUID="f5812171-b69d-461b-8cfb-d23a17f7c64b" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303651 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5812171-b69d-461b-8cfb-d23a17f7c64b" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303662 6202 cpu_manager.go:414] "containerMap: removing container" podUID="aa8cd066-6dc6-4480-9e56-b7a45efaf112" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303665 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8cd066-6dc6-4480-9e56-b7a45efaf112" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303673 6202 cpu_manager.go:414] "containerMap: removing container" podUID="fca0e07b-d937-48e9-9261-9b5182ffba5b" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303678 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca0e07b-d937-48e9-9261-9b5182ffba5b" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303693 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc" containerName="cleanup" Oct 30 16:45:23.303746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303696 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303814 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="55faa818-b86f-4438-974e-f753872814d1" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303823 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8cd066-6dc6-4480-9e56-b7a45efaf112" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303831 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca0e07b-d937-48e9-9261-9b5182ffba5b" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303837 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303843 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="89525800-714a-4a94-a37a-170e7b6da7d0" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303849 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303854 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303860 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="44941116-b5e2-4ed0-81d5-e5bb1e8f7678" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303864 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="89525800-714a-4a94-a37a-170e7b6da7d0" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303871 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca0e07b-d937-48e9-9261-9b5182ffba5b" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303877 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="89525800-714a-4a94-a37a-170e7b6da7d0" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303883 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="44941116-b5e2-4ed0-81d5-e5bb1e8f7678" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303890 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="de229d2f-d50d-4469-b884-fe682c4e457b" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303897 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5812171-b69d-461b-8cfb-d23a17f7c64b" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303904 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="de229d2f-d50d-4469-b884-fe682c4e457b" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303909 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5812171-b69d-461b-8cfb-d23a17f7c64b" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303915 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="55faa818-b86f-4438-974e-f753872814d1" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303919 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8cd066-6dc6-4480-9e56-b7a45efaf112" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303923 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca0e07b-d937-48e9-9261-9b5182ffba5b" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303928 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="de229d2f-d50d-4469-b884-fe682c4e457b" containerName="cleanup" Oct 30 16:45:23.304617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.303936 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5812171-b69d-461b-8cfb-d23a17f7c64b" containerName="cleanup" Oct 30 16:45:23.350236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.350191 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl"] Oct 30 16:45:23.352884 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.351205 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.370088 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.370059 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-968df4-openshift-default-dockercfg-5mhcj\"" Oct 30 16:45:23.456203 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.440470 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-podinfo\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.456203 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.440551 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-envoy\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.456203 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.440637 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-workload-socket\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.456203 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.440672 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-token\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.456203 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.440703 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97pn2\" (UniqueName: \"kubernetes.io/projected/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-kube-api-access-97pn2\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.456203 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.440816 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-workload-certs\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.456203 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.440844 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-credential-socket\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.456203 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.440876 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-data\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.456203 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.440904 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istiod-ca-cert\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.508160 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.508126 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:23.544289 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.544199 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-workload-socket\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.544581 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.544551 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-token\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.544753 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.544739 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97pn2\" (UniqueName: \"kubernetes.io/projected/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-kube-api-access-97pn2\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.544970 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.544949 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-workload-certs\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.549860 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.549832 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-credential-socket\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.550153 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.550131 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-data\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.550289 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.550274 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istiod-ca-cert\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.565829 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.550577 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-podinfo\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.566283 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.566243 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-envoy\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.587055 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.548359 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-workload-certs\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.587300 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.546219 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-workload-socket\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.587406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.553060 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istiod-ca-cert\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.587500 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.553977 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-credential-socket\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.587589 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.552504 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-data\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.653470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.647094 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-envoy\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.661787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.659512 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-podinfo\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.667978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.662589 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97pn2\" (UniqueName: \"kubernetes.io/projected/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-kube-api-access-97pn2\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.667978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.664879 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-token\") pod \"e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:23.672964 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.672852 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:23.683905 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:23.683867 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:24.002311 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:24.002265 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:24.032954 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:45:24.032905 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f45a80_37c0_47fe_95d2_ab1f6da1cd48.slice/crio-bc83259142479de6ee5aa819d429f39ee793101c04c5ecb38f7fb45741171180 WatchSource:0}: Error finding container bc83259142479de6ee5aa819d429f39ee793101c04c5ecb38f7fb45741171180: Status 404 returned error can't find the container with id bc83259142479de6ee5aa819d429f39ee793101c04c5ecb38f7fb45741171180 Oct 30 16:45:24.036830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:24.036569 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl"] Oct 30 16:45:24.038758 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:24.037735 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:24.038758 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:24.037845 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:24.038758 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:24.037893 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:24.414051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:24.414015 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de229d2f-d50d-4469-b884-fe682c4e457b" path="/var/lib/kubelet/pods/de229d2f-d50d-4469-b884-fe682c4e457b/volumes" Oct 30 16:45:24.705840 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:24.705742 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" event={"ID":"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48","Type":"ContainerStarted","Data":"d52da4f3cfbaf5adab63dc37792286a479d63af462dd30f0a734ca29f4e0c7cc"} Oct 30 16:45:24.705840 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:24.705820 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" event={"ID":"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48","Type":"ContainerStarted","Data":"bc83259142479de6ee5aa819d429f39ee793101c04c5ecb38f7fb45741171180"} Oct 30 16:45:24.829821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:24.824825 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" podStartSLOduration=1.824788566 podStartE2EDuration="1.824788566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:24.777070844 +0000 UTC m=+1438.487525588" watchObservedRunningTime="2026-10-30 16:45:24.824788566 +0000 UTC m=+1438.535243311" Oct 30 16:45:25.636679 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:25.636634 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:25.684220 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:25.684191 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:25.690803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:25.690736 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:25.714753 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:25.714720 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:25.715544 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:25.715526 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:45:25.759341 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:25.759312 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:26.190953 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.190530 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:45:26.218044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.216127 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-data\") pod \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " Oct 30 16:45:26.218044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.216189 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-envoy\") pod \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " Oct 30 16:45:26.218044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.216324 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-token\" (UniqueName: \"kubernetes.io/projected/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-token\") pod \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " Oct 30 16:45:26.218044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.216358 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860b7b50-ea1d-4832-80e1-f842e1e9f74c-kube-api-access-tpldp\" (UniqueName: \"kubernetes.io/projected/860b7b50-ea1d-4832-80e1-f842e1e9f74c-kube-api-access-tpldp\") pod \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " Oct 30 16:45:26.218044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.216416 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istiod-ca-cert\") pod \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " Oct 30 16:45:26.218044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.216483 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-credential-socket\") pod \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " Oct 30 16:45:26.218044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.216743 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-workload-socket\") pod \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " Oct 30 16:45:26.218044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.216896 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-podinfo\") pod \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " Oct 30 16:45:26.218044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.216930 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-workload-certs\") pod \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\" (UID: \"860b7b50-ea1d-4832-80e1-f842e1e9f74c\") " Oct 30 16:45:26.219159 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.218749 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-workload-certs" pod "860b7b50-ea1d-4832-80e1-f842e1e9f74c" (UID: "860b7b50-ea1d-4832-80e1-f842e1e9f74c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.219748 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.219558 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-credential-socket" pod "860b7b50-ea1d-4832-80e1-f842e1e9f74c" (UID: "860b7b50-ea1d-4832-80e1-f842e1e9f74c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.220131 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.219922 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-workload-socket" pod "860b7b50-ea1d-4832-80e1-f842e1e9f74c" (UID: "860b7b50-ea1d-4832-80e1-f842e1e9f74c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.220331 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.220283 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-data" pod "860b7b50-ea1d-4832-80e1-f842e1e9f74c" (UID: "860b7b50-ea1d-4832-80e1-f842e1e9f74c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.220331 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.220309 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istiod-ca-cert" pod "860b7b50-ea1d-4832-80e1-f842e1e9f74c" (UID: "860b7b50-ea1d-4832-80e1-f842e1e9f74c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:26.226411 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.226372 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-token" pod "860b7b50-ea1d-4832-80e1-f842e1e9f74c" (UID: "860b7b50-ea1d-4832-80e1-f842e1e9f74c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:26.226834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.226789 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-envoy" pod "860b7b50-ea1d-4832-80e1-f842e1e9f74c" (UID: "860b7b50-ea1d-4832-80e1-f842e1e9f74c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.226923 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.226882 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860b7b50-ea1d-4832-80e1-f842e1e9f74c-kube-api-access-tpldp" pod "860b7b50-ea1d-4832-80e1-f842e1e9f74c" (UID: "860b7b50-ea1d-4832-80e1-f842e1e9f74c"). InnerVolumeSpecName "kube-api-access-tpldp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:26.228441 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.228401 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-podinfo" pod "860b7b50-ea1d-4832-80e1-f842e1e9f74c" (UID: "860b7b50-ea1d-4832-80e1-f842e1e9f74c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:45:26.264830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.254230 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:45:26.264830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.259466 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:45:26.298279 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.298248 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:45:26.326755 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.326715 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3afba1ef-8cb5-456c-8725-10ef761e3524-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afba1ef-8cb5-456c-8725-10ef761e3524-istiod-ca-cert\") pod \"3afba1ef-8cb5-456c-8725-10ef761e3524\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " Oct 30 16:45:26.326958 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.326807 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92906137-4350-4921-813c-cdcaf92a2143-istio-token\" (UniqueName: \"kubernetes.io/projected/92906137-4350-4921-813c-cdcaf92a2143-istio-token\") pod \"92906137-4350-4921-813c-cdcaf92a2143\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " Oct 30 16:45:26.326958 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.326856 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-workload-socket\") pod \"10786690-fb43-4289-826d-1bbe86793662\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " Oct 30 16:45:26.326958 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.326906 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-workload-socket\") pod \"92906137-4350-4921-813c-cdcaf92a2143\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " Oct 30 16:45:26.326958 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.326947 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-workload-certs\") pod \"10786690-fb43-4289-826d-1bbe86793662\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " Oct 30 16:45:26.327168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.326990 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-workload-certs\") pod \"3afba1ef-8cb5-456c-8725-10ef761e3524\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " Oct 30 16:45:26.327168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327036 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-istio-envoy\") pod \"92906137-4350-4921-813c-cdcaf92a2143\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " Oct 30 16:45:26.327168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327081 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92906137-4350-4921-813c-cdcaf92a2143-kube-api-access-fhs8h\" (UniqueName: \"kubernetes.io/projected/92906137-4350-4921-813c-cdcaf92a2143-kube-api-access-fhs8h\") pod \"92906137-4350-4921-813c-cdcaf92a2143\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " Oct 30 16:45:26.327168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327132 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-workload-socket\") pod \"3afba1ef-8cb5-456c-8725-10ef761e3524\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " Oct 30 16:45:26.327304 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327184 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-workload-certs\") pod \"92906137-4350-4921-813c-cdcaf92a2143\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " Oct 30 16:45:26.327304 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327224 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-istio-data\") pod \"92906137-4350-4921-813c-cdcaf92a2143\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " Oct 30 16:45:26.327304 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327283 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10786690-fb43-4289-826d-1bbe86793662-istio-token\" (UniqueName: \"kubernetes.io/projected/10786690-fb43-4289-826d-1bbe86793662-istio-token\") pod \"10786690-fb43-4289-826d-1bbe86793662\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " Oct 30 16:45:26.327487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327373 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-podinfo\") pod \"3afba1ef-8cb5-456c-8725-10ef761e3524\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " Oct 30 16:45:26.327487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327424 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-token\" (UniqueName: \"kubernetes.io/projected/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-token\") pod \"3afba1ef-8cb5-456c-8725-10ef761e3524\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " Oct 30 16:45:26.327487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327458 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92906137-4350-4921-813c-cdcaf92a2143-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92906137-4350-4921-813c-cdcaf92a2143-istio-podinfo\") pod \"92906137-4350-4921-813c-cdcaf92a2143\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " Oct 30 16:45:26.328175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327490 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-credential-socket\") pod \"3afba1ef-8cb5-456c-8725-10ef761e3524\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " Oct 30 16:45:26.328175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327521 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10786690-fb43-4289-826d-1bbe86793662-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10786690-fb43-4289-826d-1bbe86793662-istio-podinfo\") pod \"10786690-fb43-4289-826d-1bbe86793662\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " Oct 30 16:45:26.328175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327554 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-istio-envoy\") pod \"10786690-fb43-4289-826d-1bbe86793662\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " Oct 30 16:45:26.328175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327597 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-credential-socket\") pod \"92906137-4350-4921-813c-cdcaf92a2143\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " Oct 30 16:45:26.328175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327626 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3afba1ef-8cb5-456c-8725-10ef761e3524-kube-api-access-s4ktj\" (UniqueName: \"kubernetes.io/projected/3afba1ef-8cb5-456c-8725-10ef761e3524-kube-api-access-s4ktj\") pod \"3afba1ef-8cb5-456c-8725-10ef761e3524\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " Oct 30 16:45:26.328175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327655 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-credential-socket\") pod \"10786690-fb43-4289-826d-1bbe86793662\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " Oct 30 16:45:26.328175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327694 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10786690-fb43-4289-826d-1bbe86793662-kube-api-access-4z2bq\" (UniqueName: \"kubernetes.io/projected/10786690-fb43-4289-826d-1bbe86793662-kube-api-access-4z2bq\") pod \"10786690-fb43-4289-826d-1bbe86793662\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " Oct 30 16:45:26.328175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327813 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-istio-data\") pod \"10786690-fb43-4289-826d-1bbe86793662\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " Oct 30 16:45:26.328175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327878 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10786690-fb43-4289-826d-1bbe86793662-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10786690-fb43-4289-826d-1bbe86793662-istiod-ca-cert\") pod \"10786690-fb43-4289-826d-1bbe86793662\" (UID: \"10786690-fb43-4289-826d-1bbe86793662\") " Oct 30 16:45:26.328175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327947 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-envoy\") pod \"3afba1ef-8cb5-456c-8725-10ef761e3524\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " Oct 30 16:45:26.328175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.327979 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-data\") pod \"3afba1ef-8cb5-456c-8725-10ef761e3524\" (UID: \"3afba1ef-8cb5-456c-8725-10ef761e3524\") " Oct 30 16:45:26.328175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.328038 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92906137-4350-4921-813c-cdcaf92a2143-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92906137-4350-4921-813c-cdcaf92a2143-istiod-ca-cert\") pod \"92906137-4350-4921-813c-cdcaf92a2143\" (UID: \"92906137-4350-4921-813c-cdcaf92a2143\") " Oct 30 16:45:26.329062 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.329043 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.329119 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.329067 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.329119 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.329081 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.329119 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.329092 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.329119 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.329102 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.329119 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.329113 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.329263 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.329125 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpldp\" (UniqueName: \"kubernetes.io/projected/860b7b50-ea1d-4832-80e1-f842e1e9f74c-kube-api-access-tpldp\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.329263 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.329137 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860b7b50-ea1d-4832-80e1-f842e1e9f74c-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.329263 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.329149 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860b7b50-ea1d-4832-80e1-f842e1e9f74c-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.329516 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.329496 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92906137-4350-4921-813c-cdcaf92a2143-istiod-ca-cert" pod "92906137-4350-4921-813c-cdcaf92a2143" (UID: "92906137-4350-4921-813c-cdcaf92a2143"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:26.330268 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.330199 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-credential-socket" pod "92906137-4350-4921-813c-cdcaf92a2143" (UID: "92906137-4350-4921-813c-cdcaf92a2143"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.346479 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.339447 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afba1ef-8cb5-456c-8725-10ef761e3524-istiod-ca-cert" pod "3afba1ef-8cb5-456c-8725-10ef761e3524" (UID: "3afba1ef-8cb5-456c-8725-10ef761e3524"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:26.346479 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.341844 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-istio-data" pod "92906137-4350-4921-813c-cdcaf92a2143" (UID: "92906137-4350-4921-813c-cdcaf92a2143"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.364337 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.364154 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-workload-socket" pod "10786690-fb43-4289-826d-1bbe86793662" (UID: "10786690-fb43-4289-826d-1bbe86793662"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.364653 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.364378 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-workload-socket" pod "92906137-4350-4921-813c-cdcaf92a2143" (UID: "92906137-4350-4921-813c-cdcaf92a2143"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.364653 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.364562 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-workload-certs" pod "10786690-fb43-4289-826d-1bbe86793662" (UID: "10786690-fb43-4289-826d-1bbe86793662"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.365711 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.365674 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-credential-socket" pod "10786690-fb43-4289-826d-1bbe86793662" (UID: "10786690-fb43-4289-826d-1bbe86793662"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.366420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.366342 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-workload-certs" pod "3afba1ef-8cb5-456c-8725-10ef761e3524" (UID: "3afba1ef-8cb5-456c-8725-10ef761e3524"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.369007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.368592 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10786690-fb43-4289-826d-1bbe86793662-kube-api-access-4z2bq" pod "10786690-fb43-4289-826d-1bbe86793662" (UID: "10786690-fb43-4289-826d-1bbe86793662"). InnerVolumeSpecName "kube-api-access-4z2bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:26.369261 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.369169 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-istio-data" pod "10786690-fb43-4289-826d-1bbe86793662" (UID: "10786690-fb43-4289-826d-1bbe86793662"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.369351 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.369297 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-workload-socket" pod "3afba1ef-8cb5-456c-8725-10ef761e3524" (UID: "3afba1ef-8cb5-456c-8725-10ef761e3524"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.369530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.369498 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-workload-certs" pod "92906137-4350-4921-813c-cdcaf92a2143" (UID: "92906137-4350-4921-813c-cdcaf92a2143"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.369711 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.369684 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10786690-fb43-4289-826d-1bbe86793662-istiod-ca-cert" pod "10786690-fb43-4289-826d-1bbe86793662" (UID: "10786690-fb43-4289-826d-1bbe86793662"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:26.370432 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.370194 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-data" pod "3afba1ef-8cb5-456c-8725-10ef761e3524" (UID: "3afba1ef-8cb5-456c-8725-10ef761e3524"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.370432 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.370393 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-credential-socket" pod "3afba1ef-8cb5-456c-8725-10ef761e3524" (UID: "3afba1ef-8cb5-456c-8725-10ef761e3524"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.370432 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.370412 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-istio-envoy" pod "10786690-fb43-4289-826d-1bbe86793662" (UID: "10786690-fb43-4289-826d-1bbe86793662"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.371911 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.371886 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10786690-fb43-4289-826d-1bbe86793662-istio-token" pod "10786690-fb43-4289-826d-1bbe86793662" (UID: "10786690-fb43-4289-826d-1bbe86793662"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:26.372011 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.371956 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-envoy" pod "3afba1ef-8cb5-456c-8725-10ef761e3524" (UID: "3afba1ef-8cb5-456c-8725-10ef761e3524"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.376868 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.376825 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-istio-envoy" pod "92906137-4350-4921-813c-cdcaf92a2143" (UID: "92906137-4350-4921-813c-cdcaf92a2143"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:26.377262 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.377139 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10786690-fb43-4289-826d-1bbe86793662-istio-podinfo" pod "10786690-fb43-4289-826d-1bbe86793662" (UID: "10786690-fb43-4289-826d-1bbe86793662"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:45:26.377262 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.377224 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92906137-4350-4921-813c-cdcaf92a2143-kube-api-access-fhs8h" pod "92906137-4350-4921-813c-cdcaf92a2143" (UID: "92906137-4350-4921-813c-cdcaf92a2143"). InnerVolumeSpecName "kube-api-access-fhs8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:26.377420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.377267 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afba1ef-8cb5-456c-8725-10ef761e3524-kube-api-access-s4ktj" pod "3afba1ef-8cb5-456c-8725-10ef761e3524" (UID: "3afba1ef-8cb5-456c-8725-10ef761e3524"). InnerVolumeSpecName "kube-api-access-s4ktj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:26.377807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.377581 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92906137-4350-4921-813c-cdcaf92a2143-istio-token" pod "92906137-4350-4921-813c-cdcaf92a2143" (UID: "92906137-4350-4921-813c-cdcaf92a2143"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:26.377807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.377690 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-podinfo" pod "3afba1ef-8cb5-456c-8725-10ef761e3524" (UID: "3afba1ef-8cb5-456c-8725-10ef761e3524"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:45:26.377807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.377755 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-token" pod "3afba1ef-8cb5-456c-8725-10ef761e3524" (UID: "3afba1ef-8cb5-456c-8725-10ef761e3524"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:26.379113 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.379082 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92906137-4350-4921-813c-cdcaf92a2143-istio-podinfo" pod "92906137-4350-4921-813c-cdcaf92a2143" (UID: "92906137-4350-4921-813c-cdcaf92a2143"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:45:26.432470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432385 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432426 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432440 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92906137-4350-4921-813c-cdcaf92a2143-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432453 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432468 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10786690-fb43-4289-826d-1bbe86793662-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432481 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432493 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432505 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4ktj\" (UniqueName: \"kubernetes.io/projected/3afba1ef-8cb5-456c-8725-10ef761e3524-kube-api-access-s4ktj\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432518 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432530 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z2bq\" (UniqueName: \"kubernetes.io/projected/10786690-fb43-4289-826d-1bbe86793662-kube-api-access-4z2bq\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432542 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432555 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10786690-fb43-4289-826d-1bbe86793662-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432566 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432577 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432587 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92906137-4350-4921-813c-cdcaf92a2143-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432599 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afba1ef-8cb5-456c-8725-10ef761e3524-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432610 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92906137-4350-4921-813c-cdcaf92a2143-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432621 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432633 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432646 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10786690-fb43-4289-826d-1bbe86793662-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432658 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432668 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432679 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhs8h\" (UniqueName: \"kubernetes.io/projected/92906137-4350-4921-813c-cdcaf92a2143-kube-api-access-fhs8h\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432690 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afba1ef-8cb5-456c-8725-10ef761e3524-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432702 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432713 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92906137-4350-4921-813c-cdcaf92a2143-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.432864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.432723 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10786690-fb43-4289-826d-1bbe86793662-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:26.725526 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.725438 6202 generic.go:356] "Generic (PLEG): container finished" podID="3afba1ef-8cb5-456c-8725-10ef761e3524" containerID="b72b30c3979e5948306ce9748d3d383821063e8e09947f363aee1be8d293c72b" exitCode=0 Oct 30 16:45:26.725526 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.725506 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" Oct 30 16:45:26.726053 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.725518 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" event={"ID":"3afba1ef-8cb5-456c-8725-10ef761e3524","Type":"ContainerDied","Data":"b72b30c3979e5948306ce9748d3d383821063e8e09947f363aee1be8d293c72b"} Oct 30 16:45:26.726053 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.725577 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm" event={"ID":"3afba1ef-8cb5-456c-8725-10ef761e3524","Type":"ContainerDied","Data":"fbdd9b46ce590b260fe3f10c7655ad95bf52f80e08bfa3db0aefea5d28225a9e"} Oct 30 16:45:26.726053 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.725606 6202 scope.go:122] "RemoveContainer" containerID="b72b30c3979e5948306ce9748d3d383821063e8e09947f363aee1be8d293c72b" Oct 30 16:45:26.727445 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.727419 6202 generic.go:356] "Generic (PLEG): container finished" podID="860b7b50-ea1d-4832-80e1-f842e1e9f74c" containerID="57243f94287d5b04ab6205bc73b886b182f6e71cf968b8720f0ad7a12f87eb85" exitCode=0 Oct 30 16:45:26.727544 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.727490 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" Oct 30 16:45:26.727603 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.727548 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" event={"ID":"860b7b50-ea1d-4832-80e1-f842e1e9f74c","Type":"ContainerDied","Data":"57243f94287d5b04ab6205bc73b886b182f6e71cf968b8720f0ad7a12f87eb85"} Oct 30 16:45:26.727650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.727616 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm" event={"ID":"860b7b50-ea1d-4832-80e1-f842e1e9f74c","Type":"ContainerDied","Data":"ed655a4bf18c3e7d8edfdac26cd79b02804ccad23fd4d9f0404270412d8f4f8e"} Oct 30 16:45:26.729511 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.729488 6202 generic.go:356] "Generic (PLEG): container finished" podID="92906137-4350-4921-813c-cdcaf92a2143" containerID="778e36bef68f6300f8f67641050430f9b2aa17650583af9a7334eaa26422cde3" exitCode=0 Oct 30 16:45:26.729599 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.729549 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" Oct 30 16:45:26.729599 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.729574 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" event={"ID":"92906137-4350-4921-813c-cdcaf92a2143","Type":"ContainerDied","Data":"778e36bef68f6300f8f67641050430f9b2aa17650583af9a7334eaa26422cde3"} Oct 30 16:45:26.729726 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.729626 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk" event={"ID":"92906137-4350-4921-813c-cdcaf92a2143","Type":"ContainerDied","Data":"f1669976bf6d5125c0897712297a5b2dd5bab7d5555ad27a4c78af183808165e"} Oct 30 16:45:26.731476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.731456 6202 generic.go:356] "Generic (PLEG): container finished" podID="10786690-fb43-4289-826d-1bbe86793662" containerID="f3955af3bb8f925971b694a0658de920514c8aacfca451863deb645d971950d6" exitCode=0 Oct 30 16:45:26.731557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.731517 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" Oct 30 16:45:26.731557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.731523 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" event={"ID":"10786690-fb43-4289-826d-1bbe86793662","Type":"ContainerDied","Data":"f3955af3bb8f925971b694a0658de920514c8aacfca451863deb645d971950d6"} Oct 30 16:45:26.731666 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.731583 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk" event={"ID":"10786690-fb43-4289-826d-1bbe86793662","Type":"ContainerDied","Data":"21c2df5fbd22b9203a72ca2c04099ecd0651e1e56f12c359c957ece4735aca9a"} Oct 30 16:45:26.740435 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.740415 6202 scope.go:122] "RemoveContainer" containerID="b72b30c3979e5948306ce9748d3d383821063e8e09947f363aee1be8d293c72b" Oct 30 16:45:26.740724 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:26.740704 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b72b30c3979e5948306ce9748d3d383821063e8e09947f363aee1be8d293c72b\": container with ID starting with b72b30c3979e5948306ce9748d3d383821063e8e09947f363aee1be8d293c72b not found: ID does not exist" containerID="b72b30c3979e5948306ce9748d3d383821063e8e09947f363aee1be8d293c72b" Oct 30 16:45:26.740830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.740736 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b72b30c3979e5948306ce9748d3d383821063e8e09947f363aee1be8d293c72b"} err="failed to get container status \"b72b30c3979e5948306ce9748d3d383821063e8e09947f363aee1be8d293c72b\": rpc error: code = NotFound desc = could not find container \"b72b30c3979e5948306ce9748d3d383821063e8e09947f363aee1be8d293c72b\": container with ID starting with b72b30c3979e5948306ce9748d3d383821063e8e09947f363aee1be8d293c72b not found: ID does not exist" Oct 30 16:45:26.740830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.740760 6202 scope.go:122] "RemoveContainer" containerID="57243f94287d5b04ab6205bc73b886b182f6e71cf968b8720f0ad7a12f87eb85" Oct 30 16:45:26.755092 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.755068 6202 scope.go:122] "RemoveContainer" containerID="57243f94287d5b04ab6205bc73b886b182f6e71cf968b8720f0ad7a12f87eb85" Oct 30 16:45:26.755387 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:26.755365 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57243f94287d5b04ab6205bc73b886b182f6e71cf968b8720f0ad7a12f87eb85\": container with ID starting with 57243f94287d5b04ab6205bc73b886b182f6e71cf968b8720f0ad7a12f87eb85 not found: ID does not exist" containerID="57243f94287d5b04ab6205bc73b886b182f6e71cf968b8720f0ad7a12f87eb85" Oct 30 16:45:26.755480 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.755394 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57243f94287d5b04ab6205bc73b886b182f6e71cf968b8720f0ad7a12f87eb85"} err="failed to get container status \"57243f94287d5b04ab6205bc73b886b182f6e71cf968b8720f0ad7a12f87eb85\": rpc error: code = NotFound desc = could not find container \"57243f94287d5b04ab6205bc73b886b182f6e71cf968b8720f0ad7a12f87eb85\": container with ID starting with 57243f94287d5b04ab6205bc73b886b182f6e71cf968b8720f0ad7a12f87eb85 not found: ID does not exist" Oct 30 16:45:26.755480 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.755413 6202 scope.go:122] "RemoveContainer" containerID="778e36bef68f6300f8f67641050430f9b2aa17650583af9a7334eaa26422cde3" Oct 30 16:45:26.766508 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.766489 6202 scope.go:122] "RemoveContainer" containerID="778e36bef68f6300f8f67641050430f9b2aa17650583af9a7334eaa26422cde3" Oct 30 16:45:26.766847 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:26.766816 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778e36bef68f6300f8f67641050430f9b2aa17650583af9a7334eaa26422cde3\": container with ID starting with 778e36bef68f6300f8f67641050430f9b2aa17650583af9a7334eaa26422cde3 not found: ID does not exist" containerID="778e36bef68f6300f8f67641050430f9b2aa17650583af9a7334eaa26422cde3" Oct 30 16:45:26.766941 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.766858 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778e36bef68f6300f8f67641050430f9b2aa17650583af9a7334eaa26422cde3"} err="failed to get container status \"778e36bef68f6300f8f67641050430f9b2aa17650583af9a7334eaa26422cde3\": rpc error: code = NotFound desc = could not find container \"778e36bef68f6300f8f67641050430f9b2aa17650583af9a7334eaa26422cde3\": container with ID starting with 778e36bef68f6300f8f67641050430f9b2aa17650583af9a7334eaa26422cde3 not found: ID does not exist" Oct 30 16:45:26.766941 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.766884 6202 scope.go:122] "RemoveContainer" containerID="f3955af3bb8f925971b694a0658de920514c8aacfca451863deb645d971950d6" Oct 30 16:45:26.777710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.777689 6202 scope.go:122] "RemoveContainer" containerID="f3955af3bb8f925971b694a0658de920514c8aacfca451863deb645d971950d6" Oct 30 16:45:26.778108 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:26.778085 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3955af3bb8f925971b694a0658de920514c8aacfca451863deb645d971950d6\": container with ID starting with f3955af3bb8f925971b694a0658de920514c8aacfca451863deb645d971950d6 not found: ID does not exist" containerID="f3955af3bb8f925971b694a0658de920514c8aacfca451863deb645d971950d6" Oct 30 16:45:26.778199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.778129 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3955af3bb8f925971b694a0658de920514c8aacfca451863deb645d971950d6"} err="failed to get container status \"f3955af3bb8f925971b694a0658de920514c8aacfca451863deb645d971950d6\": rpc error: code = NotFound desc = could not find container \"f3955af3bb8f925971b694a0658de920514c8aacfca451863deb645d971950d6\": container with ID starting with f3955af3bb8f925971b694a0658de920514c8aacfca451863deb645d971950d6 not found: ID does not exist" Oct 30 16:45:26.781824 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.781804 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk"] Oct 30 16:45:26.793571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.793549 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-z6tsk"] Oct 30 16:45:26.823994 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.823971 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk"] Oct 30 16:45:26.827830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.827812 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-qjvtk"] Oct 30 16:45:26.863875 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.861700 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm"] Oct 30 16:45:26.868529 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.868495 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-8nsvm"] Oct 30 16:45:26.899813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.899782 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm"] Oct 30 16:45:26.909071 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:26.909038 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-82bfm"] Oct 30 16:45:27.492544 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:27.492505 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:28.415790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:28.415527 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10786690-fb43-4289-826d-1bbe86793662" path="/var/lib/kubelet/pods/10786690-fb43-4289-826d-1bbe86793662/volumes" Oct 30 16:45:28.422993 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:28.416485 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3afba1ef-8cb5-456c-8725-10ef761e3524" path="/var/lib/kubelet/pods/3afba1ef-8cb5-456c-8725-10ef761e3524/volumes" Oct 30 16:45:28.422993 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:28.417238 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="860b7b50-ea1d-4832-80e1-f842e1e9f74c" path="/var/lib/kubelet/pods/860b7b50-ea1d-4832-80e1-f842e1e9f74c/volumes" Oct 30 16:45:28.422993 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:28.418382 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92906137-4350-4921-813c-cdcaf92a2143" path="/var/lib/kubelet/pods/92906137-4350-4921-813c-cdcaf92a2143/volumes" Oct 30 16:45:28.979665 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:28.979633 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:29.362826 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.362792 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj"] Oct 30 16:45:29.363927 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.363901 6202 cpu_manager.go:414] "containerMap: removing container" podUID="92906137-4350-4921-813c-cdcaf92a2143" containerName="istio-proxy" Oct 30 16:45:29.363927 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.363926 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92906137-4350-4921-813c-cdcaf92a2143" containerName="istio-proxy" Oct 30 16:45:29.364111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.363982 6202 cpu_manager.go:414] "containerMap: removing container" podUID="860b7b50-ea1d-4832-80e1-f842e1e9f74c" containerName="istio-proxy" Oct 30 16:45:29.364111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.363991 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860b7b50-ea1d-4832-80e1-f842e1e9f74c" containerName="istio-proxy" Oct 30 16:45:29.364111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364003 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc" containerName="cleanup" Oct 30 16:45:29.364111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364011 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dbe8ab5-ebfd-42cf-acd5-a2a279c874cc" containerName="cleanup" Oct 30 16:45:29.364111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364021 6202 cpu_manager.go:414] "containerMap: removing container" podUID="44941116-b5e2-4ed0-81d5-e5bb1e8f7678" containerName="cleanup" Oct 30 16:45:29.364111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364028 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44941116-b5e2-4ed0-81d5-e5bb1e8f7678" containerName="cleanup" Oct 30 16:45:29.364111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364040 6202 cpu_manager.go:414] "containerMap: removing container" podUID="fca0e07b-d937-48e9-9261-9b5182ffba5b" containerName="cleanup" Oct 30 16:45:29.364111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364047 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca0e07b-d937-48e9-9261-9b5182ffba5b" containerName="cleanup" Oct 30 16:45:29.364111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364083 6202 cpu_manager.go:414] "containerMap: removing container" podUID="55faa818-b86f-4438-974e-f753872814d1" containerName="cleanup" Oct 30 16:45:29.364111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364091 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55faa818-b86f-4438-974e-f753872814d1" containerName="cleanup" Oct 30 16:45:29.364111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364107 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3afba1ef-8cb5-456c-8725-10ef761e3524" containerName="istio-proxy" Oct 30 16:45:29.364507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364117 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3afba1ef-8cb5-456c-8725-10ef761e3524" containerName="istio-proxy" Oct 30 16:45:29.364507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364126 6202 cpu_manager.go:414] "containerMap: removing container" podUID="10786690-fb43-4289-826d-1bbe86793662" containerName="istio-proxy" Oct 30 16:45:29.364507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364133 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10786690-fb43-4289-826d-1bbe86793662" containerName="istio-proxy" Oct 30 16:45:29.364507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364290 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="44941116-b5e2-4ed0-81d5-e5bb1e8f7678" containerName="cleanup" Oct 30 16:45:29.364507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364308 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3afba1ef-8cb5-456c-8725-10ef761e3524" containerName="istio-proxy" Oct 30 16:45:29.364507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364318 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="10786690-fb43-4289-826d-1bbe86793662" containerName="istio-proxy" Oct 30 16:45:29.364507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364333 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="92906137-4350-4921-813c-cdcaf92a2143" containerName="istio-proxy" Oct 30 16:45:29.364507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364343 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="55faa818-b86f-4438-974e-f753872814d1" containerName="cleanup" Oct 30 16:45:29.364507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364353 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8cd066-6dc6-4480-9e56-b7a45efaf112" containerName="cleanup" Oct 30 16:45:29.364507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.364367 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="860b7b50-ea1d-4832-80e1-f842e1e9f74c" containerName="istio-proxy" Oct 30 16:45:29.371306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.371280 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.388709 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.388678 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj"] Oct 30 16:45:29.392388 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.392058 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b8af660d\"/\"kube-root-ca.crt\"" Oct 30 16:45:29.392388 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.392058 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b8af660d\"/\"openshift-service-ca.crt\"" Oct 30 16:45:29.392388 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.392182 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b8af660d\"/\"default-dockercfg-2p8g9\"" Oct 30 16:45:29.399787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.398576 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b8af660d\"/\"e2e-lifecycle-model-b8af660d-kserve-self-signed-certs\"" Oct 30 16:45:29.434675 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.434643 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:29.490138 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.490106 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-model-cache\") pod \"e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.490277 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.490151 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgcn\" (UniqueName: \"kubernetes.io/projected/d83ab97a-69c3-4758-9b33-da6bb816a575-kube-api-access-5fgcn\") pod \"e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.490351 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.490338 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-dshm\") pod \"e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.490411 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.490384 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-tmp-dir\") pod \"e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.490449 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.490436 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-home\") pod \"e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.490540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.490527 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83ab97a-69c3-4758-9b33-da6bb816a575-tls-certs\") pod \"e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.592142 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.592113 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-dshm\") pod \"e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.592330 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.592172 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-tmp-dir\") pod \"e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.592330 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.592198 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-home\") pod \"e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.592330 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.592229 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83ab97a-69c3-4758-9b33-da6bb816a575-tls-certs\") pod \"e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.592511 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.592362 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-model-cache\") pod \"e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.592511 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.592410 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgcn\" (UniqueName: \"kubernetes.io/projected/d83ab97a-69c3-4758-9b33-da6bb816a575-kube-api-access-5fgcn\") pod \"e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.592691 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.592669 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-home\") pod \"e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.592846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.592816 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-model-cache\") pod \"e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.593097 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.593045 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-tmp-dir\") pod \"e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.594585 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.594560 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-dshm\") pod \"e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.595130 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.595114 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83ab97a-69c3-4758-9b33-da6bb816a575-tls-certs\") pod \"e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.638839 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.631190 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgcn\" (UniqueName: \"kubernetes.io/projected/d83ab97a-69c3-4758-9b33-da6bb816a575-kube-api-access-5fgcn\") pod \"e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:29.715881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:29.695549 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:30.075508 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:45:30.075476 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd83ab97a_69c3_4758_9b33_da6bb816a575.slice/crio-e447c8fdd56774a1965f079f886ff5b846785595ade19ec704dc93f567f04db4 WatchSource:0}: Error finding container e447c8fdd56774a1965f079f886ff5b846785595ade19ec704dc93f567f04db4: Status 404 returned error can't find the container with id e447c8fdd56774a1965f079f886ff5b846785595ade19ec704dc93f567f04db4 Oct 30 16:45:30.075977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:30.075941 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj"] Oct 30 16:45:30.144095 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:30.144063 6202 scope.go:122] "RemoveContainer" containerID="1a8801ba81a8d89d034a606ec5162b92e61ce23d3b11076c93bb589f85721c18" Oct 30 16:45:30.156583 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:30.156545 6202 scope.go:122] "RemoveContainer" containerID="64e7e2866f0c54f0995b52354c849bf393cecad225854b2b3e0f3f42c79cd38f" Oct 30 16:45:30.170887 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:30.170862 6202 scope.go:122] "RemoveContainer" containerID="22a47864ccfc2e4360dfce577b65bad93f8a4d1bba6c8c8081958503826487a5" Oct 30 16:45:30.184681 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:30.184654 6202 scope.go:122] "RemoveContainer" containerID="fa45eb852be00c4aa611a8694f030238d953c424c49a21e386bb0c502c0b11f5" Oct 30 16:45:30.196586 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:30.196565 6202 scope.go:122] "RemoveContainer" containerID="d6d6e913625c75d5bdb2847a0d84125c01e421fb0f716e213378b7205fe95bd7" Oct 30 16:45:30.207375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:30.207353 6202 scope.go:122] "RemoveContainer" containerID="ae1ecd0319c472802f9581b1cd1aaac5effe3ccd6e94a68da1e2a35bf79f91e1" Oct 30 16:45:30.794927 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:30.794841 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" event={"ID":"d83ab97a-69c3-4758-9b33-da6bb816a575","Type":"ContainerStarted","Data":"8d450b225f03d13c4db98a0328f9f642524d34d53ea231a385ec57a96859d4c3"} Oct 30 16:45:30.794927 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:30.794884 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" event={"ID":"d83ab97a-69c3-4758-9b33-da6bb816a575","Type":"ContainerStarted","Data":"e447c8fdd56774a1965f079f886ff5b846785595ade19ec704dc93f567f04db4"} Oct 30 16:45:30.836373 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:30.836323 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" podStartSLOduration=1.836309097 podStartE2EDuration="1.836309097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:30.835238299 +0000 UTC m=+1444.545693043" watchObservedRunningTime="2026-10-30 16:45:30.836309097 +0000 UTC m=+1444.546763840" Oct 30 16:45:30.851313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:30.851280 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:31.031625 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:31.031493 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv"] Oct 30 16:45:31.040791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:31.038276 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ee42639-htrzv"] Oct 30 16:45:31.695864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:31.695827 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:32.370551 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:32.370514 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:32.454837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:32.454228 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56ddb302-59b5-476e-a04a-5c2186ff2581" path="/var/lib/kubelet/pods/56ddb302-59b5-476e-a04a-5c2186ff2581/volumes" Oct 30 16:45:33.883520 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:33.883484 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:35.542814 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:35.542706 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:36.029717 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:36.029682 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7"] Oct 30 16:45:36.041506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:36.041479 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6828c9dc-q6pt7"] Oct 30 16:45:36.412895 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:36.412858 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="416dd70d-f8a3-453f-b00b-7cd422ae3768" path="/var/lib/kubelet/pods/416dd70d-f8a3-453f-b00b-7cd422ae3768/volumes" Oct 30 16:45:37.154564 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:37.154529 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:37.319988 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:37.319951 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:37.442303 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:37.442213 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:38.507906 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:38.507872 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:38.997874 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:38.997832 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:39.002151 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.002126 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7"] Oct 30 16:45:39.011627 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.011599 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7" Oct 30 16:45:39.029386 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.029346 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7"] Oct 30 16:45:39.030364 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.030342 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:45:39.030534 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.030427 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:45:39.076078 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.076045 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:45:39.076280 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.076181 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:45:39.099411 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.099380 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt"] Oct 30 16:45:39.107364 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.107329 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt" Oct 30 16:45:39.112917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.112891 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt"] Oct 30 16:45:39.126823 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.126789 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:39.126990 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.126952 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-968df4\"" Oct 30 16:45:39.140177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.140138 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fe738b-864a-4ff6-b9f8-57d90c84788d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b11f8661-wh8v7\" (UID: \"48fe738b-864a-4ff6-b9f8-57d90c84788d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7" Oct 30 16:45:39.140356 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.140318 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fe738b-864a-4ff6-b9f8-57d90c84788d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b11f8661-wh8v7\" (UID: \"48fe738b-864a-4ff6-b9f8-57d90c84788d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7" Oct 30 16:45:39.140472 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.140449 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxjfp\" (UniqueName: \"kubernetes.io/projected/48fe738b-864a-4ff6-b9f8-57d90c84788d-kube-api-access-qxjfp\") pod \"maas-api-revoke-sub-b11f8661-wh8v7\" (UID: \"48fe738b-864a-4ff6-b9f8-57d90c84788d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7" Oct 30 16:45:39.161699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.161666 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm"] Oct 30 16:45:39.167892 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.167866 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm" Oct 30 16:45:39.184024 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.183991 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-968df4-dockercfg-5fvlv\"" Oct 30 16:45:39.186631 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.186444 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm"] Oct 30 16:45:39.297845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.254725 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxjfp\" (UniqueName: \"kubernetes.io/projected/48fe738b-864a-4ff6-b9f8-57d90c84788d-kube-api-access-qxjfp\") pod \"maas-api-revoke-sub-b11f8661-wh8v7\" (UID: \"48fe738b-864a-4ff6-b9f8-57d90c84788d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7" Oct 30 16:45:39.306875 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.298295 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fe738b-864a-4ff6-b9f8-57d90c84788d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b11f8661-wh8v7\" (UID: \"48fe738b-864a-4ff6-b9f8-57d90c84788d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7" Oct 30 16:45:39.314663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.314620 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgvv4\" (UniqueName: \"kubernetes.io/projected/adf56f97-c26b-4f47-bb3d-8d1b43b2b800-kube-api-access-xgvv4\") pod \"payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt\" (UID: \"adf56f97-c26b-4f47-bb3d-8d1b43b2b800\") " pod="openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt" Oct 30 16:45:39.323028 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.260211 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:39.323283 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.323251 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd634aa-9f6b-4341-a572-2e24161d692f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm\" (UID: \"fcd634aa-9f6b-4341-a572-2e24161d692f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm" Oct 30 16:45:39.334791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.323403 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nzks\" (UniqueName: \"kubernetes.io/projected/fcd634aa-9f6b-4341-a572-2e24161d692f-kube-api-access-9nzks\") pod \"payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm\" (UID: \"fcd634aa-9f6b-4341-a572-2e24161d692f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm" Oct 30 16:45:39.334791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.325626 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fe738b-864a-4ff6-b9f8-57d90c84788d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b11f8661-wh8v7\" (UID: \"48fe738b-864a-4ff6-b9f8-57d90c84788d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7" Oct 30 16:45:39.334791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.326293 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf56f97-c26b-4f47-bb3d-8d1b43b2b800-plugins-config-volume\") pod \"payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt\" (UID: \"adf56f97-c26b-4f47-bb3d-8d1b43b2b800\") " pod="openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt" Oct 30 16:45:39.334791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.327832 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fe738b-864a-4ff6-b9f8-57d90c84788d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b11f8661-wh8v7\" (UID: \"48fe738b-864a-4ff6-b9f8-57d90c84788d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7" Oct 30 16:45:39.381857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.348456 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxjfp\" (UniqueName: \"kubernetes.io/projected/48fe738b-864a-4ff6-b9f8-57d90c84788d-kube-api-access-qxjfp\") pod \"maas-api-revoke-sub-b11f8661-wh8v7\" (UID: \"48fe738b-864a-4ff6-b9f8-57d90c84788d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7" Oct 30 16:45:39.413790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.413408 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fe738b-864a-4ff6-b9f8-57d90c84788d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b11f8661-wh8v7\" (UID: \"48fe738b-864a-4ff6-b9f8-57d90c84788d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7" Oct 30 16:45:39.428869 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.428832 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgvv4\" (UniqueName: \"kubernetes.io/projected/adf56f97-c26b-4f47-bb3d-8d1b43b2b800-kube-api-access-xgvv4\") pod \"payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt\" (UID: \"adf56f97-c26b-4f47-bb3d-8d1b43b2b800\") " pod="openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt" Oct 30 16:45:39.435847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.431885 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd634aa-9f6b-4341-a572-2e24161d692f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm\" (UID: \"fcd634aa-9f6b-4341-a572-2e24161d692f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm" Oct 30 16:45:39.435847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.431967 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nzks\" (UniqueName: \"kubernetes.io/projected/fcd634aa-9f6b-4341-a572-2e24161d692f-kube-api-access-9nzks\") pod \"payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm\" (UID: \"fcd634aa-9f6b-4341-a572-2e24161d692f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm" Oct 30 16:45:39.435847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.432180 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf56f97-c26b-4f47-bb3d-8d1b43b2b800-plugins-config-volume\") pod \"payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt\" (UID: \"adf56f97-c26b-4f47-bb3d-8d1b43b2b800\") " pod="openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt" Oct 30 16:45:39.435847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.433462 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf56f97-c26b-4f47-bb3d-8d1b43b2b800-plugins-config-volume\") pod \"payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt\" (UID: \"adf56f97-c26b-4f47-bb3d-8d1b43b2b800\") " pod="openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt" Oct 30 16:45:39.435847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.434297 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd634aa-9f6b-4341-a572-2e24161d692f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm\" (UID: \"fcd634aa-9f6b-4341-a572-2e24161d692f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm" Oct 30 16:45:39.471793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.468647 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgvv4\" (UniqueName: \"kubernetes.io/projected/adf56f97-c26b-4f47-bb3d-8d1b43b2b800-kube-api-access-xgvv4\") pod \"payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt\" (UID: \"adf56f97-c26b-4f47-bb3d-8d1b43b2b800\") " pod="openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt" Oct 30 16:45:39.471793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.470688 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nzks\" (UniqueName: \"kubernetes.io/projected/fcd634aa-9f6b-4341-a572-2e24161d692f-kube-api-access-9nzks\") pod \"payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm\" (UID: \"fcd634aa-9f6b-4341-a572-2e24161d692f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm" Oct 30 16:45:39.493805 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.488806 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm" Oct 30 16:45:39.643652 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.643604 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7" Oct 30 16:45:39.695868 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.695744 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:39.715485 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.715191 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:39.727209 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.727182 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt" Oct 30 16:45:39.826296 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.826271 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm"] Oct 30 16:45:39.893327 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.893284 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm" event={"ID":"fcd634aa-9f6b-4341-a572-2e24161d692f","Type":"ContainerStarted","Data":"87cce2b9c88d21f1333faa678388b37413c1ba905dd7832a0cc9df35be2d8a86"} Oct 30 16:45:39.918021 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.917991 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:45:39.977216 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:39.976038 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7"] Oct 30 16:45:40.076085 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:40.076062 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt"] Oct 30 16:45:40.077306 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:45:40.077275 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf56f97_c26b_4f47_bb3d_8d1b43b2b800.slice/crio-639e55d8478582a418c3f726474bd3d852c8247bd8b2deec28c84341fad768ae WatchSource:0}: Error finding container 639e55d8478582a418c3f726474bd3d852c8247bd8b2deec28c84341fad768ae: Status 404 returned error can't find the container with id 639e55d8478582a418c3f726474bd3d852c8247bd8b2deec28c84341fad768ae Oct 30 16:45:40.905452 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:40.905419 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt" event={"ID":"adf56f97-c26b-4f47-bb3d-8d1b43b2b800","Type":"ContainerStarted","Data":"6d0421e1730cf350f4038ff3c5f0d11596896aa27c0c0b8271fe777e5c8716ca"} Oct 30 16:45:40.905917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:40.905472 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt" event={"ID":"adf56f97-c26b-4f47-bb3d-8d1b43b2b800","Type":"ContainerStarted","Data":"639e55d8478582a418c3f726474bd3d852c8247bd8b2deec28c84341fad768ae"} Oct 30 16:45:40.906445 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:40.906416 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt" Oct 30 16:45:40.938342 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:40.938292 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7" event={"ID":"48fe738b-864a-4ff6-b9f8-57d90c84788d","Type":"ContainerStarted","Data":"9c3f2cc44e023f1d325ab0396d0a4cb7df04b8e335e590272d965df44fd2deef"} Oct 30 16:45:40.938650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:40.938609 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7" event={"ID":"48fe738b-864a-4ff6-b9f8-57d90c84788d","Type":"ContainerStarted","Data":"3ae2e5a86197ba50a7f497761a76aed3cbe251c75f79a7dd3cd02f6bd62ecd99"} Oct 30 16:45:40.942184 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:40.941357 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm" event={"ID":"fcd634aa-9f6b-4341-a572-2e24161d692f","Type":"ContainerStarted","Data":"c69d96d9c51661fef2168e7fde6fa329cb6f133d08bec5effda6d3c5253a1a4b"} Oct 30 16:45:40.942184 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:40.941408 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm" Oct 30 16:45:40.959781 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:40.953748 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt" podStartSLOduration=1.953730306 podStartE2EDuration="1.953730306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:40.949622947 +0000 UTC m=+1454.660077692" watchObservedRunningTime="2026-10-30 16:45:40.953730306 +0000 UTC m=+1454.664185054" Oct 30 16:45:41.077327 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:41.077251 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm" podStartSLOduration=2.077227957 podStartE2EDuration="2.077227957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:41.070140289 +0000 UTC m=+1454.780595033" watchObservedRunningTime="2026-10-30 16:45:41.077227957 +0000 UTC m=+1454.787682707" Oct 30 16:45:41.088813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:41.081074 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7" podStartSLOduration=3.081053735 podStartE2EDuration="3.081053735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:41.001578402 +0000 UTC m=+1454.712033146" watchObservedRunningTime="2026-10-30 16:45:41.081053735 +0000 UTC m=+1454.791508480" Oct 30 16:45:41.660847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:41.656392 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:41.978557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:41.978458 6202 generic.go:356] "Generic (PLEG): container finished" podID="48fe738b-864a-4ff6-b9f8-57d90c84788d" containerID="9c3f2cc44e023f1d325ab0396d0a4cb7df04b8e335e590272d965df44fd2deef" exitCode=0 Oct 30 16:45:41.978557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:41.978539 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7" event={"ID":"48fe738b-864a-4ff6-b9f8-57d90c84788d","Type":"ContainerDied","Data":"9c3f2cc44e023f1d325ab0396d0a4cb7df04b8e335e590272d965df44fd2deef"} Oct 30 16:45:43.232580 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:43.232548 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7" Oct 30 16:45:43.335213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:43.335165 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48fe738b-864a-4ff6-b9f8-57d90c84788d-kube-api-access-qxjfp\" (UniqueName: \"kubernetes.io/projected/48fe738b-864a-4ff6-b9f8-57d90c84788d-kube-api-access-qxjfp\") pod \"48fe738b-864a-4ff6-b9f8-57d90c84788d\" (UID: \"48fe738b-864a-4ff6-b9f8-57d90c84788d\") " Oct 30 16:45:43.335410 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:43.335305 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48fe738b-864a-4ff6-b9f8-57d90c84788d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fe738b-864a-4ff6-b9f8-57d90c84788d-maas-api-xks-ca\") pod \"48fe738b-864a-4ff6-b9f8-57d90c84788d\" (UID: \"48fe738b-864a-4ff6-b9f8-57d90c84788d\") " Oct 30 16:45:43.335489 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:43.335412 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48fe738b-864a-4ff6-b9f8-57d90c84788d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fe738b-864a-4ff6-b9f8-57d90c84788d-maas-api-openshift-ca\") pod \"48fe738b-864a-4ff6-b9f8-57d90c84788d\" (UID: \"48fe738b-864a-4ff6-b9f8-57d90c84788d\") " Oct 30 16:45:43.335935 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:43.335908 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48fe738b-864a-4ff6-b9f8-57d90c84788d-maas-api-openshift-ca" pod "48fe738b-864a-4ff6-b9f8-57d90c84788d" (UID: "48fe738b-864a-4ff6-b9f8-57d90c84788d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:43.336177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:43.336155 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fe738b-864a-4ff6-b9f8-57d90c84788d-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:43.360971 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:43.354882 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48fe738b-864a-4ff6-b9f8-57d90c84788d-maas-api-xks-ca" pod "48fe738b-864a-4ff6-b9f8-57d90c84788d" (UID: "48fe738b-864a-4ff6-b9f8-57d90c84788d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:45:43.360971 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:43.359954 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48fe738b-864a-4ff6-b9f8-57d90c84788d-kube-api-access-qxjfp" pod "48fe738b-864a-4ff6-b9f8-57d90c84788d" (UID: "48fe738b-864a-4ff6-b9f8-57d90c84788d"). InnerVolumeSpecName "kube-api-access-qxjfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:43.437730 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:43.437695 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxjfp\" (UniqueName: \"kubernetes.io/projected/48fe738b-864a-4ff6-b9f8-57d90c84788d-kube-api-access-qxjfp\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:43.437730 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:43.437726 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fe738b-864a-4ff6-b9f8-57d90c84788d-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:45:43.463789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:43.463675 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:43.999194 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:43.999158 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7" event={"ID":"48fe738b-864a-4ff6-b9f8-57d90c84788d","Type":"ContainerDied","Data":"3ae2e5a86197ba50a7f497761a76aed3cbe251c75f79a7dd3cd02f6bd62ecd99"} Oct 30 16:45:43.999194 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:43.999210 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ae2e5a86197ba50a7f497761a76aed3cbe251c75f79a7dd3cd02f6bd62ecd99" Oct 30 16:45:43.999414 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:43.999174 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7" Oct 30 16:45:44.103399 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:45:44.103358 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:45:45.135947 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:45.135871 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:46.959943 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:46.959847 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:48.731893 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:48.731859 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:49.314935 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.314896 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:45:49.315120 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.314977 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" podUID="cb03379b-4139-4f06-a67b-73249017ccb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:49.344277 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.344236 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:49.703530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.703493 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h"] Oct 30 16:45:49.704546 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.704522 6202 cpu_manager.go:414] "containerMap: removing container" podUID="48fe738b-864a-4ff6-b9f8-57d90c84788d" containerName="revoke-keys" Oct 30 16:45:49.704546 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.704546 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48fe738b-864a-4ff6-b9f8-57d90c84788d" containerName="revoke-keys" Oct 30 16:45:49.704752 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.704725 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="48fe738b-864a-4ff6-b9f8-57d90c84788d" containerName="revoke-keys" Oct 30 16:45:49.708552 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.708532 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.715503 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.715474 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:49.732979 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.732946 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-05810b-openshift-default-dockercfg-cf5d9\"" Oct 30 16:45:49.758144 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.758107 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k"] Oct 30 16:45:49.806820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.806790 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:49.809272 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.809026 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h"] Oct 30 16:45:49.813061 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.813035 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.813200 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.813093 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.813200 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.813118 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.813200 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.813170 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.813382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.813205 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.813382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.813249 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5hs5\" (UniqueName: \"kubernetes.io/projected/6eac6c0b-6f5c-4e8c-8034-a045f3530426-kube-api-access-d5hs5\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.813382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.813281 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.813382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.813306 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.813382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.813334 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.814606 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.814535 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k"] Oct 30 16:45:49.914974 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.914938 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.915144 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.915005 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e38c4872-9809-48e2-a8f5-9a0076deddd5-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:49.915144 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.915126 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:49.915270 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.915251 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:49.915325 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.915311 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.915403 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.915384 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.915448 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.915420 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.915503 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.915460 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:49.915503 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.915494 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:49.915580 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.915526 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.915872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.915750 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.915872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.915800 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:49.915872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.915813 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.915872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.915838 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:49.916186 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.915914 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:49.916186 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.915936 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.916186 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.916004 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5hs5\" (UniqueName: \"kubernetes.io/projected/6eac6c0b-6f5c-4e8c-8034-a045f3530426-kube-api-access-d5hs5\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.916186 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.916025 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bmg8\" (UniqueName: \"kubernetes.io/projected/e38c4872-9809-48e2-a8f5-9a0076deddd5-kube-api-access-8bmg8\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:49.916186 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.916066 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.916186 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.916096 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.916508 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.916203 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.916508 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.916419 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.916623 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.916577 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.918291 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.918269 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.918424 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.918411 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.941228 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.941195 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5hs5\" (UniqueName: \"kubernetes.io/projected/6eac6c0b-6f5c-4e8c-8034-a045f3530426-kube-api-access-d5hs5\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:49.941688 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:49.941666 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:50.017168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.017084 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.017168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.017133 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.017405 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.017194 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.017405 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.017218 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.017405 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.017248 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.017405 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.017317 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bmg8\" (UniqueName: \"kubernetes.io/projected/e38c4872-9809-48e2-a8f5-9a0076deddd5-kube-api-access-8bmg8\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.017609 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.017437 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e38c4872-9809-48e2-a8f5-9a0076deddd5-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.017609 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.017484 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.017609 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.017552 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.017609 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.017592 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.017890 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.017646 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.017951 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.017929 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.018058 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.018041 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.018491 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.018464 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e38c4872-9809-48e2-a8f5-9a0076deddd5-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.019878 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.019847 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.024515 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.024482 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.031170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.031146 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4"] Oct 30 16:45:50.041335 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.041308 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83c165f8-b44h4"] Oct 30 16:45:50.044517 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.044495 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:50.045295 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.045229 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.047336 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.047309 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bmg8\" (UniqueName: \"kubernetes.io/projected/e38c4872-9809-48e2-a8f5-9a0076deddd5-kube-api-access-8bmg8\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.127384 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.126095 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:50.195889 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.193561 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:50.372605 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.372570 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h"] Oct 30 16:45:50.375455 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.375416 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:50.375530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.375490 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:50.375530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.375516 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:50.413357 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.413327 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="938846c6-098f-493f-834f-d3c77839aece" path="/var/lib/kubelet/pods/938846c6-098f-493f-834f-d3c77839aece/volumes" Oct 30 16:45:50.494040 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:45:50.493982 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38c4872_9809_48e2_a8f5_9a0076deddd5.slice/crio-2e161751cd9f1d2efb2d71f150bf5efe3d93615fa9ce36a45251cb187e7d6e1d WatchSource:0}: Error finding container 2e161751cd9f1d2efb2d71f150bf5efe3d93615fa9ce36a45251cb187e7d6e1d: Status 404 returned error can't find the container with id 2e161751cd9f1d2efb2d71f150bf5efe3d93615fa9ce36a45251cb187e7d6e1d Oct 30 16:45:50.494714 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.494690 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k"] Oct 30 16:45:50.498742 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.497755 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:50.498742 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.497862 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:50.498742 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.497904 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:50.603304 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.603112 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:50.610106 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.610078 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb"] Oct 30 16:45:50.616449 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.616428 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.647104 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.647075 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c8db42-openshift-default-dockercfg-nsgbm\"" Oct 30 16:45:50.775791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.768866 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-workload-socket\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.775791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.768971 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istiod-ca-cert\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.775791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.769023 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hxrj\" (UniqueName: \"kubernetes.io/projected/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-kube-api-access-6hxrj\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.775791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.769068 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-credential-socket\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.775791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.769107 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-podinfo\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.775791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.769154 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-token\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.775791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.769347 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-envoy\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.775791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.769381 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-workload-certs\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.775791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.769409 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-data\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.776724 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.775888 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb"] Oct 30 16:45:50.853792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.853544 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:50.878834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.871024 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-envoy\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.878834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.871085 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-workload-certs\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.878834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.871116 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-data\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.878834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.871262 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-workload-socket\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.878834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.871339 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istiod-ca-cert\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.878834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.871387 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hxrj\" (UniqueName: \"kubernetes.io/projected/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-kube-api-access-6hxrj\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.878834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.871431 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-credential-socket\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.878834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.871469 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-podinfo\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.878834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.871520 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-token\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.878834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.873509 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istiod-ca-cert\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.878834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.873733 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-workload-certs\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.878834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.873956 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-data\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.878834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.874145 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-workload-socket\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.878834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.874358 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-credential-socket\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.892755 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.892155 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-envoy\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:50.911793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:50.911177 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-podinfo\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:51.033293 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:51.033152 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:51.049791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:51.035198 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j"] Oct 30 16:45:51.067918 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:51.055157 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hxrj\" (UniqueName: \"kubernetes.io/projected/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-kube-api-access-6hxrj\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:51.067918 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:51.061009 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-token\") pod \"e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:51.084035 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:51.083350 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bddc700-6xc6j"] Oct 30 16:45:51.183874 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:51.183448 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" event={"ID":"e38c4872-9809-48e2-a8f5-9a0076deddd5","Type":"ContainerStarted","Data":"e4a7280f35037c8160667af6852578229aa6fa3eda75cb06cf4c8189696e6c32"} Oct 30 16:45:51.183874 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:51.183511 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" event={"ID":"e38c4872-9809-48e2-a8f5-9a0076deddd5","Type":"ContainerStarted","Data":"2e161751cd9f1d2efb2d71f150bf5efe3d93615fa9ce36a45251cb187e7d6e1d"} Oct 30 16:45:51.188353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:51.187618 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" event={"ID":"6eac6c0b-6f5c-4e8c-8034-a045f3530426","Type":"ContainerStarted","Data":"07ccaeca3e5343427dad0b1bac0130ae5e0cb7bde596c44c4f61aa391ae49b94"} Oct 30 16:45:51.188353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:51.187662 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" event={"ID":"6eac6c0b-6f5c-4e8c-8034-a045f3530426","Type":"ContainerStarted","Data":"a54fdcf3254121bca01803a5ddce214d2856a4851790e7be09fb6ffe83c2c814"} Oct 30 16:45:51.230479 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:51.230417 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" podStartSLOduration=2.230399362 podStartE2EDuration="2.230399362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:51.228326998 +0000 UTC m=+1464.938781743" watchObservedRunningTime="2026-10-30 16:45:51.230399362 +0000 UTC m=+1464.940854106" Oct 30 16:45:51.263043 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:51.262996 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:51.281955 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:51.281223 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:51.306794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:51.305812 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" podStartSLOduration=2.305787439 podStartE2EDuration="2.305787439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:51.273150474 +0000 UTC m=+1464.983605244" watchObservedRunningTime="2026-10-30 16:45:51.305787439 +0000 UTC m=+1465.016242180" Oct 30 16:45:51.571015 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:51.570898 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:51.698791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:51.691296 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb"] Oct 30 16:45:51.719318 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:45:51.719289 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83d85c0_d7aa_4bdd_9e23_0fe2e30bdaa2.slice/crio-e340adcaf5c1805299743b777184fffe14c1a3c450c9c2ad91b325acc496c5c6 WatchSource:0}: Error finding container e340adcaf5c1805299743b777184fffe14c1a3c450c9c2ad91b325acc496c5c6: Status 404 returned error can't find the container with id e340adcaf5c1805299743b777184fffe14c1a3c450c9c2ad91b325acc496c5c6 Oct 30 16:45:51.722718 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:51.722685 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:51.722830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:51.722760 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:51.722882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:51.722833 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:52.000791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.000714 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm" Oct 30 16:45:52.027977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.027915 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4"] Oct 30 16:45:52.116844 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.090175 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 16:45:52.116844 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.090236 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" podUID="6eac6c0b-6f5c-4e8c-8034-a045f3530426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 16:45:52.173796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.155881 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 16:45:52.173796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.155941 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" podUID="e38c4872-9809-48e2-a8f5-9a0076deddd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 16:45:52.185259 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.185222 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:52.358076 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.358035 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4"] Oct 30 16:45:52.365910 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.365880 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:52.374847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.373977 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:52.374847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.360890 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:52.422792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.406164 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-661509-openshift-default-dockercfg-fzxw6\"" Oct 30 16:45:52.730862 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.730737 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-credential-socket\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:52.771262 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.771214 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-token\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:52.830795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.828305 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-podinfo\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:52.835559 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.835527 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f9d3a6-5911-45b0-84fb-12b6ad4274f0" path="/var/lib/kubelet/pods/22f9d3a6-5911-45b0-84fb-12b6ad4274f0/volumes" Oct 30 16:45:52.841810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.837857 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-envoy\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:52.841810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.837910 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrwww\" (UniqueName: \"kubernetes.io/projected/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-kube-api-access-vrwww\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:52.841810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.838078 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-workload-socket\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:52.841810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.838103 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-data\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:52.841810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.838180 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istiod-ca-cert\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:52.841810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.838239 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-workload-certs\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:52.886444 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.886407 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:52.924904 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.915379 6202 generic.go:356] "Generic (PLEG): container finished" podID="cb03379b-4139-4f06-a67b-73249017ccb8" containerID="8e9e9bdaa8de63e518d25dcfcf0cb2cfe66445e6a3ea6b22fc0c517903e46b4e" exitCode=0 Oct 30 16:45:52.924904 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.916133 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" event={"ID":"cb03379b-4139-4f06-a67b-73249017ccb8","Type":"ContainerDied","Data":"8e9e9bdaa8de63e518d25dcfcf0cb2cfe66445e6a3ea6b22fc0c517903e46b4e"} Oct 30 16:45:52.924904 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.916667 6202 scope.go:122] "RemoveContainer" containerID="8e9e9bdaa8de63e518d25dcfcf0cb2cfe66445e6a3ea6b22fc0c517903e46b4e" Oct 30 16:45:52.932386 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.932352 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:52.932576 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.932560 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:52.932672 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.932660 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:52.934336 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.934309 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" event={"ID":"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2","Type":"ContainerStarted","Data":"7d88a428f5b39bb6d46ebb63910aaf26c2b899da2884e7ff3b06eee592b30ce5"} Oct 30 16:45:52.934431 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.934361 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" event={"ID":"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2","Type":"ContainerStarted","Data":"e340adcaf5c1805299743b777184fffe14c1a3c450c9c2ad91b325acc496c5c6"} Oct 30 16:45:52.971189 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.971133 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-envoy\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:52.982327 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.982244 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrwww\" (UniqueName: \"kubernetes.io/projected/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-kube-api-access-vrwww\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:52.983151 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:52.983120 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-workload-socket\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:53.007685 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.007652 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-data\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:53.008357 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.008328 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istiod-ca-cert\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:53.008521 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.008505 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-workload-certs\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:53.031257 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.031210 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-credential-socket\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:53.275491 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.275405 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-token\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:53.275909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.275886 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-podinfo\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:53.276168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.015660 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-workload-certs\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:53.276271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.007258 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-workload-socket\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:53.276372 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.032609 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-credential-socket\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:53.276458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.038215 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-envoy\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:53.276544 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.131992 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:53.276721 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.011370 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-data\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:53.276860 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.207305 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrwww\" (UniqueName: \"kubernetes.io/projected/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-kube-api-access-vrwww\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:53.276949 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.265541 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt" Oct 30 16:45:53.305353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.184891 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 16:45:53.305353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.302746 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" podUID="6eac6c0b-6f5c-4e8c-8034-a045f3530426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 16:45:53.305353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.186613 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 16:45:53.305353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.303099 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" podUID="e38c4872-9809-48e2-a8f5-9a0076deddd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 16:45:53.305353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.012374 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istiod-ca-cert\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:53.305353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.305286 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:53.324476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.324437 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-podinfo\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:53.334896 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.334869 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj"] Oct 30 16:45:53.425114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.410466 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-token\") pod \"e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:53.431454 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.431421 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.433104 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.433080 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj"] Oct 30 16:45:53.434348 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.434330 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:53.435663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.435611 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" podStartSLOduration=3.43559696 podStartE2EDuration="3.43559696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:53.392717789 +0000 UTC m=+1467.103172534" watchObservedRunningTime="2026-10-30 16:45:53.43559696 +0000 UTC m=+1467.146051707" Oct 30 16:45:53.451796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.449141 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-1b7bb3-openshift-default-dockercfg-gqvnj\"" Oct 30 16:45:53.515253 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.515217 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-workload-certs\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.515471 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.515451 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-workload-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.515573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.515537 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-istio-envoy\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.515632 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.515591 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:53.515690 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.515629 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed723b1e-dfc2-441a-816b-1308b738215e-istio-podinfo\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.515744 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.515699 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed723b1e-dfc2-441a-816b-1308b738215e-istiod-ca-cert\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.515872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.515853 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed723b1e-dfc2-441a-816b-1308b738215e-istio-token\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.515992 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.515975 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6njgx\" (UniqueName: \"kubernetes.io/projected/ed723b1e-dfc2-441a-816b-1308b738215e-kube-api-access-6njgx\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.516252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.516223 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-credential-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.516339 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.516293 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-istio-data\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.772047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.713064 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-workload-certs\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.772047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.713118 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-workload-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.772047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.713164 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-istio-envoy\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.772047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.713268 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed723b1e-dfc2-441a-816b-1308b738215e-istio-podinfo\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.772047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.713345 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed723b1e-dfc2-441a-816b-1308b738215e-istiod-ca-cert\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.772047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.713478 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed723b1e-dfc2-441a-816b-1308b738215e-istio-token\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.772047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.713600 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6njgx\" (UniqueName: \"kubernetes.io/projected/ed723b1e-dfc2-441a-816b-1308b738215e-kube-api-access-6njgx\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.772047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.713636 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-credential-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.772047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.713679 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-istio-data\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.772047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.714226 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-istio-data\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.772047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.714466 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-workload-certs\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.772047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.714711 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-workload-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.772047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.716367 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed723b1e-dfc2-441a-816b-1308b738215e-istiod-ca-cert\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.772047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.716614 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-credential-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.772047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.768664 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed723b1e-dfc2-441a-816b-1308b738215e-istio-podinfo\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.798868 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.777152 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-istio-envoy\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.798868 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.779576 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6njgx\" (UniqueName: \"kubernetes.io/projected/ed723b1e-dfc2-441a-816b-1308b738215e-kube-api-access-6njgx\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:53.798868 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:53.780048 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed723b1e-dfc2-441a-816b-1308b738215e-istio-token\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:54.025788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:54.024914 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:54.128742 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:54.128706 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 16:45:54.129109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:54.129081 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" podUID="6eac6c0b-6f5c-4e8c-8034-a045f3530426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 16:45:54.131901 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:54.131874 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 16:45:54.132551 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:54.132525 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" podUID="e38c4872-9809-48e2-a8f5-9a0076deddd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 16:45:54.200100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:54.177232 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" event={"ID":"cb03379b-4139-4f06-a67b-73249017ccb8","Type":"ContainerStarted","Data":"9df63ede14499c496f3683a2c91c76a54a3a618cbdd878a58ae2b1c585ba29c8"} Oct 30 16:45:54.200100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:54.198121 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:54.230225 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:54.227806 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:45:54.606804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:54.603270 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:54.606804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:54.603727 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:54.606804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:54.604107 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:54.829999 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:54.829962 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:54.874924 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:54.874888 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:45:54.907842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:54.907795 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4"] Oct 30 16:45:54.925177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:54.924906 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 16:45:54.935059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:54.935026 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj"] Oct 30 16:45:55.036042 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:45:55.033977 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded723b1e_dfc2_441a_816b_1308b738215e.slice/crio-d68b0ca4ccd766215916bb0afd806643e8d1b8b8f41e3f7a1dc593333abca43e WatchSource:0}: Error finding container d68b0ca4ccd766215916bb0afd806643e8d1b8b8f41e3f7a1dc593333abca43e: Status 404 returned error can't find the container with id d68b0ca4ccd766215916bb0afd806643e8d1b8b8f41e3f7a1dc593333abca43e Oct 30 16:45:55.095065 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:55.069584 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" podUID="cb03379b-4139-4f06-a67b-73249017ccb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 16:45:55.135795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:55.121217 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 16:45:55.135795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:55.121271 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" podUID="6eac6c0b-6f5c-4e8c-8034-a045f3530426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 16:45:55.167982 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:55.166901 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 16:45:55.167982 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:55.166962 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" podUID="e38c4872-9809-48e2-a8f5-9a0076deddd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 16:45:55.368799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:55.359282 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:55.368799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:55.359389 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:55.368799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:55.359433 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:55.464107 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:55.464074 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:55.672808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:55.661456 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" event={"ID":"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd","Type":"ContainerStarted","Data":"2cd62faf605496ce2e18b6972aaa9107f90d716eeadff36f5434fa3abde3a7ef"} Oct 30 16:45:55.672808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:55.664824 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" event={"ID":"ed723b1e-dfc2-441a-816b-1308b738215e","Type":"ContainerStarted","Data":"d68b0ca4ccd766215916bb0afd806643e8d1b8b8f41e3f7a1dc593333abca43e"} Oct 30 16:45:55.864254 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:55.864218 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 16:45:55.864561 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:55.864515 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" podUID="cb03379b-4139-4f06-a67b-73249017ccb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 16:45:56.076984 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:56.066394 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:56.103719 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:56.078541 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 16:45:56.104201 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:56.104164 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" podUID="6eac6c0b-6f5c-4e8c-8034-a045f3530426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 16:45:56.138847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:56.137166 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 16:45:56.138847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:56.137229 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" podUID="e38c4872-9809-48e2-a8f5-9a0076deddd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 16:45:56.166046 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:56.165946 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:56.316871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:56.316783 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:56.706604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:56.706566 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" event={"ID":"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd","Type":"ContainerStarted","Data":"7586261b741c10e7484cf3219c709e5b65080b977caf1ca27e51b0f2cdf1e0c9"} Oct 30 16:45:56.736795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:56.720895 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" event={"ID":"ed723b1e-dfc2-441a-816b-1308b738215e","Type":"ContainerStarted","Data":"d61ca463996b4678c48ac6112d328f85565fc581d2ab72e3a2356a383c2ff35b"} Oct 30 16:45:56.810794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:56.803753 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" podStartSLOduration=5.803729881 podStartE2EDuration="5.803729881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:56.772564451 +0000 UTC m=+1470.483019224" watchObservedRunningTime="2026-10-30 16:45:56.803729881 +0000 UTC m=+1470.514184626" Oct 30 16:45:56.853972 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:56.853932 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 16:45:56.854266 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:56.854233 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" podUID="cb03379b-4139-4f06-a67b-73249017ccb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 16:45:56.859985 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:56.859909 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" podStartSLOduration=3.859887251 podStartE2EDuration="3.859887251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:56.834087351 +0000 UTC m=+1470.544542096" watchObservedRunningTime="2026-10-30 16:45:56.859887251 +0000 UTC m=+1470.570341996" Oct 30 16:45:57.031387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.031293 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:45:57.051756 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.051722 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 16:45:57.052560 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.052531 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" podUID="ed723b1e-dfc2-441a-816b-1308b738215e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 16:45:57.052916 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.052460 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 16:45:57.053060 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.053041 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" podUID="6eac6c0b-6f5c-4e8c-8034-a045f3530426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 16:45:57.091793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.077255 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:57.091793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.078360 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:45:57.126793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.106599 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:57.126793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.109238 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:57.311388 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.233141 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 16:45:57.311388 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.233200 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" podUID="e38c4872-9809-48e2-a8f5-9a0076deddd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 16:45:57.423796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.421314 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded" Oct 30 16:45:57.576873 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.576818 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:57.589828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.586689 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:57.589828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.588359 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:57.608160 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.608118 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:57.669804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.668051 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:57.847867 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.844087 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:57.878793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.861395 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 16:45:57.878793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.861457 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" podUID="cb03379b-4139-4f06-a67b-73249017ccb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 16:45:57.878793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.866101 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:45:57.992911 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:57.992860 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:58.087059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.060904 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 16:45:58.087059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.060975 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" podUID="6eac6c0b-6f5c-4e8c-8034-a045f3530426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 16:45:58.087059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.061435 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 16:45:58.087059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.061476 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" podUID="ed723b1e-dfc2-441a-816b-1308b738215e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 16:45:58.087059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.061564 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 16:45:58.087059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.061618 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:58.144147 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.116088 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:58.144147 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.117889 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:58.172061 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.170164 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 16:45:58.172061 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.170225 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" podUID="e38c4872-9809-48e2-a8f5-9a0076deddd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 16:45:58.290791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.236820 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:58.357790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.335193 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:58.474852 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.442847 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:58.727974 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.589703 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 16:45:58.883905 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.883862 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 16:45:58.885666 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.885636 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 16:45:58.885873 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.885852 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" podUID="cb03379b-4139-4f06-a67b-73249017ccb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 16:45:58.909072 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.909029 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" podUID="d83ab97a-69c3-4758-9b33-da6bb816a575" containerName="main" probeResult="failure" output="Get \"https://10.133.0.150:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:58.989854 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:58.987554 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj"] Oct 30 16:45:59.036495 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.036461 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 16:45:59.036760 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.036735 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" podUID="ed723b1e-dfc2-441a-816b-1308b738215e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 16:45:59.037461 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.037432 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj"] Oct 30 16:45:59.037661 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.037646 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:45:59.037823 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.037812 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:45:59.037955 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.037942 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:45:59.038108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.038094 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 16:45:59.038243 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.038232 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 16:45:59.038334 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.038325 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2"] Oct 30 16:45:59.040557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.040531 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" Oct 30 16:45:59.066847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.066706 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c8db42\"" Oct 30 16:45:59.066847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.066807 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c8db42-dockercfg-dg5tv\"" Oct 30 16:45:59.096835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.096777 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2"] Oct 30 16:45:59.097245 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.097208 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:59.100549 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.100525 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 16:45:59.100914 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.100896 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" Oct 30 16:45:59.107085 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.106993 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:59.110429 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.109674 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:45:59.134183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.134141 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:59.135268 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.135243 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:59.135499 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.135476 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f4c1389-5421-498a-bc8d-05ca2ec455f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj\" (UID: \"3f4c1389-5421-498a-bc8d-05ca2ec455f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" Oct 30 16:45:59.136176 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.136045 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtjn2\" (UniqueName: \"kubernetes.io/projected/3f4c1389-5421-498a-bc8d-05ca2ec455f6-kube-api-access-jtjn2\") pod \"payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj\" (UID: \"3f4c1389-5421-498a-bc8d-05ca2ec455f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" Oct 30 16:45:59.136176 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.136161 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 16:45:59.237391 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.237347 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14671cc-c74d-4fb5-b363-e060629ad7cc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2\" (UID: \"e14671cc-c74d-4fb5-b363-e060629ad7cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" Oct 30 16:45:59.238208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.238175 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtjn2\" (UniqueName: \"kubernetes.io/projected/3f4c1389-5421-498a-bc8d-05ca2ec455f6-kube-api-access-jtjn2\") pod \"payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj\" (UID: \"3f4c1389-5421-498a-bc8d-05ca2ec455f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" Oct 30 16:45:59.247856 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.238535 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s8k4\" (UniqueName: \"kubernetes.io/projected/e14671cc-c74d-4fb5-b363-e060629ad7cc-kube-api-access-6s8k4\") pod \"payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2\" (UID: \"e14671cc-c74d-4fb5-b363-e060629ad7cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" Oct 30 16:45:59.248183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.248154 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f4c1389-5421-498a-bc8d-05ca2ec455f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj\" (UID: \"3f4c1389-5421-498a-bc8d-05ca2ec455f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" Oct 30 16:45:59.272141 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.272097 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f4c1389-5421-498a-bc8d-05ca2ec455f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj\" (UID: \"3f4c1389-5421-498a-bc8d-05ca2ec455f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" Oct 30 16:45:59.305757 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.305717 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtjn2\" (UniqueName: \"kubernetes.io/projected/3f4c1389-5421-498a-bc8d-05ca2ec455f6-kube-api-access-jtjn2\") pod \"payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj\" (UID: \"3f4c1389-5421-498a-bc8d-05ca2ec455f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" Oct 30 16:45:59.316456 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.316396 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:59.411368 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.411321 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s8k4\" (UniqueName: \"kubernetes.io/projected/e14671cc-c74d-4fb5-b363-e060629ad7cc-kube-api-access-6s8k4\") pod \"payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2\" (UID: \"e14671cc-c74d-4fb5-b363-e060629ad7cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" Oct 30 16:45:59.411889 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.411863 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14671cc-c74d-4fb5-b363-e060629ad7cc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2\" (UID: \"e14671cc-c74d-4fb5-b363-e060629ad7cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" Oct 30 16:45:59.413484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.413458 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14671cc-c74d-4fb5-b363-e060629ad7cc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2\" (UID: \"e14671cc-c74d-4fb5-b363-e060629ad7cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" Oct 30 16:45:59.415490 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.415468 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" Oct 30 16:45:59.436390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.436359 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s8k4\" (UniqueName: \"kubernetes.io/projected/e14671cc-c74d-4fb5-b363-e060629ad7cc-kube-api-access-6s8k4\") pod \"payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2\" (UID: \"e14671cc-c74d-4fb5-b363-e060629ad7cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" Oct 30 16:45:59.587353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.585315 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:59.728356 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.728073 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" Oct 30 16:45:59.913233 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.913140 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 16:45:59.913517 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.913488 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" podUID="cb03379b-4139-4f06-a67b-73249017ccb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 16:45:59.932200 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:45:59.932121 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:00.104857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:00.103001 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 16:46:00.104857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:00.103063 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" podUID="ed723b1e-dfc2-441a-816b-1308b738215e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 16:46:00.313633 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:00.313543 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj"] Oct 30 16:46:00.718300 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:00.718275 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2"] Oct 30 16:46:00.867871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:00.864172 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 16:46:00.867871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:00.864236 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" podUID="cb03379b-4139-4f06-a67b-73249017ccb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 16:46:01.025699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.025663 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 16:46:01.025888 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.025722 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" podUID="ed723b1e-dfc2-441a-816b-1308b738215e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 16:46:01.297977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.297884 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" event={"ID":"e14671cc-c74d-4fb5-b363-e060629ad7cc","Type":"ContainerStarted","Data":"37a3befbc8522cf8318ac96e80acd10114a37051ff7d863e98650bbe37db96b6"} Oct 30 16:46:01.297977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.297940 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" event={"ID":"e14671cc-c74d-4fb5-b363-e060629ad7cc","Type":"ContainerStarted","Data":"177c08a83743d6c16a7d1477a149f02ff138e2d72b1f70fbb7840516e2c09164"} Oct 30 16:46:01.297977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.297961 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" Oct 30 16:46:01.299815 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.299786 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" event={"ID":"3f4c1389-5421-498a-bc8d-05ca2ec455f6","Type":"ContainerStarted","Data":"3aff97e1feebf4ff7b82943dbeb7dc020c1b6c5f719028b0669defb4de4d39d5"} Oct 30 16:46:01.299971 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.299832 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" event={"ID":"3f4c1389-5421-498a-bc8d-05ca2ec455f6","Type":"ContainerStarted","Data":"15f9edff52b5317a4d6e5a03a9aaaab513414ac3625264b3c758763cbcea5530"} Oct 30 16:46:01.299971 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.299860 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" Oct 30 16:46:01.361794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.359743 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" podStartSLOduration=3.3597233490000002 podStartE2EDuration="3.359723349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:01.347240805 +0000 UTC m=+1475.057695551" watchObservedRunningTime="2026-10-30 16:46:01.359723349 +0000 UTC m=+1475.070178093" Oct 30 16:46:01.403081 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.402947 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" podStartSLOduration=3.4029267819999998 podStartE2EDuration="3.402926782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:01.387195831 +0000 UTC m=+1475.097650575" watchObservedRunningTime="2026-10-30 16:46:01.402926782 +0000 UTC m=+1475.113381527" Oct 30 16:46:01.434191 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.434144 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:01.597838 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.587383 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:01.621207 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.621138 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg"] Oct 30 16:46:01.627417 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.627387 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" Oct 30 16:46:01.646599 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.646566 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-661509\"" Oct 30 16:46:01.684896 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.679067 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-661509-dockercfg-gghvn\"" Oct 30 16:46:01.704841 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.703740 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg"] Oct 30 16:46:01.731795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.714543 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bec536b-a62b-42d7-810d-fa8090168784-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg\" (UID: \"7bec536b-a62b-42d7-810d-fa8090168784\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" Oct 30 16:46:01.731795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.714638 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbz9w\" (UniqueName: \"kubernetes.io/projected/7bec536b-a62b-42d7-810d-fa8090168784-kube-api-access-pbz9w\") pod \"payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg\" (UID: \"7bec536b-a62b-42d7-810d-fa8090168784\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" Oct 30 16:46:01.773880 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.773837 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml"] Oct 30 16:46:01.855847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.826265 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bec536b-a62b-42d7-810d-fa8090168784-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg\" (UID: \"7bec536b-a62b-42d7-810d-fa8090168784\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" Oct 30 16:46:01.863795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.856355 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbz9w\" (UniqueName: \"kubernetes.io/projected/7bec536b-a62b-42d7-810d-fa8090168784-kube-api-access-pbz9w\") pod \"payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg\" (UID: \"7bec536b-a62b-42d7-810d-fa8090168784\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" Oct 30 16:46:01.879858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.835700 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:02.084077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.844812 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bec536b-a62b-42d7-810d-fa8090168784-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg\" (UID: \"7bec536b-a62b-42d7-810d-fa8090168784\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" Oct 30 16:46:02.114909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.876360 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 16:46:02.115214 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:02.115169 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" podUID="cb03379b-4139-4f06-a67b-73249017ccb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 16:46:02.115709 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:01.897378 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" Oct 30 16:46:02.172803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:02.118086 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 16:46:02.189898 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:02.189846 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" podUID="ed723b1e-dfc2-441a-816b-1308b738215e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 16:46:02.190161 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:02.133509 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q5xc\" (UniqueName: \"kubernetes.io/projected/2dd4faa7-90ff-4815-b7c5-69081a00ed05-kube-api-access-8q5xc\") pod \"payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml\" (UID: \"2dd4faa7-90ff-4815-b7c5-69081a00ed05\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" Oct 30 16:46:02.201106 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:02.201064 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd4faa7-90ff-4815-b7c5-69081a00ed05-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml\" (UID: \"2dd4faa7-90ff-4815-b7c5-69081a00ed05\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" Oct 30 16:46:02.233896 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:02.210892 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml"] Oct 30 16:46:02.284378 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:02.284338 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbz9w\" (UniqueName: \"kubernetes.io/projected/7bec536b-a62b-42d7-810d-fa8090168784-kube-api-access-pbz9w\") pod \"payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg\" (UID: \"7bec536b-a62b-42d7-810d-fa8090168784\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" Oct 30 16:46:02.287373 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:02.287348 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" Oct 30 16:46:02.336672 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:02.336615 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q5xc\" (UniqueName: \"kubernetes.io/projected/2dd4faa7-90ff-4815-b7c5-69081a00ed05-kube-api-access-8q5xc\") pod \"payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml\" (UID: \"2dd4faa7-90ff-4815-b7c5-69081a00ed05\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" Oct 30 16:46:02.336826 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:02.336805 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd4faa7-90ff-4815-b7c5-69081a00ed05-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml\" (UID: \"2dd4faa7-90ff-4815-b7c5-69081a00ed05\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" Oct 30 16:46:02.337723 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:02.337693 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd4faa7-90ff-4815-b7c5-69081a00ed05-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml\" (UID: \"2dd4faa7-90ff-4815-b7c5-69081a00ed05\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" Oct 30 16:46:02.391676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:02.391462 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q5xc\" (UniqueName: \"kubernetes.io/projected/2dd4faa7-90ff-4815-b7c5-69081a00ed05-kube-api-access-8q5xc\") pod \"payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml\" (UID: \"2dd4faa7-90ff-4815-b7c5-69081a00ed05\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" Oct 30 16:46:02.534057 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:02.534019 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" Oct 30 16:46:02.775801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:02.771506 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg"] Oct 30 16:46:02.801287 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:46:02.801255 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bec536b_a62b_42d7_810d_fa8090168784.slice/crio-18ebe20c5f6e289a8392d85c3552ba88ae34de3ec326c6fe4981806449e4f229 WatchSource:0}: Error finding container 18ebe20c5f6e289a8392d85c3552ba88ae34de3ec326c6fe4981806449e4f229: Status 404 returned error can't find the container with id 18ebe20c5f6e289a8392d85c3552ba88ae34de3ec326c6fe4981806449e4f229 Oct 30 16:46:02.884069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:02.883949 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:46:02.885282 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:02.885233 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:46:02.887201 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:02.887164 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:46:02.930660 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:46:02.930627 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd4faa7_90ff_4815_b7c5_69081a00ed05.slice/crio-5828f598c33d34c03a019fb6aa13083052fc985a0f38f7e45522ae8fcf4e1990 WatchSource:0}: Error finding container 5828f598c33d34c03a019fb6aa13083052fc985a0f38f7e45522ae8fcf4e1990: Status 404 returned error can't find the container with id 5828f598c33d34c03a019fb6aa13083052fc985a0f38f7e45522ae8fcf4e1990 Oct 30 16:46:02.931688 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:02.931653 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml"] Oct 30 16:46:03.025996 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:03.025958 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 16:46:03.026185 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:03.026018 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" podUID="ed723b1e-dfc2-441a-816b-1308b738215e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 16:46:03.085996 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:03.085964 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks"] Oct 30 16:46:03.094841 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:03.094807 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0539a2b1-4hvks"] Oct 30 16:46:03.379498 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:03.379461 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" event={"ID":"2dd4faa7-90ff-4815-b7c5-69081a00ed05","Type":"ContainerStarted","Data":"7802d6433d34e774bed4f1d0b9a263d7a00db6863d2314768684bd51f103469a"} Oct 30 16:46:03.379867 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:03.379844 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" Oct 30 16:46:03.380052 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:03.380034 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" event={"ID":"2dd4faa7-90ff-4815-b7c5-69081a00ed05","Type":"ContainerStarted","Data":"5828f598c33d34c03a019fb6aa13083052fc985a0f38f7e45522ae8fcf4e1990"} Oct 30 16:46:03.382078 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:03.382047 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" event={"ID":"7bec536b-a62b-42d7-810d-fa8090168784","Type":"ContainerStarted","Data":"9e9ea2e407ca284194fde9dd557d968c2c2b224c6018ac0ff8e3815fa1b680fd"} Oct 30 16:46:03.382197 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:03.382103 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" event={"ID":"7bec536b-a62b-42d7-810d-fa8090168784","Type":"ContainerStarted","Data":"18ebe20c5f6e289a8392d85c3552ba88ae34de3ec326c6fe4981806449e4f229"} Oct 30 16:46:03.382336 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:03.382311 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" Oct 30 16:46:03.424476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:03.424433 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" podStartSLOduration=2.424419939 podStartE2EDuration="2.424419939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:03.419879347 +0000 UTC m=+1477.130334091" watchObservedRunningTime="2026-10-30 16:46:03.424419939 +0000 UTC m=+1477.134874716" Oct 30 16:46:03.467793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:03.467393 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" podStartSLOduration=2.467374794 podStartE2EDuration="2.467374794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:03.461125989 +0000 UTC m=+1477.171580734" watchObservedRunningTime="2026-10-30 16:46:03.467374794 +0000 UTC m=+1477.177829540" Oct 30 16:46:03.581005 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:03.580965 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:03.709850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:03.705611 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:04.019032 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.018958 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w"] Oct 30 16:46:04.024283 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.024247 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.062366 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.054568 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b315d4-openshift-default-dockercfg-9hvk4\"" Oct 30 16:46:04.064268 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.064241 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w"] Oct 30 16:46:04.122681 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.120935 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-workload-socket\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.131271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.131235 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-workload-certs\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.131594 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.131569 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4d8936-b54c-43f0-acfb-6edc1871e260-istiod-ca-cert\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.131847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.125585 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:04.131945 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.131829 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8qvp\" (UniqueName: \"kubernetes.io/projected/9f4d8936-b54c-43f0-acfb-6edc1871e260-kube-api-access-f8qvp\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.132136 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.132113 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-envoy\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.132314 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.132295 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-data\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.132479 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.132462 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-credential-socket\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.132700 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.132682 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-podinfo\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.132879 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.132862 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-token\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.197555 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.197504 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:46:04.199140 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.199098 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:46:04.199747 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.199728 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 16:46:04.266791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.266461 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:04.284358 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.283913 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-workload-certs\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.284358 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.284002 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4d8936-b54c-43f0-acfb-6edc1871e260-istiod-ca-cert\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.284358 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.284059 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8qvp\" (UniqueName: \"kubernetes.io/projected/9f4d8936-b54c-43f0-acfb-6edc1871e260-kube-api-access-f8qvp\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.284358 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.284209 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-envoy\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.284358 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.284235 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-data\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.284358 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.284271 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-credential-socket\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.284843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.284385 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-podinfo\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.284843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.284453 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-token\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.284843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.284636 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-workload-socket\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.293808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.285159 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-workload-socket\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.293808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.285426 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-workload-certs\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.302651 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.302148 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-data\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.302651 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.302416 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-credential-socket\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.304183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.304131 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4d8936-b54c-43f0-acfb-6edc1871e260-istiod-ca-cert\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.319809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.319715 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-envoy\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.355792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.355161 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-podinfo\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.530519 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.530449 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8qvp\" (UniqueName: \"kubernetes.io/projected/9f4d8936-b54c-43f0-acfb-6edc1871e260-kube-api-access-f8qvp\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.556850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.540417 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-token\") pod \"e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.606566 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.606529 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57258ca6-00aa-4c78-a3d1-10f414b58247" path="/var/lib/kubelet/pods/57258ca6-00aa-4c78-a3d1-10f414b58247/volumes" Oct 30 16:46:04.655402 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.655368 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:04.988864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.985296 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w"] Oct 30 16:46:04.988864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.987166 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:04.988864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.987319 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:04.988864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:04.987409 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:05.383100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:05.383062 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:05.579164 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:05.579131 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" event={"ID":"9f4d8936-b54c-43f0-acfb-6edc1871e260","Type":"ContainerStarted","Data":"02e9dd07c45d190160cf388a4b085dff2107acc4eeded43bc3b55a1a39b826bb"} Oct 30 16:46:05.579628 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:05.579176 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" event={"ID":"9f4d8936-b54c-43f0-acfb-6edc1871e260","Type":"ContainerStarted","Data":"6451d4e409297182e57978ae41b9b6f71495da38c5b4c64adf6bc8cf88898f38"} Oct 30 16:46:05.638310 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:05.638195 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" podStartSLOduration=2.638177072 podStartE2EDuration="2.638177072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:05.636076582 +0000 UTC m=+1479.346531327" watchObservedRunningTime="2026-10-30 16:46:05.638177072 +0000 UTC m=+1479.348631817" Oct 30 16:46:05.656012 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:05.655984 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:06.029582 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:06.029502 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs"] Oct 30 16:46:06.034025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:06.034001 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6456ffb9-7rxgs"] Oct 30 16:46:06.412966 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:06.412933 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a5940bd-e36f-448b-a43d-bb2febc2b8c9" path="/var/lib/kubelet/pods/9a5940bd-e36f-448b-a43d-bb2febc2b8c9/volumes" Oct 30 16:46:06.662180 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:06.662144 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:07.012693 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:07.012659 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:07.143790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:07.139211 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:07.272594 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:07.272510 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:07.598615 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:07.598585 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:07.599497 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:07.599479 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:46:08.970695 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:08.970648 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:10.608585 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:10.608550 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:12.354565 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:12.354535 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" Oct 30 16:46:12.355091 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:12.354613 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" Oct 30 16:46:12.380327 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:12.380290 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:13.748171 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:13.748135 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:14.046896 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:14.046821 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5"] Oct 30 16:46:14.057723 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:14.057693 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec27a38-sgzq5"] Oct 30 16:46:14.082566 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:46:14.082520 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:46:14.364820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:14.364785 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:14.417123 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:14.417095 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a30cd40-6649-46fc-98ab-3b398fb53d38" path="/var/lib/kubelet/pods/2a30cd40-6649-46fc-98ab-3b398fb53d38/volumes" Oct 30 16:46:14.567308 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:14.567280 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" Oct 30 16:46:14.568452 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:14.568428 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" Oct 30 16:46:16.694793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:16.693776 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:16.849793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:16.848801 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:17.054624 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:17.054532 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:17.516801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:17.514959 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-sm8wx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:46:17.516801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:17.515049 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-sm8wx" podUID="f4591587-01a3-4160-a8b7-d25450be9054" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:46:18.449977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:18.449943 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:18.580272 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:18.580242 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:18.767986 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:18.767886 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj"] Oct 30 16:46:19.050194 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:19.050092 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj"] Oct 30 16:46:19.050613 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:19.050592 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj" Oct 30 16:46:19.142869 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:19.142838 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b315d4\"" Oct 30 16:46:19.147145 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:19.147121 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b315d4-dockercfg-9fjqf\"" Oct 30 16:46:19.220138 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:19.220109 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1516689f-6030-4980-9440-ed9d4e41a310-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj\" (UID: \"1516689f-6030-4980-9440-ed9d4e41a310\") " pod="openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj" Oct 30 16:46:19.220321 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:19.220222 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc29h\" (UniqueName: \"kubernetes.io/projected/1516689f-6030-4980-9440-ed9d4e41a310-kube-api-access-jc29h\") pod \"payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj\" (UID: \"1516689f-6030-4980-9440-ed9d4e41a310\") " pod="openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj" Oct 30 16:46:19.322072 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:19.322044 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1516689f-6030-4980-9440-ed9d4e41a310-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj\" (UID: \"1516689f-6030-4980-9440-ed9d4e41a310\") " pod="openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj" Oct 30 16:46:19.322246 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:19.322142 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc29h\" (UniqueName: \"kubernetes.io/projected/1516689f-6030-4980-9440-ed9d4e41a310-kube-api-access-jc29h\") pod \"payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj\" (UID: \"1516689f-6030-4980-9440-ed9d4e41a310\") " pod="openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj" Oct 30 16:46:19.322742 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:19.322719 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1516689f-6030-4980-9440-ed9d4e41a310-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj\" (UID: \"1516689f-6030-4980-9440-ed9d4e41a310\") " pod="openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj" Oct 30 16:46:19.346037 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:19.346010 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc29h\" (UniqueName: \"kubernetes.io/projected/1516689f-6030-4980-9440-ed9d4e41a310-kube-api-access-jc29h\") pod \"payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj\" (UID: \"1516689f-6030-4980-9440-ed9d4e41a310\") " pod="openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj" Oct 30 16:46:19.374916 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:19.374885 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj" Oct 30 16:46:19.681933 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:46:19.681903 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1516689f_6030_4980_9440_ed9d4e41a310.slice/crio-690b432c42d1f05c6cf375f49da516d1fc66577351c057780490ad1ce76148b3 WatchSource:0}: Error finding container 690b432c42d1f05c6cf375f49da516d1fc66577351c057780490ad1ce76148b3: Status 404 returned error can't find the container with id 690b432c42d1f05c6cf375f49da516d1fc66577351c057780490ad1ce76148b3 Oct 30 16:46:19.683369 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:19.683346 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj"] Oct 30 16:46:19.868981 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:19.868922 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:20.072959 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:20.072922 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj" event={"ID":"1516689f-6030-4980-9440-ed9d4e41a310","Type":"ContainerStarted","Data":"fcfe02fbf7e049f56495b5aa2027c7d97e928911c93e905966d0fa81dd306b46"} Oct 30 16:46:20.073170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:20.072985 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj" event={"ID":"1516689f-6030-4980-9440-ed9d4e41a310","Type":"ContainerStarted","Data":"690b432c42d1f05c6cf375f49da516d1fc66577351c057780490ad1ce76148b3"} Oct 30 16:46:20.073170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:20.073045 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj" Oct 30 16:46:20.117977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:20.117917 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj" podStartSLOduration=2.117897592 podStartE2EDuration="2.117897592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:20.115961652 +0000 UTC m=+1493.826416397" watchObservedRunningTime="2026-10-30 16:46:20.117897592 +0000 UTC m=+1493.828352334" Oct 30 16:46:21.563322 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:21.563287 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:23.064303 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:23.064198 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:24.743167 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:24.743131 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:25.520201 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:25.520168 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:25.640607 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:25.640565 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:26.032438 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:26.032400 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9"] Oct 30 16:46:26.046820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:26.046786 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca519378-25kf9"] Oct 30 16:46:26.322357 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:26.322326 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:26.414574 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:26.414533 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeb9db09-b582-470c-bcb2-b9a22839fc71" path="/var/lib/kubelet/pods/eeb9db09-b582-470c-bcb2-b9a22839fc71/volumes" Oct 30 16:46:26.435885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:26.435852 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:26.552837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:26.552799 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:28.246622 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:28.246574 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:29.692095 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:29.692070 6202 kubelet.go:1689] "Image garbage collection succeeded" Oct 30 16:46:29.975144 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:29.975045 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:30.344175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:30.344145 6202 scope.go:122] "RemoveContainer" containerID="14f44ff63dcb1b9bd5eb58c39ce4c21acc495f2313c0ed71f6b0dfac219be44e" Oct 30 16:46:30.359627 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:30.359597 6202 scope.go:122] "RemoveContainer" containerID="d209eda07c2b6e94b871f045dfb95b74ea2406a66dd0639fca5907208dadae0b" Oct 30 16:46:30.372509 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:30.372485 6202 scope.go:122] "RemoveContainer" containerID="fa16b029fb579b7063124cbe73f4b9579e84dd2fda141d5827996a930d857576" Oct 30 16:46:30.383412 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:30.383389 6202 scope.go:122] "RemoveContainer" containerID="20f1aa5a120ccc346bb2270ef339f89dc04868990c7436403e45b20a9345c71d" Oct 30 16:46:30.393406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:30.393385 6202 scope.go:122] "RemoveContainer" containerID="3bf83cc0167fe2038e47815d8a11d10bb49f470ad791811695ec888c7efc596a" Oct 30 16:46:30.403191 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:30.403166 6202 scope.go:122] "RemoveContainer" containerID="5940e3b0a48fba163fa737275fd9d3be5095b3c3e17aa90c6d21c8baa104b2bc" Oct 30 16:46:30.413509 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:30.413491 6202 scope.go:122] "RemoveContainer" containerID="f20f4cd1b0ee8ec9c4a36857a7a0cff8802a2f702f5d6032456111ef299ac981" Oct 30 16:46:30.428511 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:30.428487 6202 scope.go:122] "RemoveContainer" containerID="1aba65d61c5ba2d5420f66d3bea0e3ce9a019f5dd723d24b3ef3c907b9749c4d" Oct 30 16:46:30.439574 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:30.439553 6202 scope.go:122] "RemoveContainer" containerID="fd71d23467470b9dfc6d1c5d4f1af68fff92bc7f35dd1361e9ad5fc620ab5809" Oct 30 16:46:30.450389 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:30.450363 6202 scope.go:122] "RemoveContainer" containerID="74f81fadf13a80d989ee766e8841cf76e7e27a6b740f41af96c635d050255473" Oct 30 16:46:31.089691 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:31.089659 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj" Oct 30 16:46:31.527192 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:31.527109 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:31.733550 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:31.733519 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:33.400183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:33.400145 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:34.706208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:34.706177 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:35.805825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:35.805761 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:37.543872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:37.543822 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:37.673534 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:37.673495 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:37.990843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:37.990803 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk"] Oct 30 16:46:38.003721 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.003690 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.016829 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.016692 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk"] Oct 30 16:46:38.020325 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.020297 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-00fd1a-openshift-default-dockercfg-4vd85\"" Oct 30 16:46:38.065197 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.065162 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:38.220540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.172669 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcr7z\" (UniqueName: \"kubernetes.io/projected/8109a3b3-6567-49bc-b409-58309ac8bb62-kube-api-access-hcr7z\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.220540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.172730 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-envoy\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.220540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.172868 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8109a3b3-6567-49bc-b409-58309ac8bb62-istiod-ca-cert\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.220540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.172913 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-data\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.220540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.173053 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-workload-socket\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.220540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.173180 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-podinfo\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.220540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.173297 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-token\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.220540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.173348 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-workload-certs\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.220540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.173387 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-credential-socket\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.297208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.297112 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-token\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.297483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.297462 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-workload-certs\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.320007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.319965 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-credential-socket\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.320375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.320350 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcr7z\" (UniqueName: \"kubernetes.io/projected/8109a3b3-6567-49bc-b409-58309ac8bb62-kube-api-access-hcr7z\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.320510 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.320496 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-envoy\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.320628 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.320615 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8109a3b3-6567-49bc-b409-58309ac8bb62-istiod-ca-cert\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.321153 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.321128 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-data\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.324697 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.298475 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-workload-certs\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.324893 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.321857 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-credential-socket\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.325452 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.322853 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8109a3b3-6567-49bc-b409-58309ac8bb62-istiod-ca-cert\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.325585 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.320783 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-data\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.340255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.340222 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-workload-socket\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.340609 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.340593 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-podinfo\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.357684 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.341444 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-workload-socket\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.369843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.369805 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcr7z\" (UniqueName: \"kubernetes.io/projected/8109a3b3-6567-49bc-b409-58309ac8bb62-kube-api-access-hcr7z\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.413819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.378327 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-podinfo\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.445536 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.445503 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-envoy\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.465508 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.465441 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-token\") pod \"e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.630679 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.630642 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:38.737502 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.737256 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:38.865127 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.855294 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:38.997110 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:46:38.997078 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8109a3b3_6567_49bc_b409_58309ac8bb62.slice/crio-9bf029aed60a9d7c509ce261d109877771cc2c217a1862893dcf3d5b88e6db1a WatchSource:0}: Error finding container 9bf029aed60a9d7c509ce261d109877771cc2c217a1862893dcf3d5b88e6db1a: Status 404 returned error can't find the container with id 9bf029aed60a9d7c509ce261d109877771cc2c217a1862893dcf3d5b88e6db1a Oct 30 16:46:38.999371 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:38.999343 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk"] Oct 30 16:46:39.002119 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:39.002085 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:39.002273 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:39.002168 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:39.002273 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:39.002213 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:39.089955 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:39.087187 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5"] Oct 30 16:46:39.094255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:39.094227 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c928adc4-wb6p5"] Oct 30 16:46:39.364794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:39.363175 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:39.480794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:39.480380 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" event={"ID":"8109a3b3-6567-49bc-b409-58309ac8bb62","Type":"ContainerStarted","Data":"5029ec767242166c92c33ab92d6d4013c2b092d235cdc5307e775c193f1a8b6f"} Oct 30 16:46:39.480794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:39.480434 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" event={"ID":"8109a3b3-6567-49bc-b409-58309ac8bb62","Type":"ContainerStarted","Data":"9bf029aed60a9d7c509ce261d109877771cc2c217a1862893dcf3d5b88e6db1a"} Oct 30 16:46:39.529729 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:39.529661 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" podStartSLOduration=2.52964537 podStartE2EDuration="2.52964537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:39.528020594 +0000 UTC m=+1513.238475339" watchObservedRunningTime="2026-10-30 16:46:39.52964537 +0000 UTC m=+1513.240100111" Oct 30 16:46:39.631373 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:39.631304 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:40.413046 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:40.413016 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="964580e0-4bc1-4b73-8694-3e2dcb3da8eb" path="/var/lib/kubelet/pods/964580e0-4bc1-4b73-8694-3e2dcb3da8eb/volumes" Oct 30 16:46:40.651793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:40.651111 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:41.497266 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:41.497235 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:41.498306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:41.498284 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:46:41.530155 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:41.530127 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:42.239908 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:42.239853 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:42.364054 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:42.364013 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:43.683404 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:43.683365 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:44.070064 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:46:44.069967 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:46:44.830212 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:44.830179 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:44.935933 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:44.935901 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9"] Oct 30 16:46:44.942145 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:44.942122 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9" Oct 30 16:46:44.960458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:44.960428 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:44.962496 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:44.962471 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9"] Oct 30 16:46:44.968475 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:44.968449 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-00fd1a\"" Oct 30 16:46:44.990962 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:44.990922 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-00fd1a-dockercfg-69rwx\"" Oct 30 16:46:44.999504 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:44.999474 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f816f8bb-62e1-4fa0-abd1-37671f959fa2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9\" (UID: \"f816f8bb-62e1-4fa0-abd1-37671f959fa2\") " pod="openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9" Oct 30 16:46:44.999672 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:44.999527 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb8bp\" (UniqueName: \"kubernetes.io/projected/f816f8bb-62e1-4fa0-abd1-37671f959fa2-kube-api-access-vb8bp\") pod \"payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9\" (UID: \"f816f8bb-62e1-4fa0-abd1-37671f959fa2\") " pod="openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9" Oct 30 16:46:45.118835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:45.118288 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f816f8bb-62e1-4fa0-abd1-37671f959fa2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9\" (UID: \"f816f8bb-62e1-4fa0-abd1-37671f959fa2\") " pod="openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9" Oct 30 16:46:45.118835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:45.118362 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb8bp\" (UniqueName: \"kubernetes.io/projected/f816f8bb-62e1-4fa0-abd1-37671f959fa2-kube-api-access-vb8bp\") pod \"payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9\" (UID: \"f816f8bb-62e1-4fa0-abd1-37671f959fa2\") " pod="openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9" Oct 30 16:46:45.126792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:45.120218 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f816f8bb-62e1-4fa0-abd1-37671f959fa2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9\" (UID: \"f816f8bb-62e1-4fa0-abd1-37671f959fa2\") " pod="openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9" Oct 30 16:46:45.327275 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:45.327218 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb8bp\" (UniqueName: \"kubernetes.io/projected/f816f8bb-62e1-4fa0-abd1-37671f959fa2-kube-api-access-vb8bp\") pod \"payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9\" (UID: \"f816f8bb-62e1-4fa0-abd1-37671f959fa2\") " pod="openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9" Oct 30 16:46:45.560245 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:45.560159 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9" Oct 30 16:46:45.876239 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:45.876204 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9"] Oct 30 16:46:45.881455 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:46:45.881422 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf816f8bb_62e1_4fa0_abd1_37671f959fa2.slice/crio-b5bdf45c45a1f0f18ae1cb08cb79efbe6d2a39bafb61601ea339969a02b75bc8 WatchSource:0}: Error finding container b5bdf45c45a1f0f18ae1cb08cb79efbe6d2a39bafb61601ea339969a02b75bc8: Status 404 returned error can't find the container with id b5bdf45c45a1f0f18ae1cb08cb79efbe6d2a39bafb61601ea339969a02b75bc8 Oct 30 16:46:45.978494 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:45.978462 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:46.565745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:46.565651 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9" event={"ID":"f816f8bb-62e1-4fa0-abd1-37671f959fa2","Type":"ContainerStarted","Data":"41e25693bf7f48434ffaf03c1950f11a1b1e83bcf5b50bd710af411f88dccc9e"} Oct 30 16:46:46.565745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:46.565718 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9" event={"ID":"f816f8bb-62e1-4fa0-abd1-37671f959fa2","Type":"ContainerStarted","Data":"b5bdf45c45a1f0f18ae1cb08cb79efbe6d2a39bafb61601ea339969a02b75bc8"} Oct 30 16:46:46.566048 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:46.565791 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9" Oct 30 16:46:46.567718 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:46.567687 6202 generic.go:356] "Generic (PLEG): container finished" podID="d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" containerID="d52da4f3cfbaf5adab63dc37792286a479d63af462dd30f0a734ca29f4e0c7cc" exitCode=0 Oct 30 16:46:46.567873 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:46.567756 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" event={"ID":"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48","Type":"ContainerDied","Data":"d52da4f3cfbaf5adab63dc37792286a479d63af462dd30f0a734ca29f4e0c7cc"} Oct 30 16:46:46.568215 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:46.568201 6202 scope.go:122] "RemoveContainer" containerID="d52da4f3cfbaf5adab63dc37792286a479d63af462dd30f0a734ca29f4e0c7cc" Oct 30 16:46:46.570389 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:46.570352 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:46.570479 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:46.570443 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:46.570516 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:46.570492 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:46.609857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:46.609811 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9" podStartSLOduration=2.609797284 podStartE2EDuration="2.609797284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:46.606293243 +0000 UTC m=+1520.316747984" watchObservedRunningTime="2026-10-30 16:46:46.609797284 +0000 UTC m=+1520.320252027" Oct 30 16:46:47.569211 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:47.569163 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:47.587321 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:47.587281 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" event={"ID":"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48","Type":"ContainerStarted","Data":"b38baebc14951408e8c0c3a27e9c584d29f4b1ec5695936fa47a2de688cdffc9"} Oct 30 16:46:47.686525 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:47.686447 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:46:47.688350 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:47.688321 6202 patch_prober.go:29] interesting pod/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 16:46:47.688569 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:47.688543 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" podUID="d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 16:46:48.684710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:48.684672 6202 patch_prober.go:29] interesting pod/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 16:46:48.685208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:48.684729 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" podUID="d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 16:46:49.097073 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:49.097013 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:49.257457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:49.257403 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:49.387907 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:49.387813 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:49.684736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:49.684651 6202 patch_prober.go:29] interesting pod/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 16:46:49.684736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:49.684719 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" podUID="d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 16:46:49.764792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:49.760252 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:49.898642 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:49.898606 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:50.213529 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.213498 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj"] Oct 30 16:46:50.219323 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.219293 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.264207 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.264175 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e5674c-openshift-default-dockercfg-4jl2x\"" Oct 30 16:46:50.284193 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.284111 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-workload-socket\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.284193 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.284189 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istiod-ca-cert\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.284470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.284226 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-workload-certs\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.284470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.284318 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-credential-socket\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.284470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.284414 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-podinfo\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.284657 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.284472 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj2pd\" (UniqueName: \"kubernetes.io/projected/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-kube-api-access-tj2pd\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.284657 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.284551 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-token\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.284837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.284667 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-data\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.284837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.284704 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-envoy\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.290383 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.290343 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj"] Oct 30 16:46:50.295439 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.295407 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:50.457784 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.394072 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-podinfo\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.457784 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.394156 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj2pd\" (UniqueName: \"kubernetes.io/projected/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-kube-api-access-tj2pd\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.457784 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.394213 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-token\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.457784 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.394325 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-data\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.457784 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.394358 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-envoy\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.457784 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.394537 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-workload-socket\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.457784 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.394591 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istiod-ca-cert\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.457784 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.394625 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-workload-certs\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.457784 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.394738 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-credential-socket\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.457784 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.395391 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-credential-socket\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.457784 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.401705 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-workload-socket\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.457784 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.406094 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-data\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.524839 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.521244 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-workload-certs\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.539794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.534889 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:50.605926 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.592982 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istiod-ca-cert\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.640790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.638698 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-podinfo\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.640790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.639010 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-envoy\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.650049 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.649963 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj2pd\" (UniqueName: \"kubernetes.io/projected/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-kube-api-access-tj2pd\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.689791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.679985 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-token\") pod \"e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.714797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.702938 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:50.714797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.703872 6202 patch_prober.go:29] interesting pod/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 16:46:50.714797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.703932 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" podUID="d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 16:46:50.865799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.863035 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:50.911914 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.911886 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc"] Oct 30 16:46:50.912422 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.912088 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" podUID="c3f813c4-823b-4109-a2e4-6027d5fabf2e" containerName="istio-proxy" containerID="cri-o://7a41858953788d1abeb4a2f05bd7d09017e4d57bca1a7bdc67ddf57e85f4fd5c" gracePeriod=30 Oct 30 16:46:50.917441 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.917135 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q"] Oct 30 16:46:50.917441 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.917303 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" podUID="ded079d5-65af-48a8-82d2-922eb053d8c2" containerName="istio-proxy" containerID="cri-o://18f855d140aebe67422e376d53027b5c80553d1f19eeb00b4e4d2372ff7e78bf" gracePeriod=30 Oct 30 16:46:50.923237 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.922707 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9"] Oct 30 16:46:50.923237 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.922952 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" podUID="119c2424-9656-41b0-9202-477228e632c2" containerName="istio-proxy" containerID="cri-o://d469be6f7acbb8a9449e1ca032cade02dc23e7c0279d005974cd930e5645dc40" gracePeriod=30 Oct 30 16:46:50.927416 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.927393 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv"] Oct 30 16:46:50.927588 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:50.927554 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" podUID="71652f10-3fa9-4e81-8394-f631af440cae" containerName="istio-proxy" containerID="cri-o://09db65d1e0f1ae274e6bc0de74b7edaa1eb747bd326164da56372ee7ac4b7f72" gracePeriod=30 Oct 30 16:46:51.273843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:51.273209 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:51.279442 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:51.279415 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj"] Oct 30 16:46:51.282004 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:46:51.281912 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbbb77be_53e1_4ede_bfe4_59f7f2cf5d0b.slice/crio-7b36c227cd7d250d4ef967ea1496cbdf3762e0a1c19271aa94af63d83869ebe0 WatchSource:0}: Error finding container 7b36c227cd7d250d4ef967ea1496cbdf3762e0a1c19271aa94af63d83869ebe0: Status 404 returned error can't find the container with id 7b36c227cd7d250d4ef967ea1496cbdf3762e0a1c19271aa94af63d83869ebe0 Oct 30 16:46:51.285995 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:51.285951 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:51.286099 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:51.286043 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:51.286099 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:51.286085 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:51.444792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:51.427709 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:51.687628 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:51.687169 6202 patch_prober.go:29] interesting pod/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 16:46:51.687628 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:51.687243 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" podUID="d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 16:46:51.891871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:51.890440 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" event={"ID":"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b","Type":"ContainerStarted","Data":"99c07b28fa71a9da8c59c9aa59e2b90745a025152b2d1d22e4ebce6fbcc070f7"} Oct 30 16:46:51.891871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:51.890505 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" event={"ID":"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b","Type":"ContainerStarted","Data":"7b36c227cd7d250d4ef967ea1496cbdf3762e0a1c19271aa94af63d83869ebe0"} Oct 30 16:46:51.958844 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:51.944548 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" podStartSLOduration=1.944528355 podStartE2EDuration="1.944528355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:51.942266467 +0000 UTC m=+1525.652721212" watchObservedRunningTime="2026-10-30 16:46:51.944528355 +0000 UTC m=+1525.654983101" Oct 30 16:46:52.708793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:52.706228 6202 patch_prober.go:29] interesting pod/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 16:46:52.708793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:52.706292 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" podUID="d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 16:46:52.867792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:52.867198 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:52.880787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:52.880655 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:52.902728 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:52.902699 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:52.903930 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:52.903907 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:46:53.180776 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:53.180734 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:53.690279 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:53.690245 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:46:53.690667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:53.690630 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:46:53.691585 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:53.691565 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:46:56.302146 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.302111 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:46:56.324924 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.324893 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:46:56.330714 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.330675 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:56.343755 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.343693 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx"] Oct 30 16:46:56.354690 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.354653 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c3f813c4-823b-4109-a2e4-6027d5fabf2e" containerName="istio-proxy" Oct 30 16:46:56.354690 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.354689 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3f813c4-823b-4109-a2e4-6027d5fabf2e" containerName="istio-proxy" Oct 30 16:46:56.355045 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.354711 6202 cpu_manager.go:414] "containerMap: removing container" podUID="119c2424-9656-41b0-9202-477228e632c2" containerName="istio-proxy" Oct 30 16:46:56.355045 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.354719 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119c2424-9656-41b0-9202-477228e632c2" containerName="istio-proxy" Oct 30 16:46:56.355045 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.354985 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3f813c4-823b-4109-a2e4-6027d5fabf2e" containerName="istio-proxy" Oct 30 16:46:56.355045 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.355002 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="119c2424-9656-41b0-9202-477228e632c2" containerName="istio-proxy" Oct 30 16:46:56.362985 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.362959 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx" Oct 30 16:46:56.390226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.387246 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/119c2424-9656-41b0-9202-477228e632c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119c2424-9656-41b0-9202-477228e632c2-istiod-ca-cert\") pod \"119c2424-9656-41b0-9202-477228e632c2\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " Oct 30 16:46:56.390226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.387318 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-istio-envoy\") pod \"119c2424-9656-41b0-9202-477228e632c2\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " Oct 30 16:46:56.390226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.387433 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-istio-data\") pod \"119c2424-9656-41b0-9202-477228e632c2\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " Oct 30 16:46:56.390226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.387469 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-workload-certs\") pod \"119c2424-9656-41b0-9202-477228e632c2\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " Oct 30 16:46:56.390226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.387509 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119c2424-9656-41b0-9202-477228e632c2-kube-api-access-jq52x\" (UniqueName: \"kubernetes.io/projected/119c2424-9656-41b0-9202-477228e632c2-kube-api-access-jq52x\") pod \"119c2424-9656-41b0-9202-477228e632c2\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " Oct 30 16:46:56.390226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.387581 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-credential-socket\") pod \"119c2424-9656-41b0-9202-477228e632c2\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " Oct 30 16:46:56.390226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.387710 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-workload-socket\") pod \"119c2424-9656-41b0-9202-477228e632c2\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " Oct 30 16:46:56.390226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.387857 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/119c2424-9656-41b0-9202-477228e632c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119c2424-9656-41b0-9202-477228e632c2-istio-podinfo\") pod \"119c2424-9656-41b0-9202-477228e632c2\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " Oct 30 16:46:56.390226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.388026 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119c2424-9656-41b0-9202-477228e632c2-istio-token\" (UniqueName: \"kubernetes.io/projected/119c2424-9656-41b0-9202-477228e632c2-istio-token\") pod \"119c2424-9656-41b0-9202-477228e632c2\" (UID: \"119c2424-9656-41b0-9202-477228e632c2\") " Oct 30 16:46:56.391613 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.391378 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-workload-certs" pod "119c2424-9656-41b0-9202-477228e632c2" (UID: "119c2424-9656-41b0-9202-477228e632c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.392209 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.391973 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.392789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.392329 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-istio-data" pod "119c2424-9656-41b0-9202-477228e632c2" (UID: "119c2424-9656-41b0-9202-477228e632c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.393852 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.393628 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/119c2424-9656-41b0-9202-477228e632c2-istiod-ca-cert" pod "119c2424-9656-41b0-9202-477228e632c2" (UID: "119c2424-9656-41b0-9202-477228e632c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:56.394202 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.393997 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-workload-socket" pod "119c2424-9656-41b0-9202-477228e632c2" (UID: "119c2424-9656-41b0-9202-477228e632c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.394582 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.394332 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-credential-socket" pod "119c2424-9656-41b0-9202-477228e632c2" (UID: "119c2424-9656-41b0-9202-477228e632c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.398201 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.397904 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx"] Oct 30 16:46:56.398201 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.398001 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119c2424-9656-41b0-9202-477228e632c2-istio-token" pod "119c2424-9656-41b0-9202-477228e632c2" (UID: "119c2424-9656-41b0-9202-477228e632c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:56.403504 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.403289 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119c2424-9656-41b0-9202-477228e632c2-kube-api-access-jq52x" pod "119c2424-9656-41b0-9202-477228e632c2" (UID: "119c2424-9656-41b0-9202-477228e632c2"). InnerVolumeSpecName "kube-api-access-jq52x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:56.403504 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.403474 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/119c2424-9656-41b0-9202-477228e632c2-istio-podinfo" pod "119c2424-9656-41b0-9202-477228e632c2" (UID: "119c2424-9656-41b0-9202-477228e632c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:46:56.407077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.405710 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e5674c-dockercfg-k5rmd\"" Oct 30 16:46:56.407077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.406030 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e5674c\"" Oct 30 16:46:56.408621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.407702 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-istio-envoy" pod "119c2424-9656-41b0-9202-477228e632c2" (UID: "119c2424-9656-41b0-9202-477228e632c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.432096 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.432029 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf"] Oct 30 16:46:56.433538 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.433480 6202 cpu_manager.go:414] "containerMap: removing container" podUID="119c2424-9656-41b0-9202-477228e632c2" containerName="istio-proxy" Oct 30 16:46:56.433538 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.433503 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119c2424-9656-41b0-9202-477228e632c2" containerName="istio-proxy" Oct 30 16:46:56.433808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.433747 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="119c2424-9656-41b0-9202-477228e632c2" containerName="istio-proxy" Oct 30 16:46:56.436099 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.436046 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:46:56.437785 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.437739 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf"] Oct 30 16:46:56.437897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.437875 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf" Oct 30 16:46:56.459369 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.459339 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:56.460992 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.460966 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:46:56.494857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.493399 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-workload-certs\") pod \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " Oct 30 16:46:56.494857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.493497 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-envoy\") pod \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " Oct 30 16:46:56.494857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.493590 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-credential-socket\") pod \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " Oct 30 16:46:56.494857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.493625 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-data\") pod \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " Oct 30 16:46:56.494857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.493668 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-podinfo\") pod \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " Oct 30 16:46:56.494857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.493750 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-workload-socket\") pod \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " Oct 30 16:46:56.494857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.493860 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istiod-ca-cert\") pod \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " Oct 30 16:46:56.494857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.494027 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-token\" (UniqueName: \"kubernetes.io/projected/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-token\") pod \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " Oct 30 16:46:56.494857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.494081 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3f813c4-823b-4109-a2e4-6027d5fabf2e-kube-api-access-kdbzn\" (UniqueName: \"kubernetes.io/projected/c3f813c4-823b-4109-a2e4-6027d5fabf2e-kube-api-access-kdbzn\") pod \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\" (UID: \"c3f813c4-823b-4109-a2e4-6027d5fabf2e\") " Oct 30 16:46:56.494857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.494453 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4830c6cf-3abd-485a-87ee-6b094461b5ae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx\" (UID: \"4830c6cf-3abd-485a-87ee-6b094461b5ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx" Oct 30 16:46:56.494857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.494667 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxrsp\" (UniqueName: \"kubernetes.io/projected/4830c6cf-3abd-485a-87ee-6b094461b5ae-kube-api-access-jxrsp\") pod \"payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx\" (UID: \"4830c6cf-3abd-485a-87ee-6b094461b5ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx" Oct 30 16:46:56.496050 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.496016 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119c2424-9656-41b0-9202-477228e632c2-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.496259 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.496199 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119c2424-9656-41b0-9202-477228e632c2-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.496259 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.496216 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119c2424-9656-41b0-9202-477228e632c2-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.496259 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.496228 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.496259 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.496240 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.496600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.496537 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq52x\" (UniqueName: \"kubernetes.io/projected/119c2424-9656-41b0-9202-477228e632c2-kube-api-access-jq52x\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.496600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.496562 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.496600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.496576 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119c2424-9656-41b0-9202-477228e632c2-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.498865 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.498756 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-workload-certs" pod "c3f813c4-823b-4109-a2e4-6027d5fabf2e" (UID: "c3f813c4-823b-4109-a2e4-6027d5fabf2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.499364 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.499343 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-workload-socket" pod "c3f813c4-823b-4109-a2e4-6027d5fabf2e" (UID: "c3f813c4-823b-4109-a2e4-6027d5fabf2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.501371 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.501350 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-credential-socket" pod "c3f813c4-823b-4109-a2e4-6027d5fabf2e" (UID: "c3f813c4-823b-4109-a2e4-6027d5fabf2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.502111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.502063 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-data" pod "c3f813c4-823b-4109-a2e4-6027d5fabf2e" (UID: "c3f813c4-823b-4109-a2e4-6027d5fabf2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.502519 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.502123 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istiod-ca-cert" pod "c3f813c4-823b-4109-a2e4-6027d5fabf2e" (UID: "c3f813c4-823b-4109-a2e4-6027d5fabf2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:56.507726 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.507689 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-envoy" pod "c3f813c4-823b-4109-a2e4-6027d5fabf2e" (UID: "c3f813c4-823b-4109-a2e4-6027d5fabf2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.509676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.509638 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-token" pod "c3f813c4-823b-4109-a2e4-6027d5fabf2e" (UID: "c3f813c4-823b-4109-a2e4-6027d5fabf2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:56.510747 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.510722 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-podinfo" pod "c3f813c4-823b-4109-a2e4-6027d5fabf2e" (UID: "c3f813c4-823b-4109-a2e4-6027d5fabf2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:46:56.512043 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.512013 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f813c4-823b-4109-a2e4-6027d5fabf2e-kube-api-access-kdbzn" pod "c3f813c4-823b-4109-a2e4-6027d5fabf2e" (UID: "c3f813c4-823b-4109-a2e4-6027d5fabf2e"). InnerVolumeSpecName "kube-api-access-kdbzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:56.597607 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.597571 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-envoy\") pod \"ded079d5-65af-48a8-82d2-922eb053d8c2\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " Oct 30 16:46:56.597852 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.597628 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-istio-envoy\") pod \"71652f10-3fa9-4e81-8394-f631af440cae\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " Oct 30 16:46:56.597852 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.597706 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-workload-socket\") pod \"71652f10-3fa9-4e81-8394-f631af440cae\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " Oct 30 16:46:56.597852 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.597828 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-token\" (UniqueName: \"kubernetes.io/projected/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-token\") pod \"ded079d5-65af-48a8-82d2-922eb053d8c2\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " Oct 30 16:46:56.598051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.597873 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-data\") pod \"ded079d5-65af-48a8-82d2-922eb053d8c2\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " Oct 30 16:46:56.598051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.597924 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71652f10-3fa9-4e81-8394-f631af440cae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71652f10-3fa9-4e81-8394-f631af440cae-istiod-ca-cert\") pod \"71652f10-3fa9-4e81-8394-f631af440cae\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " Oct 30 16:46:56.598051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.597966 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71652f10-3fa9-4e81-8394-f631af440cae-istio-token\" (UniqueName: \"kubernetes.io/projected/71652f10-3fa9-4e81-8394-f631af440cae-istio-token\") pod \"71652f10-3fa9-4e81-8394-f631af440cae\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " Oct 30 16:46:56.598051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.598022 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-workload-socket" pod "71652f10-3fa9-4e81-8394-f631af440cae" (UID: "71652f10-3fa9-4e81-8394-f631af440cae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.598256 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.598037 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-workload-socket\") pod \"ded079d5-65af-48a8-82d2-922eb053d8c2\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " Oct 30 16:46:56.598256 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.598106 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-podinfo\") pod \"ded079d5-65af-48a8-82d2-922eb053d8c2\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " Oct 30 16:46:56.598256 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.598150 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71652f10-3fa9-4e81-8394-f631af440cae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71652f10-3fa9-4e81-8394-f631af440cae-istio-podinfo\") pod \"71652f10-3fa9-4e81-8394-f631af440cae\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " Oct 30 16:46:56.598256 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.598187 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-credential-socket\") pod \"ded079d5-65af-48a8-82d2-922eb053d8c2\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " Oct 30 16:46:56.598472 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.598286 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-workload-socket" pod "ded079d5-65af-48a8-82d2-922eb053d8c2" (UID: "ded079d5-65af-48a8-82d2-922eb053d8c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.598533 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.598491 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-data" pod "ded079d5-65af-48a8-82d2-922eb053d8c2" (UID: "ded079d5-65af-48a8-82d2-922eb053d8c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.598533 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.598496 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-credential-socket\") pod \"71652f10-3fa9-4e81-8394-f631af440cae\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " Oct 30 16:46:56.598649 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.598600 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ded079d5-65af-48a8-82d2-922eb053d8c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded079d5-65af-48a8-82d2-922eb053d8c2-istiod-ca-cert\") pod \"ded079d5-65af-48a8-82d2-922eb053d8c2\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " Oct 30 16:46:56.598740 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.598696 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-workload-certs\") pod \"ded079d5-65af-48a8-82d2-922eb053d8c2\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " Oct 30 16:46:56.598960 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.598945 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-workload-certs\") pod \"71652f10-3fa9-4e81-8394-f631af440cae\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " Oct 30 16:46:56.599054 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.598782 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-credential-socket" pod "71652f10-3fa9-4e81-8394-f631af440cae" (UID: "71652f10-3fa9-4e81-8394-f631af440cae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.599223 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.599019 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-credential-socket" pod "ded079d5-65af-48a8-82d2-922eb053d8c2" (UID: "ded079d5-65af-48a8-82d2-922eb053d8c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.599325 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.599177 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-workload-certs" pod "ded079d5-65af-48a8-82d2-922eb053d8c2" (UID: "ded079d5-65af-48a8-82d2-922eb053d8c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.599414 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.599115 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded079d5-65af-48a8-82d2-922eb053d8c2-kube-api-access-h4gr2\" (UniqueName: \"kubernetes.io/projected/ded079d5-65af-48a8-82d2-922eb053d8c2-kube-api-access-h4gr2\") pod \"ded079d5-65af-48a8-82d2-922eb053d8c2\" (UID: \"ded079d5-65af-48a8-82d2-922eb053d8c2\") " Oct 30 16:46:56.599550 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.599533 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-istio-data\") pod \"71652f10-3fa9-4e81-8394-f631af440cae\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " Oct 30 16:46:56.600226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.599702 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71652f10-3fa9-4e81-8394-f631af440cae-kube-api-access-qnkf7\" (UniqueName: \"kubernetes.io/projected/71652f10-3fa9-4e81-8394-f631af440cae-kube-api-access-qnkf7\") pod \"71652f10-3fa9-4e81-8394-f631af440cae\" (UID: \"71652f10-3fa9-4e81-8394-f631af440cae\") " Oct 30 16:46:56.600874 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.600850 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4830c6cf-3abd-485a-87ee-6b094461b5ae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx\" (UID: \"4830c6cf-3abd-485a-87ee-6b094461b5ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx" Oct 30 16:46:56.601145 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.601093 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq9c9\" (UniqueName: \"kubernetes.io/projected/b37d7321-2cef-42aa-80ff-515da8f8a2df-kube-api-access-wq9c9\") pod \"payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf\" (UID: \"b37d7321-2cef-42aa-80ff-515da8f8a2df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf" Oct 30 16:46:56.601412 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.601390 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxrsp\" (UniqueName: \"kubernetes.io/projected/4830c6cf-3abd-485a-87ee-6b094461b5ae-kube-api-access-jxrsp\") pod \"payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx\" (UID: \"4830c6cf-3abd-485a-87ee-6b094461b5ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx" Oct 30 16:46:56.601618 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.601599 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37d7321-2cef-42aa-80ff-515da8f8a2df-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf\" (UID: \"b37d7321-2cef-42aa-80ff-515da8f8a2df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf" Oct 30 16:46:56.602011 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.601992 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.602127 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.602113 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.602237 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.602225 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.602340 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.602327 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.602450 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.602437 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.602566 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.602551 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.602673 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.602660 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.602789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.602759 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.602900 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.602888 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.602993 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.602978 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.603157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.603143 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f813c4-823b-4109-a2e4-6027d5fabf2e-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.603278 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.603264 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.603386 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.603374 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.603509 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.603497 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3f813c4-823b-4109-a2e4-6027d5fabf2e-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.603626 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.603613 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdbzn\" (UniqueName: \"kubernetes.io/projected/c3f813c4-823b-4109-a2e4-6027d5fabf2e-kube-api-access-kdbzn\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.603718 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.602461 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4830c6cf-3abd-485a-87ee-6b094461b5ae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx\" (UID: \"4830c6cf-3abd-485a-87ee-6b094461b5ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx" Oct 30 16:46:56.603850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.599453 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded079d5-65af-48a8-82d2-922eb053d8c2-istiod-ca-cert" pod "ded079d5-65af-48a8-82d2-922eb053d8c2" (UID: "ded079d5-65af-48a8-82d2-922eb053d8c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:56.603957 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.599826 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71652f10-3fa9-4e81-8394-f631af440cae-istiod-ca-cert" pod "71652f10-3fa9-4e81-8394-f631af440cae" (UID: "71652f10-3fa9-4e81-8394-f631af440cae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:56.604062 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.601133 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-istio-data" pod "71652f10-3fa9-4e81-8394-f631af440cae" (UID: "71652f10-3fa9-4e81-8394-f631af440cae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.604389 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.601999 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-workload-certs" pod "71652f10-3fa9-4e81-8394-f631af440cae" (UID: "71652f10-3fa9-4e81-8394-f631af440cae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.604549 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.602671 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71652f10-3fa9-4e81-8394-f631af440cae-istio-podinfo" pod "71652f10-3fa9-4e81-8394-f631af440cae" (UID: "71652f10-3fa9-4e81-8394-f631af440cae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:46:56.604659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.602965 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded079d5-65af-48a8-82d2-922eb053d8c2-kube-api-access-h4gr2" pod "ded079d5-65af-48a8-82d2-922eb053d8c2" (UID: "ded079d5-65af-48a8-82d2-922eb053d8c2"). InnerVolumeSpecName "kube-api-access-h4gr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:56.604777 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.603871 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71652f10-3fa9-4e81-8394-f631af440cae-kube-api-access-qnkf7" pod "71652f10-3fa9-4e81-8394-f631af440cae" (UID: "71652f10-3fa9-4e81-8394-f631af440cae"). InnerVolumeSpecName "kube-api-access-qnkf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:56.604883 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.604587 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71652f10-3fa9-4e81-8394-f631af440cae-istio-token" pod "71652f10-3fa9-4e81-8394-f631af440cae" (UID: "71652f10-3fa9-4e81-8394-f631af440cae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:56.605391 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.605358 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-envoy" pod "ded079d5-65af-48a8-82d2-922eb053d8c2" (UID: "ded079d5-65af-48a8-82d2-922eb053d8c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.606143 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.606116 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-istio-envoy" pod "71652f10-3fa9-4e81-8394-f631af440cae" (UID: "71652f10-3fa9-4e81-8394-f631af440cae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:56.607373 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.607343 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-podinfo" pod "ded079d5-65af-48a8-82d2-922eb053d8c2" (UID: "ded079d5-65af-48a8-82d2-922eb053d8c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:46:56.607550 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.607526 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-token" pod "ded079d5-65af-48a8-82d2-922eb053d8c2" (UID: "ded079d5-65af-48a8-82d2-922eb053d8c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:56.629834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.626280 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxrsp\" (UniqueName: \"kubernetes.io/projected/4830c6cf-3abd-485a-87ee-6b094461b5ae-kube-api-access-jxrsp\") pod \"payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx\" (UID: \"4830c6cf-3abd-485a-87ee-6b094461b5ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx" Oct 30 16:46:56.704903 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.704812 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq9c9\" (UniqueName: \"kubernetes.io/projected/b37d7321-2cef-42aa-80ff-515da8f8a2df-kube-api-access-wq9c9\") pod \"payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf\" (UID: \"b37d7321-2cef-42aa-80ff-515da8f8a2df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf" Oct 30 16:46:56.705261 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.705237 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37d7321-2cef-42aa-80ff-515da8f8a2df-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf\" (UID: \"b37d7321-2cef-42aa-80ff-515da8f8a2df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf" Oct 30 16:46:56.705444 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.705429 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded079d5-65af-48a8-82d2-922eb053d8c2-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.705525 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.705450 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.705525 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.705464 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4gr2\" (UniqueName: \"kubernetes.io/projected/ded079d5-65af-48a8-82d2-922eb053d8c2-kube-api-access-h4gr2\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.705525 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.705479 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.705525 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.705493 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnkf7\" (UniqueName: \"kubernetes.io/projected/71652f10-3fa9-4e81-8394-f631af440cae-kube-api-access-qnkf7\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.705525 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.705507 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.705525 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.705521 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71652f10-3fa9-4e81-8394-f631af440cae-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.705746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.705534 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.705746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.705547 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71652f10-3fa9-4e81-8394-f631af440cae-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.705746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.705559 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71652f10-3fa9-4e81-8394-f631af440cae-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.705746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.705571 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded079d5-65af-48a8-82d2-922eb053d8c2-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.705746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.705584 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71652f10-3fa9-4e81-8394-f631af440cae-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.706257 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.706229 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37d7321-2cef-42aa-80ff-515da8f8a2df-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf\" (UID: \"b37d7321-2cef-42aa-80ff-515da8f8a2df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf" Oct 30 16:46:56.729516 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.729484 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq9c9\" (UniqueName: \"kubernetes.io/projected/b37d7321-2cef-42aa-80ff-515da8f8a2df-kube-api-access-wq9c9\") pod \"payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf\" (UID: \"b37d7321-2cef-42aa-80ff-515da8f8a2df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf" Oct 30 16:46:56.730339 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.730305 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx" Oct 30 16:46:56.776647 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:56.776303 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf" Oct 30 16:46:57.268592 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.265199 6202 generic.go:356] "Generic (PLEG): container finished" podID="c3f813c4-823b-4109-a2e4-6027d5fabf2e" containerID="7a41858953788d1abeb4a2f05bd7d09017e4d57bca1a7bdc67ddf57e85f4fd5c" exitCode=0 Oct 30 16:46:57.268592 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.265357 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" event={"ID":"c3f813c4-823b-4109-a2e4-6027d5fabf2e","Type":"ContainerDied","Data":"7a41858953788d1abeb4a2f05bd7d09017e4d57bca1a7bdc67ddf57e85f4fd5c"} Oct 30 16:46:57.268592 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.265412 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" event={"ID":"c3f813c4-823b-4109-a2e4-6027d5fabf2e","Type":"ContainerDied","Data":"c588e7eee015f8505ffbf53bdf1910947a5e92f08925efe7a4230607bfc8606e"} Oct 30 16:46:57.268592 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.265444 6202 scope.go:122] "RemoveContainer" containerID="7a41858953788d1abeb4a2f05bd7d09017e4d57bca1a7bdc67ddf57e85f4fd5c" Oct 30 16:46:57.268592 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.265611 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc" Oct 30 16:46:57.283578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.280866 6202 generic.go:356] "Generic (PLEG): container finished" podID="ded079d5-65af-48a8-82d2-922eb053d8c2" containerID="18f855d140aebe67422e376d53027b5c80553d1f19eeb00b4e4d2372ff7e78bf" exitCode=0 Oct 30 16:46:57.283578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.280987 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" event={"ID":"ded079d5-65af-48a8-82d2-922eb053d8c2","Type":"ContainerDied","Data":"18f855d140aebe67422e376d53027b5c80553d1f19eeb00b4e4d2372ff7e78bf"} Oct 30 16:46:57.283578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.281037 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" event={"ID":"ded079d5-65af-48a8-82d2-922eb053d8c2","Type":"ContainerDied","Data":"04484285fc7dac305b53fed9c6e004774916e8a258a25b938e5a4496d95287ba"} Oct 30 16:46:57.283578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.281123 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q" Oct 30 16:46:57.304303 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.304217 6202 scope.go:122] "RemoveContainer" containerID="7a41858953788d1abeb4a2f05bd7d09017e4d57bca1a7bdc67ddf57e85f4fd5c" Oct 30 16:46:57.306750 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:46:57.306534 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a41858953788d1abeb4a2f05bd7d09017e4d57bca1a7bdc67ddf57e85f4fd5c\": container with ID starting with 7a41858953788d1abeb4a2f05bd7d09017e4d57bca1a7bdc67ddf57e85f4fd5c not found: ID does not exist" containerID="7a41858953788d1abeb4a2f05bd7d09017e4d57bca1a7bdc67ddf57e85f4fd5c" Oct 30 16:46:57.306750 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.306695 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a41858953788d1abeb4a2f05bd7d09017e4d57bca1a7bdc67ddf57e85f4fd5c"} err="failed to get container status \"7a41858953788d1abeb4a2f05bd7d09017e4d57bca1a7bdc67ddf57e85f4fd5c\": rpc error: code = NotFound desc = could not find container \"7a41858953788d1abeb4a2f05bd7d09017e4d57bca1a7bdc67ddf57e85f4fd5c\": container with ID starting with 7a41858953788d1abeb4a2f05bd7d09017e4d57bca1a7bdc67ddf57e85f4fd5c not found: ID does not exist" Oct 30 16:46:57.306750 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.306722 6202 scope.go:122] "RemoveContainer" containerID="18f855d140aebe67422e376d53027b5c80553d1f19eeb00b4e4d2372ff7e78bf" Oct 30 16:46:57.310751 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.310712 6202 generic.go:356] "Generic (PLEG): container finished" podID="119c2424-9656-41b0-9202-477228e632c2" containerID="d469be6f7acbb8a9449e1ca032cade02dc23e7c0279d005974cd930e5645dc40" exitCode=0 Oct 30 16:46:57.311109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.311086 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" event={"ID":"119c2424-9656-41b0-9202-477228e632c2","Type":"ContainerDied","Data":"d469be6f7acbb8a9449e1ca032cade02dc23e7c0279d005974cd930e5645dc40"} Oct 30 16:46:57.311291 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.311261 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" event={"ID":"119c2424-9656-41b0-9202-477228e632c2","Type":"ContainerDied","Data":"9f55cced3ff9a6fe54360611a71c5a1e525e79315aab8963fe186c7fc5a69420"} Oct 30 16:46:57.311420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.311135 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9" Oct 30 16:46:57.315240 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.315200 6202 generic.go:356] "Generic (PLEG): container finished" podID="71652f10-3fa9-4e81-8394-f631af440cae" containerID="09db65d1e0f1ae274e6bc0de74b7edaa1eb747bd326164da56372ee7ac4b7f72" exitCode=0 Oct 30 16:46:57.315364 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.315251 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" event={"ID":"71652f10-3fa9-4e81-8394-f631af440cae","Type":"ContainerDied","Data":"09db65d1e0f1ae274e6bc0de74b7edaa1eb747bd326164da56372ee7ac4b7f72"} Oct 30 16:46:57.315364 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.315287 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" event={"ID":"71652f10-3fa9-4e81-8394-f631af440cae","Type":"ContainerDied","Data":"759a2fcd4ea42dd6a717d9543ba60681bf71d723a2dab7c1af3e18d1d6de0a31"} Oct 30 16:46:57.316183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.315653 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv" Oct 30 16:46:57.330051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.330026 6202 scope.go:122] "RemoveContainer" containerID="18f855d140aebe67422e376d53027b5c80553d1f19eeb00b4e4d2372ff7e78bf" Oct 30 16:46:57.330353 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:46:57.330331 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f855d140aebe67422e376d53027b5c80553d1f19eeb00b4e4d2372ff7e78bf\": container with ID starting with 18f855d140aebe67422e376d53027b5c80553d1f19eeb00b4e4d2372ff7e78bf not found: ID does not exist" containerID="18f855d140aebe67422e376d53027b5c80553d1f19eeb00b4e4d2372ff7e78bf" Oct 30 16:46:57.330434 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.330373 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f855d140aebe67422e376d53027b5c80553d1f19eeb00b4e4d2372ff7e78bf"} err="failed to get container status \"18f855d140aebe67422e376d53027b5c80553d1f19eeb00b4e4d2372ff7e78bf\": rpc error: code = NotFound desc = could not find container \"18f855d140aebe67422e376d53027b5c80553d1f19eeb00b4e4d2372ff7e78bf\": container with ID starting with 18f855d140aebe67422e376d53027b5c80553d1f19eeb00b4e4d2372ff7e78bf not found: ID does not exist" Oct 30 16:46:57.330434 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.330407 6202 scope.go:122] "RemoveContainer" containerID="d469be6f7acbb8a9449e1ca032cade02dc23e7c0279d005974cd930e5645dc40" Oct 30 16:46:57.347728 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.347644 6202 scope.go:122] "RemoveContainer" containerID="d7f4f4fe6abbe924891c9c8c3ce5557bd6a4f3c51b69a12ab2a24b9c06358c7b" Oct 30 16:46:57.362270 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.360778 6202 scope.go:122] "RemoveContainer" containerID="d469be6f7acbb8a9449e1ca032cade02dc23e7c0279d005974cd930e5645dc40" Oct 30 16:46:57.362491 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:46:57.362465 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d469be6f7acbb8a9449e1ca032cade02dc23e7c0279d005974cd930e5645dc40\": container with ID starting with d469be6f7acbb8a9449e1ca032cade02dc23e7c0279d005974cd930e5645dc40 not found: ID does not exist" containerID="d469be6f7acbb8a9449e1ca032cade02dc23e7c0279d005974cd930e5645dc40" Oct 30 16:46:57.362598 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.362501 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d469be6f7acbb8a9449e1ca032cade02dc23e7c0279d005974cd930e5645dc40"} err="failed to get container status \"d469be6f7acbb8a9449e1ca032cade02dc23e7c0279d005974cd930e5645dc40\": rpc error: code = NotFound desc = could not find container \"d469be6f7acbb8a9449e1ca032cade02dc23e7c0279d005974cd930e5645dc40\": container with ID starting with d469be6f7acbb8a9449e1ca032cade02dc23e7c0279d005974cd930e5645dc40 not found: ID does not exist" Oct 30 16:46:57.362598 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.362528 6202 scope.go:122] "RemoveContainer" containerID="d7f4f4fe6abbe924891c9c8c3ce5557bd6a4f3c51b69a12ab2a24b9c06358c7b" Oct 30 16:46:57.363519 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:46:57.363361 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7f4f4fe6abbe924891c9c8c3ce5557bd6a4f3c51b69a12ab2a24b9c06358c7b\": container with ID starting with d7f4f4fe6abbe924891c9c8c3ce5557bd6a4f3c51b69a12ab2a24b9c06358c7b not found: ID does not exist" containerID="d7f4f4fe6abbe924891c9c8c3ce5557bd6a4f3c51b69a12ab2a24b9c06358c7b" Oct 30 16:46:57.363519 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.363400 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7f4f4fe6abbe924891c9c8c3ce5557bd6a4f3c51b69a12ab2a24b9c06358c7b"} err="failed to get container status \"d7f4f4fe6abbe924891c9c8c3ce5557bd6a4f3c51b69a12ab2a24b9c06358c7b\": rpc error: code = NotFound desc = could not find container \"d7f4f4fe6abbe924891c9c8c3ce5557bd6a4f3c51b69a12ab2a24b9c06358c7b\": container with ID starting with d7f4f4fe6abbe924891c9c8c3ce5557bd6a4f3c51b69a12ab2a24b9c06358c7b not found: ID does not exist" Oct 30 16:46:57.363519 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.363424 6202 scope.go:122] "RemoveContainer" containerID="09db65d1e0f1ae274e6bc0de74b7edaa1eb747bd326164da56372ee7ac4b7f72" Oct 30 16:46:57.384978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.384864 6202 scope.go:122] "RemoveContainer" containerID="09db65d1e0f1ae274e6bc0de74b7edaa1eb747bd326164da56372ee7ac4b7f72" Oct 30 16:46:57.385254 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:46:57.385223 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09db65d1e0f1ae274e6bc0de74b7edaa1eb747bd326164da56372ee7ac4b7f72\": container with ID starting with 09db65d1e0f1ae274e6bc0de74b7edaa1eb747bd326164da56372ee7ac4b7f72 not found: ID does not exist" containerID="09db65d1e0f1ae274e6bc0de74b7edaa1eb747bd326164da56372ee7ac4b7f72" Oct 30 16:46:57.385350 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.385268 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09db65d1e0f1ae274e6bc0de74b7edaa1eb747bd326164da56372ee7ac4b7f72"} err="failed to get container status \"09db65d1e0f1ae274e6bc0de74b7edaa1eb747bd326164da56372ee7ac4b7f72\": rpc error: code = NotFound desc = could not find container \"09db65d1e0f1ae274e6bc0de74b7edaa1eb747bd326164da56372ee7ac4b7f72\": container with ID starting with 09db65d1e0f1ae274e6bc0de74b7edaa1eb747bd326164da56372ee7ac4b7f72 not found: ID does not exist" Oct 30 16:46:57.386417 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.386360 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9"] Oct 30 16:46:57.393150 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.393129 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-72gs9"] Oct 30 16:46:57.418237 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.418212 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc"] Oct 30 16:46:57.423511 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.423486 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-cmlcc"] Oct 30 16:46:57.456348 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.456322 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv"] Oct 30 16:46:57.463717 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.463687 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-2fpdv"] Oct 30 16:46:57.493072 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.493037 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx"] Oct 30 16:46:57.513573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.513539 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:57.515583 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.515563 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q"] Oct 30 16:46:57.534029 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.529933 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-nwm5q"] Oct 30 16:46:57.574341 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.574289 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf"] Oct 30 16:46:57.598646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:57.598622 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9" Oct 30 16:46:58.361592 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:58.361554 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf" event={"ID":"b37d7321-2cef-42aa-80ff-515da8f8a2df","Type":"ContainerStarted","Data":"aed1c1229a836144f60af4b5cac1836ed16f675491117e0748b7d41cc62e733d"} Oct 30 16:46:58.362180 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:58.362155 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf" event={"ID":"b37d7321-2cef-42aa-80ff-515da8f8a2df","Type":"ContainerStarted","Data":"48a9bcc0effdf9bb748bd184fc859d814009d6f71ee509f5aa434e3aff45d9f6"} Oct 30 16:46:58.363657 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:58.363635 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf" Oct 30 16:46:58.365881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:58.365858 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx" event={"ID":"4830c6cf-3abd-485a-87ee-6b094461b5ae","Type":"ContainerStarted","Data":"e4aa1fc29c5328b22696e1a48555f4126f624559bf4670ac1f1a6df5377bea30"} Oct 30 16:46:58.366062 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:58.366046 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx" event={"ID":"4830c6cf-3abd-485a-87ee-6b094461b5ae","Type":"ContainerStarted","Data":"46eee561a20f5fe23d1a31c37a6147251fcd854c3b0d3a77c27fdf70d3a84973"} Oct 30 16:46:58.367141 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:58.367119 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx" Oct 30 16:46:58.417667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:58.417632 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="119c2424-9656-41b0-9202-477228e632c2" path="/var/lib/kubelet/pods/119c2424-9656-41b0-9202-477228e632c2/volumes" Oct 30 16:46:58.418809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:58.418746 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71652f10-3fa9-4e81-8394-f631af440cae" path="/var/lib/kubelet/pods/71652f10-3fa9-4e81-8394-f631af440cae/volumes" Oct 30 16:46:58.419424 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:58.419403 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3f813c4-823b-4109-a2e4-6027d5fabf2e" path="/var/lib/kubelet/pods/c3f813c4-823b-4109-a2e4-6027d5fabf2e/volumes" Oct 30 16:46:58.420475 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:58.420456 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded079d5-65af-48a8-82d2-922eb053d8c2" path="/var/lib/kubelet/pods/ded079d5-65af-48a8-82d2-922eb053d8c2/volumes" Oct 30 16:46:58.470390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:58.470328 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx" podStartSLOduration=2.470306198 podStartE2EDuration="2.470306198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:58.464987754 +0000 UTC m=+1532.175442512" watchObservedRunningTime="2026-10-30 16:46:58.470306198 +0000 UTC m=+1532.180760942" Oct 30 16:46:58.470630 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:58.470446 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf" podStartSLOduration=2.470439066 podStartE2EDuration="2.470439066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:58.417951414 +0000 UTC m=+1532.128406158" watchObservedRunningTime="2026-10-30 16:46:58.470439066 +0000 UTC m=+1532.180893813" Oct 30 16:46:59.571059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:46:59.571012 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:01.210387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:01.210337 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:02.820173 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:02.820134 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:04.705369 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:04.705308 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:06.463969 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:06.463716 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:07.654404 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.654359 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:07.741175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.741132 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm"] Oct 30 16:47:07.743052 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.743022 6202 cpu_manager.go:414] "containerMap: removing container" podUID="ded079d5-65af-48a8-82d2-922eb053d8c2" containerName="istio-proxy" Oct 30 16:47:07.743311 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.743291 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded079d5-65af-48a8-82d2-922eb053d8c2" containerName="istio-proxy" Oct 30 16:47:07.743472 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.743457 6202 cpu_manager.go:414] "containerMap: removing container" podUID="71652f10-3fa9-4e81-8394-f631af440cae" containerName="istio-proxy" Oct 30 16:47:07.743569 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.743558 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71652f10-3fa9-4e81-8394-f631af440cae" containerName="istio-proxy" Oct 30 16:47:07.744016 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.743994 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded079d5-65af-48a8-82d2-922eb053d8c2" containerName="istio-proxy" Oct 30 16:47:07.744164 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.744151 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="71652f10-3fa9-4e81-8394-f631af440cae" containerName="istio-proxy" Oct 30 16:47:07.752083 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.752054 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm" Oct 30 16:47:07.765111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.765082 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm"] Oct 30 16:47:07.772372 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.770084 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:47:07.772372 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.770144 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:47:07.772372 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.770230 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:47:07.778914 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.778892 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:47:07.829360 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.829327 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c04758-eb28-4b22-bd68-ea1eebae6834-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8af660d-vkshm\" (UID: \"98c04758-eb28-4b22-bd68-ea1eebae6834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm" Oct 30 16:47:07.829572 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.829548 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c04758-eb28-4b22-bd68-ea1eebae6834-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8af660d-vkshm\" (UID: \"98c04758-eb28-4b22-bd68-ea1eebae6834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm" Oct 30 16:47:07.829678 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.829666 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms9qv\" (UniqueName: \"kubernetes.io/projected/98c04758-eb28-4b22-bd68-ea1eebae6834-kube-api-access-ms9qv\") pod \"maas-api-revoke-keys-e2e-mt-b8af660d-vkshm\" (UID: \"98c04758-eb28-4b22-bd68-ea1eebae6834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm" Oct 30 16:47:07.931443 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.931349 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c04758-eb28-4b22-bd68-ea1eebae6834-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8af660d-vkshm\" (UID: \"98c04758-eb28-4b22-bd68-ea1eebae6834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm" Oct 30 16:47:07.931620 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.931469 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms9qv\" (UniqueName: \"kubernetes.io/projected/98c04758-eb28-4b22-bd68-ea1eebae6834-kube-api-access-ms9qv\") pod \"maas-api-revoke-keys-e2e-mt-b8af660d-vkshm\" (UID: \"98c04758-eb28-4b22-bd68-ea1eebae6834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm" Oct 30 16:47:07.931620 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.931582 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c04758-eb28-4b22-bd68-ea1eebae6834-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8af660d-vkshm\" (UID: \"98c04758-eb28-4b22-bd68-ea1eebae6834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm" Oct 30 16:47:07.932132 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.932105 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c04758-eb28-4b22-bd68-ea1eebae6834-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8af660d-vkshm\" (UID: \"98c04758-eb28-4b22-bd68-ea1eebae6834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm" Oct 30 16:47:07.934154 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.934126 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c04758-eb28-4b22-bd68-ea1eebae6834-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8af660d-vkshm\" (UID: \"98c04758-eb28-4b22-bd68-ea1eebae6834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm" Oct 30 16:47:07.954641 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:07.954611 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms9qv\" (UniqueName: \"kubernetes.io/projected/98c04758-eb28-4b22-bd68-ea1eebae6834-kube-api-access-ms9qv\") pod \"maas-api-revoke-keys-e2e-mt-b8af660d-vkshm\" (UID: \"98c04758-eb28-4b22-bd68-ea1eebae6834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm" Oct 30 16:47:08.078360 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:08.078325 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm" Oct 30 16:47:08.181948 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:08.181826 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:08.398782 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:08.398735 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm"] Oct 30 16:47:08.399310 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:47:08.399286 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c04758_eb28_4b22_bd68_ea1eebae6834.slice/crio-fc3f9dc2741856e7e55887e511827bcece3596cbcdc321589a3f215c8af4fc14 WatchSource:0}: Error finding container fc3f9dc2741856e7e55887e511827bcece3596cbcdc321589a3f215c8af4fc14: Status 404 returned error can't find the container with id fc3f9dc2741856e7e55887e511827bcece3596cbcdc321589a3f215c8af4fc14 Oct 30 16:47:08.497420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:08.497347 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm" event={"ID":"98c04758-eb28-4b22-bd68-ea1eebae6834","Type":"ContainerStarted","Data":"fc3f9dc2741856e7e55887e511827bcece3596cbcdc321589a3f215c8af4fc14"} Oct 30 16:47:09.176277 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:09.176222 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:09.180113 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:09.180081 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 16:47:09.180264 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:09.180139 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" podUID="a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 16:47:09.298515 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:09.298442 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:09.415099 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:09.415065 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:09.508133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:09.508096 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm" event={"ID":"98c04758-eb28-4b22-bd68-ea1eebae6834","Type":"ContainerStarted","Data":"997978b7af66b4c7370150be4ce6ee0cc724876715d72de8cdb913480c735bcd"} Oct 30 16:47:09.509858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:09.509830 6202 generic.go:356] "Generic (PLEG): container finished" podID="a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" containerID="7d88a428f5b39bb6d46ebb63910aaf26c2b899da2884e7ff3b06eee592b30ce5" exitCode=0 Oct 30 16:47:09.509993 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:09.509903 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" event={"ID":"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2","Type":"ContainerDied","Data":"7d88a428f5b39bb6d46ebb63910aaf26c2b899da2884e7ff3b06eee592b30ce5"} Oct 30 16:47:09.510325 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:09.510310 6202 scope.go:122] "RemoveContainer" containerID="7d88a428f5b39bb6d46ebb63910aaf26c2b899da2884e7ff3b06eee592b30ce5" Oct 30 16:47:09.512133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:09.512106 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:47:09.512207 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:09.512177 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:47:09.512246 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:09.512207 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:47:09.555306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:09.555248 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm" podStartSLOduration=2.555229168 podStartE2EDuration="2.555229168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:09.549214582 +0000 UTC m=+1543.259669328" watchObservedRunningTime="2026-10-30 16:47:09.555229168 +0000 UTC m=+1543.265683916" Oct 30 16:47:10.260792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:10.259786 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:10.392789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:10.390603 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf" Oct 30 16:47:10.396787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:10.394233 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx" Oct 30 16:47:10.523711 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:10.523615 6202 generic.go:356] "Generic (PLEG): container finished" podID="98c04758-eb28-4b22-bd68-ea1eebae6834" containerID="997978b7af66b4c7370150be4ce6ee0cc724876715d72de8cdb913480c735bcd" exitCode=0 Oct 30 16:47:10.523942 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:10.523717 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm" event={"ID":"98c04758-eb28-4b22-bd68-ea1eebae6834","Type":"ContainerDied","Data":"997978b7af66b4c7370150be4ce6ee0cc724876715d72de8cdb913480c735bcd"} Oct 30 16:47:10.526613 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:10.526587 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" event={"ID":"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2","Type":"ContainerStarted","Data":"5e75df24ec30f457fda6ba267ef67755946b41012e320b0ce62365cdeaf1a3b6"} Oct 30 16:47:10.603573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:10.603539 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:10.731233 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:10.731197 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:10.861790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:10.861197 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:10.975379 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:10.975337 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:11.293851 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:11.289197 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:47:11.293851 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:11.292888 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 16:47:11.293851 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:11.292962 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" podUID="a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 16:47:12.026264 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.026205 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm" Oct 30 16:47:12.134266 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.134235 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98c04758-eb28-4b22-bd68-ea1eebae6834-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c04758-eb28-4b22-bd68-ea1eebae6834-maas-api-xks-ca\") pod \"98c04758-eb28-4b22-bd68-ea1eebae6834\" (UID: \"98c04758-eb28-4b22-bd68-ea1eebae6834\") " Oct 30 16:47:12.134508 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.134421 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98c04758-eb28-4b22-bd68-ea1eebae6834-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c04758-eb28-4b22-bd68-ea1eebae6834-maas-api-openshift-ca\") pod \"98c04758-eb28-4b22-bd68-ea1eebae6834\" (UID: \"98c04758-eb28-4b22-bd68-ea1eebae6834\") " Oct 30 16:47:12.134662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.134640 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c04758-eb28-4b22-bd68-ea1eebae6834-kube-api-access-ms9qv\" (UniqueName: \"kubernetes.io/projected/98c04758-eb28-4b22-bd68-ea1eebae6834-kube-api-access-ms9qv\") pod \"98c04758-eb28-4b22-bd68-ea1eebae6834\" (UID: \"98c04758-eb28-4b22-bd68-ea1eebae6834\") " Oct 30 16:47:12.135052 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.135022 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c04758-eb28-4b22-bd68-ea1eebae6834-maas-api-openshift-ca" pod "98c04758-eb28-4b22-bd68-ea1eebae6834" (UID: "98c04758-eb28-4b22-bd68-ea1eebae6834"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:12.135592 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.135568 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c04758-eb28-4b22-bd68-ea1eebae6834-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:12.137120 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.137095 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98c04758-eb28-4b22-bd68-ea1eebae6834-maas-api-xks-ca" pod "98c04758-eb28-4b22-bd68-ea1eebae6834" (UID: "98c04758-eb28-4b22-bd68-ea1eebae6834"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:47:12.137211 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.137098 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c04758-eb28-4b22-bd68-ea1eebae6834-kube-api-access-ms9qv" pod "98c04758-eb28-4b22-bd68-ea1eebae6834" (UID: "98c04758-eb28-4b22-bd68-ea1eebae6834"). InnerVolumeSpecName "kube-api-access-ms9qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:12.227227 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.227065 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:12.237523 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.237493 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms9qv\" (UniqueName: \"kubernetes.io/projected/98c04758-eb28-4b22-bd68-ea1eebae6834-kube-api-access-ms9qv\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:12.237523 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.237520 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c04758-eb28-4b22-bd68-ea1eebae6834-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:12.295842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.291138 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 16:47:12.295842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.291202 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" podUID="a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 16:47:12.516449 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.516417 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:12.520187 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.520155 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz"] Oct 30 16:47:12.521040 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.521024 6202 cpu_manager.go:414] "containerMap: removing container" podUID="98c04758-eb28-4b22-bd68-ea1eebae6834" containerName="revoke-keys" Oct 30 16:47:12.521096 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.521042 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c04758-eb28-4b22-bd68-ea1eebae6834" containerName="revoke-keys" Oct 30 16:47:12.521205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.521195 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c04758-eb28-4b22-bd68-ea1eebae6834" containerName="revoke-keys" Oct 30 16:47:12.525054 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.525038 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz" Oct 30 16:47:12.612704 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.612671 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0bf836f-6716-4a39-8422-0c9639aa5f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-743d2769-2nwbz\" (UID: \"d0bf836f-6716-4a39-8422-0c9639aa5f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz" Oct 30 16:47:12.612957 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.612831 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0bf836f-6716-4a39-8422-0c9639aa5f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-743d2769-2nwbz\" (UID: \"d0bf836f-6716-4a39-8422-0c9639aa5f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz" Oct 30 16:47:12.612957 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.612933 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dnqw\" (UniqueName: \"kubernetes.io/projected/d0bf836f-6716-4a39-8422-0c9639aa5f8c-kube-api-access-9dnqw\") pod \"maas-api-revoke-sub-743d2769-2nwbz\" (UID: \"d0bf836f-6716-4a39-8422-0c9639aa5f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz" Oct 30 16:47:12.615064 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.615040 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz"] Oct 30 16:47:12.699809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.661100 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:12.699809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.669357 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm" event={"ID":"98c04758-eb28-4b22-bd68-ea1eebae6834","Type":"ContainerDied","Data":"fc3f9dc2741856e7e55887e511827bcece3596cbcdc321589a3f215c8af4fc14"} Oct 30 16:47:12.699809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.669454 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc3f9dc2741856e7e55887e511827bcece3596cbcdc321589a3f215c8af4fc14" Oct 30 16:47:12.699809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.669550 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm" Oct 30 16:47:12.715646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.715616 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0bf836f-6716-4a39-8422-0c9639aa5f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-743d2769-2nwbz\" (UID: \"d0bf836f-6716-4a39-8422-0c9639aa5f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz" Oct 30 16:47:12.715825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.715696 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dnqw\" (UniqueName: \"kubernetes.io/projected/d0bf836f-6716-4a39-8422-0c9639aa5f8c-kube-api-access-9dnqw\") pod \"maas-api-revoke-sub-743d2769-2nwbz\" (UID: \"d0bf836f-6716-4a39-8422-0c9639aa5f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz" Oct 30 16:47:12.715874 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.715857 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0bf836f-6716-4a39-8422-0c9639aa5f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-743d2769-2nwbz\" (UID: \"d0bf836f-6716-4a39-8422-0c9639aa5f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz" Oct 30 16:47:12.716509 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.716475 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0bf836f-6716-4a39-8422-0c9639aa5f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-743d2769-2nwbz\" (UID: \"d0bf836f-6716-4a39-8422-0c9639aa5f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz" Oct 30 16:47:12.720234 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.720201 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0bf836f-6716-4a39-8422-0c9639aa5f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-743d2769-2nwbz\" (UID: \"d0bf836f-6716-4a39-8422-0c9639aa5f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz" Oct 30 16:47:12.796793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.796552 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dnqw\" (UniqueName: \"kubernetes.io/projected/d0bf836f-6716-4a39-8422-0c9639aa5f8c-kube-api-access-9dnqw\") pod \"maas-api-revoke-sub-743d2769-2nwbz\" (UID: \"d0bf836f-6716-4a39-8422-0c9639aa5f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz" Oct 30 16:47:12.861793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:12.860812 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz" Oct 30 16:47:13.067802 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:13.066822 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj"] Oct 30 16:47:13.097857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:13.097456 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dccd7785-pnrtj"] Oct 30 16:47:13.305890 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:13.296194 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 16:47:13.305890 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:13.296258 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" podUID="a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 16:47:13.312841 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:47:13.312810 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0bf836f_6716_4a39_8422_0c9639aa5f8c.slice/crio-45d9b4980c6652926cbf1f1c1c97c5a55e1880dd0eda6adc7ef6ce05c9c31629 WatchSource:0}: Error finding container 45d9b4980c6652926cbf1f1c1c97c5a55e1880dd0eda6adc7ef6ce05c9c31629: Status 404 returned error can't find the container with id 45d9b4980c6652926cbf1f1c1c97c5a55e1880dd0eda6adc7ef6ce05c9c31629 Oct 30 16:47:13.314580 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:13.314557 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz"] Oct 30 16:47:13.714109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:13.714077 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz" event={"ID":"d0bf836f-6716-4a39-8422-0c9639aa5f8c","Type":"ContainerStarted","Data":"45d9b4980c6652926cbf1f1c1c97c5a55e1880dd0eda6adc7ef6ce05c9c31629"} Oct 30 16:47:14.006384 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:14.006303 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:14.206852 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:47:14.206515 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:47:14.290791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:14.290178 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 16:47:14.290791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:14.290241 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" podUID="a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 16:47:14.415196 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:14.414829 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae22304-8037-4bb2-9b6c-35f547407dd8" path="/var/lib/kubelet/pods/5ae22304-8037-4bb2-9b6c-35f547407dd8/volumes" Oct 30 16:47:14.726736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:14.726701 6202 generic.go:356] "Generic (PLEG): container finished" podID="d0bf836f-6716-4a39-8422-0c9639aa5f8c" containerID="b0bd806e805e56199ee57f91b736210203e3f3858efff40cab280be7d3e78cab" exitCode=0 Oct 30 16:47:14.727000 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:14.726788 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz" event={"ID":"d0bf836f-6716-4a39-8422-0c9639aa5f8c","Type":"ContainerDied","Data":"b0bd806e805e56199ee57f91b736210203e3f3858efff40cab280be7d3e78cab"} Oct 30 16:47:14.872436 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:14.872401 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:15.297851 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:15.290160 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 16:47:15.297851 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:15.290222 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" podUID="a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 16:47:15.670248 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:15.670182 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:15.811302 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:15.807071 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:16.182783 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:16.182744 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz" Oct 30 16:47:16.278884 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:16.259126 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0bf836f-6716-4a39-8422-0c9639aa5f8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0bf836f-6716-4a39-8422-0c9639aa5f8c-maas-api-openshift-ca\") pod \"d0bf836f-6716-4a39-8422-0c9639aa5f8c\" (UID: \"d0bf836f-6716-4a39-8422-0c9639aa5f8c\") " Oct 30 16:47:16.278884 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:16.259204 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0bf836f-6716-4a39-8422-0c9639aa5f8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0bf836f-6716-4a39-8422-0c9639aa5f8c-maas-api-xks-ca\") pod \"d0bf836f-6716-4a39-8422-0c9639aa5f8c\" (UID: \"d0bf836f-6716-4a39-8422-0c9639aa5f8c\") " Oct 30 16:47:16.278884 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:16.259453 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0bf836f-6716-4a39-8422-0c9639aa5f8c-kube-api-access-9dnqw\" (UniqueName: \"kubernetes.io/projected/d0bf836f-6716-4a39-8422-0c9639aa5f8c-kube-api-access-9dnqw\") pod \"d0bf836f-6716-4a39-8422-0c9639aa5f8c\" (UID: \"d0bf836f-6716-4a39-8422-0c9639aa5f8c\") " Oct 30 16:47:16.278884 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:16.263136 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bf836f-6716-4a39-8422-0c9639aa5f8c-maas-api-openshift-ca" pod "d0bf836f-6716-4a39-8422-0c9639aa5f8c" (UID: "d0bf836f-6716-4a39-8422-0c9639aa5f8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:16.289260 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:16.289211 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 16:47:16.289452 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:16.289292 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" podUID="a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 16:47:16.294020 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:16.293973 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0bf836f-6716-4a39-8422-0c9639aa5f8c-kube-api-access-9dnqw" pod "d0bf836f-6716-4a39-8422-0c9639aa5f8c" (UID: "d0bf836f-6716-4a39-8422-0c9639aa5f8c"). InnerVolumeSpecName "kube-api-access-9dnqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:16.300906 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:16.300867 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bf836f-6716-4a39-8422-0c9639aa5f8c-maas-api-xks-ca" pod "d0bf836f-6716-4a39-8422-0c9639aa5f8c" (UID: "d0bf836f-6716-4a39-8422-0c9639aa5f8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:47:16.374793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:16.373232 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dnqw\" (UniqueName: \"kubernetes.io/projected/d0bf836f-6716-4a39-8422-0c9639aa5f8c-kube-api-access-9dnqw\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:16.374793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:16.373275 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0bf836f-6716-4a39-8422-0c9639aa5f8c-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:16.374793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:16.373294 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0bf836f-6716-4a39-8422-0c9639aa5f8c-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:16.800850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:16.800821 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz" Oct 30 16:47:16.801319 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:16.800844 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz" event={"ID":"d0bf836f-6716-4a39-8422-0c9639aa5f8c","Type":"ContainerDied","Data":"45d9b4980c6652926cbf1f1c1c97c5a55e1880dd0eda6adc7ef6ce05c9c31629"} Oct 30 16:47:16.801319 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:16.800903 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45d9b4980c6652926cbf1f1c1c97c5a55e1880dd0eda6adc7ef6ce05c9c31629" Oct 30 16:47:17.291812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:17.291737 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:17.294435 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:17.294411 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:47:17.294823 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:17.294806 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:47:17.295533 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:17.295517 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 16:47:17.413826 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:17.413328 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:17.750521 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:17.750479 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:18.234871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:18.234829 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk"] Oct 30 16:47:18.348879 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:18.337257 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:18.460265 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:18.460134 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:19.336634 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:19.336597 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:21.045611 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:21.045232 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl"] Oct 30 16:47:21.148250 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:21.148151 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca4e901d-cc9zl"] Oct 30 16:47:21.303684 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:21.303582 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:21.915250 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:21.915219 6202 generic.go:356] "Generic (PLEG): container finished" podID="6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" containerID="7586261b741c10e7484cf3219c709e5b65080b977caf1ca27e51b0f2cdf1e0c9" exitCode=0 Oct 30 16:47:21.915451 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:21.915296 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" event={"ID":"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd","Type":"ContainerDied","Data":"7586261b741c10e7484cf3219c709e5b65080b977caf1ca27e51b0f2cdf1e0c9"} Oct 30 16:47:21.915862 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:21.915843 6202 scope.go:122] "RemoveContainer" containerID="7586261b741c10e7484cf3219c709e5b65080b977caf1ca27e51b0f2cdf1e0c9" Oct 30 16:47:21.917712 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:21.917685 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:47:21.917801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:21.917756 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:47:21.917842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:21.917806 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:47:22.447827 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:22.445136 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a97e64ec-670e-4bee-99a1-b681a3ea1328" path="/var/lib/kubelet/pods/a97e64ec-670e-4bee-99a1-b681a3ea1328/volumes" Oct 30 16:47:22.893025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:22.892988 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:22.925922 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:22.925887 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" event={"ID":"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd","Type":"ContainerStarted","Data":"485ba0b4fc23776c5ad4f1dbb1f9f6d0e8fb041f8fe211f74c3c4ab9ecb44190"} Oct 30 16:47:23.070122 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:23.070079 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:23.236388 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:23.236120 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" podUID="082aec51-71a4-48f7-994b-bd6cc7e05e49" containerName="payload-pre-processing" containerID="cri-o://13b333e57b4d0b695a71b2f305149001ba6a71e90d696c494a5aef8ea53a4b3b" gracePeriod=25 Oct 30 16:47:23.516248 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:23.516155 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:47:23.517341 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:23.517310 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 16:47:23.517578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:23.517385 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" podUID="6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 16:47:24.517105 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:24.517074 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 16:47:24.517498 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:24.517124 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" podUID="6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 16:47:24.633436 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:24.633391 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:25.518359 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:25.518315 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 16:47:25.518927 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:25.518385 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" podUID="6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 16:47:26.034735 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:26.034704 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m"] Oct 30 16:47:26.089263 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:26.089227 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5101796f-8847m"] Oct 30 16:47:26.385710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:26.385674 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:26.425571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:26.425536 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d71ead75-5cbc-475e-9ade-8406f28427f7" path="/var/lib/kubelet/pods/d71ead75-5cbc-475e-9ade-8406f28427f7/volumes" Oct 30 16:47:26.516981 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:26.516944 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 16:47:26.517339 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:26.517307 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" podUID="6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 16:47:27.516331 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:27.516298 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 16:47:27.516742 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:27.516352 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" podUID="6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 16:47:27.846792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:27.842854 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:47:27.928682 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:27.928643 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:28.522086 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:28.522055 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:47:28.523307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:28.523284 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 16:47:29.327420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:29.327380 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" podUID="082aec51-71a4-48f7-994b-bd6cc7e05e49" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:47:29.919142 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:29.919097 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:30.053790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:30.041945 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:30.317081 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:30.316991 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:30.493888 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:30.493848 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:30.557948 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:30.557912 6202 scope.go:122] "RemoveContainer" containerID="403873e8f0995424f767a7814e16265b410b61fed4453fe69f96780b18f8dc9f" Oct 30 16:47:30.582009 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:30.581982 6202 scope.go:122] "RemoveContainer" containerID="c74861b6080b8cc6f504716993a7a8b990784fc9eb767fdc8fd887d6de13d906" Oct 30 16:47:30.594281 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:30.594249 6202 scope.go:122] "RemoveContainer" containerID="bc14d88d9a035e2603e3e85501fddd33ab1f3bdf6f4e2f764cb3f8ef17488a71" Oct 30 16:47:30.608723 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:30.608702 6202 scope.go:122] "RemoveContainer" containerID="de7758be79136db98681aa638138a24e76d47e26cc575e9eb44070a93eb025f1" Oct 30 16:47:30.618836 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:30.618812 6202 scope.go:122] "RemoveContainer" containerID="1f0c0eace06cd246e78882bb72d8644cede8b4758c72807b4ac4933777906de2" Oct 30 16:47:30.628683 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:30.628667 6202 scope.go:122] "RemoveContainer" containerID="1528e918cf4a3df4b1568c2d527e868cd69aac8860e79fbf369c9cc856eeb572" Oct 30 16:47:30.638369 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:30.638118 6202 scope.go:122] "RemoveContainer" containerID="49271aff4561a7679c36050a88569c00749b12279905b1c82948094c0e3ae1ae" Oct 30 16:47:30.816298 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:30.816268 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:31.036092 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:31.036056 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw"] Oct 30 16:47:31.063834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:31.063803 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd47dcfc-4bjzw"] Oct 30 16:47:31.819446 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:31.819398 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:32.412983 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:32.412949 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ed0399-456d-4278-bb7d-0265626a65c6" path="/var/lib/kubelet/pods/11ed0399-456d-4278-bb7d-0265626a65c6/volumes" Oct 30 16:47:32.972248 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:32.972207 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:33.405970 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:33.405936 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:33.528214 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:33.528184 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:33.759845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:33.756153 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:34.272298 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:34.272255 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:34.789491 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:34.789455 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:34.961509 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:34.961433 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:35.390161 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.390127 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg"] Oct 30 16:47:35.390366 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.390344 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" podUID="80a5af6e-f20f-4745-83d3-fd5df7661cc3" containerName="istio-proxy" containerID="cri-o://c9d2dd04ff1419c63bdaeab4082fabb27a4ac267999a9a240987d8b00f65d8d8" gracePeriod=30 Oct 30 16:47:35.395680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.395654 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg"] Oct 30 16:47:35.395874 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.395854 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" podUID="e8bd28b1-844d-4b4f-bed7-542915f85c47" containerName="istio-proxy" containerID="cri-o://491f6cb6efb758dd1b31f9909e957fa419b32c053ee233695b38cc2ebf184f8e" gracePeriod=30 Oct 30 16:47:35.400156 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.400122 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp"] Oct 30 16:47:35.400325 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.400306 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" podUID="4951f605-1e39-41f0-8d0a-2b23791653dd" containerName="istio-proxy" containerID="cri-o://3a3887b53d5fade58db199dcd2945e18802feea2160f6b4ace28bf945edca57d" gracePeriod=30 Oct 30 16:47:35.498847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.498815 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp"] Oct 30 16:47:35.499651 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.499637 6202 cpu_manager.go:414] "containerMap: removing container" podUID="d0bf836f-6716-4a39-8422-0c9639aa5f8c" containerName="revoke-keys" Oct 30 16:47:35.499703 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.499653 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0bf836f-6716-4a39-8422-0c9639aa5f8c" containerName="revoke-keys" Oct 30 16:47:35.499805 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.499781 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0bf836f-6716-4a39-8422-0c9639aa5f8c" containerName="revoke-keys" Oct 30 16:47:35.522956 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.522824 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp" Oct 30 16:47:35.541398 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.541339 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:47:35.541788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.541726 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:47:35.548325 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.547950 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:47:35.548325 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.548232 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:47:35.551440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.551411 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp"] Oct 30 16:47:35.652341 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.652257 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z2mw\" (UniqueName: \"kubernetes.io/projected/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-kube-api-access-9z2mw\") pod \"maas-api-revoke-sub-ae109b2d-46tbp\" (UID: \"4d8a6c96-1fff-4292-88b5-4d939fd0c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp" Oct 30 16:47:35.652341 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.652318 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae109b2d-46tbp\" (UID: \"4d8a6c96-1fff-4292-88b5-4d939fd0c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp" Oct 30 16:47:35.652646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.652619 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae109b2d-46tbp\" (UID: \"4d8a6c96-1fff-4292-88b5-4d939fd0c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp" Oct 30 16:47:35.754326 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.754286 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae109b2d-46tbp\" (UID: \"4d8a6c96-1fff-4292-88b5-4d939fd0c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp" Oct 30 16:47:35.754528 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.754515 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z2mw\" (UniqueName: \"kubernetes.io/projected/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-kube-api-access-9z2mw\") pod \"maas-api-revoke-sub-ae109b2d-46tbp\" (UID: \"4d8a6c96-1fff-4292-88b5-4d939fd0c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp" Oct 30 16:47:35.754610 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.754563 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae109b2d-46tbp\" (UID: \"4d8a6c96-1fff-4292-88b5-4d939fd0c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp" Oct 30 16:47:35.755434 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.755403 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae109b2d-46tbp\" (UID: \"4d8a6c96-1fff-4292-88b5-4d939fd0c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp" Oct 30 16:47:35.757100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.757066 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae109b2d-46tbp\" (UID: \"4d8a6c96-1fff-4292-88b5-4d939fd0c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp" Oct 30 16:47:35.781621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.781577 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z2mw\" (UniqueName: \"kubernetes.io/projected/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-kube-api-access-9z2mw\") pod \"maas-api-revoke-sub-ae109b2d-46tbp\" (UID: \"4d8a6c96-1fff-4292-88b5-4d939fd0c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp" Oct 30 16:47:35.844340 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.844309 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:35.876399 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:35.876332 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp" Oct 30 16:47:36.261976 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:47:36.261942 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8a6c96_1fff_4292_88b5_4d939fd0c99a.slice/crio-3a743c6aae3396a569fc831ab92bd58a319e3d2ea28a4af440af7f0fe79b2f58 WatchSource:0}: Error finding container 3a743c6aae3396a569fc831ab92bd58a319e3d2ea28a4af440af7f0fe79b2f58: Status 404 returned error can't find the container with id 3a743c6aae3396a569fc831ab92bd58a319e3d2ea28a4af440af7f0fe79b2f58 Oct 30 16:47:36.262110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:36.262082 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp"] Oct 30 16:47:36.293937 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:36.293906 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp" event={"ID":"4d8a6c96-1fff-4292-88b5-4d939fd0c99a","Type":"ContainerStarted","Data":"3a743c6aae3396a569fc831ab92bd58a319e3d2ea28a4af440af7f0fe79b2f58"} Oct 30 16:47:37.303939 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:37.303904 6202 generic.go:356] "Generic (PLEG): container finished" podID="4d8a6c96-1fff-4292-88b5-4d939fd0c99a" containerID="1661f13d6ad63ce25ebacb6d57b6c874e7d90f2505083857d31f2394da060641" exitCode=0 Oct 30 16:47:37.304396 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:37.303999 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp" event={"ID":"4d8a6c96-1fff-4292-88b5-4d939fd0c99a","Type":"ContainerDied","Data":"1661f13d6ad63ce25ebacb6d57b6c874e7d90f2505083857d31f2394da060641"} Oct 30 16:47:37.723699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:37.723628 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:38.505895 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:38.505859 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp" Oct 30 16:47:38.599103 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:38.599059 6202 patch_prober.go:29] interesting pod/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:47:38.599307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:38.599143 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" podUID="9f4d8936-b54c-43f0-acfb-6edc1871e260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:47:38.606431 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:38.606398 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-kube-api-access-9z2mw\" (UniqueName: \"kubernetes.io/projected/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-kube-api-access-9z2mw\") pod \"4d8a6c96-1fff-4292-88b5-4d939fd0c99a\" (UID: \"4d8a6c96-1fff-4292-88b5-4d939fd0c99a\") " Oct 30 16:47:38.606550 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:38.606451 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-maas-api-xks-ca\") pod \"4d8a6c96-1fff-4292-88b5-4d939fd0c99a\" (UID: \"4d8a6c96-1fff-4292-88b5-4d939fd0c99a\") " Oct 30 16:47:38.606550 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:38.606498 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-maas-api-openshift-ca\") pod \"4d8a6c96-1fff-4292-88b5-4d939fd0c99a\" (UID: \"4d8a6c96-1fff-4292-88b5-4d939fd0c99a\") " Oct 30 16:47:38.607412 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:38.607276 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-maas-api-openshift-ca" pod "4d8a6c96-1fff-4292-88b5-4d939fd0c99a" (UID: "4d8a6c96-1fff-4292-88b5-4d939fd0c99a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:38.608927 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:38.608905 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-maas-api-xks-ca" pod "4d8a6c96-1fff-4292-88b5-4d939fd0c99a" (UID: "4d8a6c96-1fff-4292-88b5-4d939fd0c99a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:47:38.609211 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:38.609192 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-kube-api-access-9z2mw" pod "4d8a6c96-1fff-4292-88b5-4d939fd0c99a" (UID: "4d8a6c96-1fff-4292-88b5-4d939fd0c99a"). InnerVolumeSpecName "kube-api-access-9z2mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:38.709122 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:38.709091 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z2mw\" (UniqueName: \"kubernetes.io/projected/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-kube-api-access-9z2mw\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:38.709122 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:38.709119 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:38.709122 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:38.709130 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d8a6c96-1fff-4292-88b5-4d939fd0c99a-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:38.869525 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:38.869409 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:39.234858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:39.234756 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:39.323478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:39.323443 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp" Oct 30 16:47:39.323694 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:39.323518 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp" event={"ID":"4d8a6c96-1fff-4292-88b5-4d939fd0c99a","Type":"ContainerDied","Data":"3a743c6aae3396a569fc831ab92bd58a319e3d2ea28a4af440af7f0fe79b2f58"} Oct 30 16:47:39.323694 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:39.323568 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a743c6aae3396a569fc831ab92bd58a319e3d2ea28a4af440af7f0fe79b2f58" Oct 30 16:47:39.330108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:39.330083 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" podUID="082aec51-71a4-48f7-994b-bd6cc7e05e49" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:47:39.365661 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:39.365624 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:39.515858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:39.515759 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:39.645604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:39.645558 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:40.334191 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.334156 6202 generic.go:356] "Generic (PLEG): container finished" podID="9f4d8936-b54c-43f0-acfb-6edc1871e260" containerID="02e9dd07c45d190160cf388a4b085dff2107acc4eeded43bc3b55a1a39b826bb" exitCode=0 Oct 30 16:47:40.334402 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.334227 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" event={"ID":"9f4d8936-b54c-43f0-acfb-6edc1871e260","Type":"ContainerDied","Data":"02e9dd07c45d190160cf388a4b085dff2107acc4eeded43bc3b55a1a39b826bb"} Oct 30 16:47:40.334744 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.334725 6202 scope.go:122] "RemoveContainer" containerID="02e9dd07c45d190160cf388a4b085dff2107acc4eeded43bc3b55a1a39b826bb" Oct 30 16:47:40.336484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.336449 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:47:40.336593 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.336538 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:47:40.336593 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.336587 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:47:40.758131 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.758059 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:47:40.792728 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.791525 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:47:40.846123 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.846062 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-workload-socket\") pod \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " Oct 30 16:47:40.846123 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.846125 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-data\") pod \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " Oct 30 16:47:40.846390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.846210 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-podinfo\") pod \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " Oct 30 16:47:40.846390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.846322 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istiod-ca-cert\") pod \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " Oct 30 16:47:40.846513 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.846394 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-credential-socket\") pod \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " Oct 30 16:47:40.846572 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.846521 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a5af6e-f20f-4745-83d3-fd5df7661cc3-kube-api-access-9zdht\" (UniqueName: \"kubernetes.io/projected/80a5af6e-f20f-4745-83d3-fd5df7661cc3-kube-api-access-9zdht\") pod \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " Oct 30 16:47:40.846638 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.846584 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-envoy\") pod \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " Oct 30 16:47:40.846638 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.846627 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-token\" (UniqueName: \"kubernetes.io/projected/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-token\") pod \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " Oct 30 16:47:40.846748 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.846738 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-workload-certs\") pod \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\" (UID: \"80a5af6e-f20f-4745-83d3-fd5df7661cc3\") " Oct 30 16:47:40.848424 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.848385 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-workload-socket" pod "80a5af6e-f20f-4745-83d3-fd5df7661cc3" (UID: "80a5af6e-f20f-4745-83d3-fd5df7661cc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:40.852983 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.852619 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-credential-socket" pod "80a5af6e-f20f-4745-83d3-fd5df7661cc3" (UID: "80a5af6e-f20f-4745-83d3-fd5df7661cc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:40.853485 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.853164 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-data" pod "80a5af6e-f20f-4745-83d3-fd5df7661cc3" (UID: "80a5af6e-f20f-4745-83d3-fd5df7661cc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:40.858504 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.858459 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd"] Oct 30 16:47:40.859895 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.859862 6202 cpu_manager.go:414] "containerMap: removing container" podUID="80a5af6e-f20f-4745-83d3-fd5df7661cc3" containerName="istio-proxy" Oct 30 16:47:40.860361 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.860345 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a5af6e-f20f-4745-83d3-fd5df7661cc3" containerName="istio-proxy" Oct 30 16:47:40.860539 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.860512 6202 cpu_manager.go:414] "containerMap: removing container" podUID="4d8a6c96-1fff-4292-88b5-4d939fd0c99a" containerName="revoke-keys" Oct 30 16:47:40.860680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.860666 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d8a6c96-1fff-4292-88b5-4d939fd0c99a" containerName="revoke-keys" Oct 30 16:47:40.860895 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.860869 6202 cpu_manager.go:414] "containerMap: removing container" podUID="4951f605-1e39-41f0-8d0a-2b23791653dd" containerName="istio-proxy" Oct 30 16:47:40.861036 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.861022 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4951f605-1e39-41f0-8d0a-2b23791653dd" containerName="istio-proxy" Oct 30 16:47:40.861381 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.861349 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a5af6e-f20f-4745-83d3-fd5df7661cc3" containerName="istio-proxy" Oct 30 16:47:40.861533 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.861520 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d8a6c96-1fff-4292-88b5-4d939fd0c99a" containerName="revoke-keys" Oct 30 16:47:40.861684 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.861672 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="4951f605-1e39-41f0-8d0a-2b23791653dd" containerName="istio-proxy" Oct 30 16:47:40.882278 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.860227 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istiod-ca-cert" pod "80a5af6e-f20f-4745-83d3-fd5df7661cc3" (UID: "80a5af6e-f20f-4745-83d3-fd5df7661cc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:40.914406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.860297 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-workload-certs" pod "80a5af6e-f20f-4745-83d3-fd5df7661cc3" (UID: "80a5af6e-f20f-4745-83d3-fd5df7661cc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:40.914406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.882552 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-token" pod "80a5af6e-f20f-4745-83d3-fd5df7661cc3" (UID: "80a5af6e-f20f-4745-83d3-fd5df7661cc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:40.914406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.891240 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-envoy" pod "80a5af6e-f20f-4745-83d3-fd5df7661cc3" (UID: "80a5af6e-f20f-4745-83d3-fd5df7661cc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:40.914406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.891470 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:40.914406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.905264 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd"] Oct 30 16:47:40.914406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.909895 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a5af6e-f20f-4745-83d3-fd5df7661cc3-kube-api-access-9zdht" pod "80a5af6e-f20f-4745-83d3-fd5df7661cc3" (UID: "80a5af6e-f20f-4745-83d3-fd5df7661cc3"). InnerVolumeSpecName "kube-api-access-9zdht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:40.914406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.912890 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-podinfo" pod "80a5af6e-f20f-4745-83d3-fd5df7661cc3" (UID: "80a5af6e-f20f-4745-83d3-fd5df7661cc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:47:40.917851 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.917823 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-529bc7-openshift-default-dockercfg-zfd7x\"" Oct 30 16:47:40.944955 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.944925 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:47:40.958955 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.958849 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-workload-socket\") pod \"4951f605-1e39-41f0-8d0a-2b23791653dd\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " Oct 30 16:47:40.958955 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.958916 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-data\") pod \"4951f605-1e39-41f0-8d0a-2b23791653dd\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " Oct 30 16:47:40.959210 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.958984 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-credential-socket\") pod \"4951f605-1e39-41f0-8d0a-2b23791653dd\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " Oct 30 16:47:40.959210 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.959040 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4951f605-1e39-41f0-8d0a-2b23791653dd-kube-api-access-7d5lm\" (UniqueName: \"kubernetes.io/projected/4951f605-1e39-41f0-8d0a-2b23791653dd-kube-api-access-7d5lm\") pod \"4951f605-1e39-41f0-8d0a-2b23791653dd\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " Oct 30 16:47:40.959210 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.959134 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-envoy\") pod \"4951f605-1e39-41f0-8d0a-2b23791653dd\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " Oct 30 16:47:40.959210 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.959187 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-token\" (UniqueName: \"kubernetes.io/projected/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-token\") pod \"4951f605-1e39-41f0-8d0a-2b23791653dd\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " Oct 30 16:47:40.959431 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.959368 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-podinfo\") pod \"4951f605-1e39-41f0-8d0a-2b23791653dd\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " Oct 30 16:47:40.959529 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.959513 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-workload-certs\") pod \"4951f605-1e39-41f0-8d0a-2b23791653dd\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " Oct 30 16:47:40.959612 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.959565 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4951f605-1e39-41f0-8d0a-2b23791653dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4951f605-1e39-41f0-8d0a-2b23791653dd-istiod-ca-cert\") pod \"4951f605-1e39-41f0-8d0a-2b23791653dd\" (UID: \"4951f605-1e39-41f0-8d0a-2b23791653dd\") " Oct 30 16:47:40.962159 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.961879 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:40.962386 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.962328 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-data" pod "4951f605-1e39-41f0-8d0a-2b23791653dd" (UID: "4951f605-1e39-41f0-8d0a-2b23791653dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:40.962566 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.962544 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-workload-socket" pod "4951f605-1e39-41f0-8d0a-2b23791653dd" (UID: "4951f605-1e39-41f0-8d0a-2b23791653dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:40.963225 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.962903 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-credential-socket" pod "4951f605-1e39-41f0-8d0a-2b23791653dd" (UID: "4951f605-1e39-41f0-8d0a-2b23791653dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.965689 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-podinfo" pod "4951f605-1e39-41f0-8d0a-2b23791653dd" (UID: "4951f605-1e39-41f0-8d0a-2b23791653dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.966048 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-workload-certs" pod "4951f605-1e39-41f0-8d0a-2b23791653dd" (UID: "4951f605-1e39-41f0-8d0a-2b23791653dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.966306 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4951f605-1e39-41f0-8d0a-2b23791653dd-istiod-ca-cert" pod "4951f605-1e39-41f0-8d0a-2b23791653dd" (UID: "4951f605-1e39-41f0-8d0a-2b23791653dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.966531 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-envoy" pod "4951f605-1e39-41f0-8d0a-2b23791653dd" (UID: "4951f605-1e39-41f0-8d0a-2b23791653dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.966882 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.966903 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.966919 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4951f605-1e39-41f0-8d0a-2b23791653dd-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.966932 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.966942 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.966954 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.966967 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.966980 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.966996 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zdht\" (UniqueName: \"kubernetes.io/projected/80a5af6e-f20f-4745-83d3-fd5df7661cc3-kube-api-access-9zdht\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.967009 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.967021 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.967033 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.967044 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.967055 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.968667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.967068 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a5af6e-f20f-4745-83d3-fd5df7661cc3-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.972634 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.972606 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4951f605-1e39-41f0-8d0a-2b23791653dd-kube-api-access-7d5lm" pod "4951f605-1e39-41f0-8d0a-2b23791653dd" (UID: "4951f605-1e39-41f0-8d0a-2b23791653dd"). InnerVolumeSpecName "kube-api-access-7d5lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:40.973146 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:40.973129 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-token" pod "4951f605-1e39-41f0-8d0a-2b23791653dd" (UID: "4951f605-1e39-41f0-8d0a-2b23791653dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:41.071552 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.071509 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-data\") pod \"e8bd28b1-844d-4b4f-bed7-542915f85c47\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " Oct 30 16:47:41.072174 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.072040 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-data" pod "e8bd28b1-844d-4b4f-bed7-542915f85c47" (UID: "e8bd28b1-844d-4b4f-bed7-542915f85c47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:41.072371 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.072354 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8bd28b1-844d-4b4f-bed7-542915f85c47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bd28b1-844d-4b4f-bed7-542915f85c47-istiod-ca-cert\") pod \"e8bd28b1-844d-4b4f-bed7-542915f85c47\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " Oct 30 16:47:41.072981 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.072864 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8bd28b1-844d-4b4f-bed7-542915f85c47-istiod-ca-cert" pod "e8bd28b1-844d-4b4f-bed7-542915f85c47" (UID: "e8bd28b1-844d-4b4f-bed7-542915f85c47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:41.073147 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.073130 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-credential-socket\") pod \"e8bd28b1-844d-4b4f-bed7-542915f85c47\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " Oct 30 16:47:41.073656 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.073541 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-credential-socket" pod "e8bd28b1-844d-4b4f-bed7-542915f85c47" (UID: "e8bd28b1-844d-4b4f-bed7-542915f85c47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:41.074099 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.074082 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-token\" (UniqueName: \"kubernetes.io/projected/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-token\") pod \"e8bd28b1-844d-4b4f-bed7-542915f85c47\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " Oct 30 16:47:41.074291 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.074276 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-envoy\") pod \"e8bd28b1-844d-4b4f-bed7-542915f85c47\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " Oct 30 16:47:41.075077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.075059 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-podinfo\") pod \"e8bd28b1-844d-4b4f-bed7-542915f85c47\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " Oct 30 16:47:41.075374 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.075358 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-workload-socket\") pod \"e8bd28b1-844d-4b4f-bed7-542915f85c47\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " Oct 30 16:47:41.075810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.075718 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-workload-socket" pod "e8bd28b1-844d-4b4f-bed7-542915f85c47" (UID: "e8bd28b1-844d-4b4f-bed7-542915f85c47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:41.076006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.075991 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-workload-certs\") pod \"e8bd28b1-844d-4b4f-bed7-542915f85c47\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " Oct 30 16:47:41.076680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.076418 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-workload-certs" pod "e8bd28b1-844d-4b4f-bed7-542915f85c47" (UID: "e8bd28b1-844d-4b4f-bed7-542915f85c47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:41.076680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.076555 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-token" pod "e8bd28b1-844d-4b4f-bed7-542915f85c47" (UID: "e8bd28b1-844d-4b4f-bed7-542915f85c47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:41.076912 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.076895 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bd28b1-844d-4b4f-bed7-542915f85c47-kube-api-access-qs24n\" (UniqueName: \"kubernetes.io/projected/e8bd28b1-844d-4b4f-bed7-542915f85c47-kube-api-access-qs24n\") pod \"e8bd28b1-844d-4b4f-bed7-542915f85c47\" (UID: \"e8bd28b1-844d-4b4f-bed7-542915f85c47\") " Oct 30 16:47:41.077821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.077797 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-istio-envoy\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.078084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.078066 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f65a98a-1301-4d39-a40f-df19ec451712-istio-podinfo\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.078266 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.078251 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-workload-socket\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.078432 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.078417 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f65a98a-1301-4d39-a40f-df19ec451712-istio-token\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.078631 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.078614 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blmth\" (UniqueName: \"kubernetes.io/projected/1f65a98a-1301-4d39-a40f-df19ec451712-kube-api-access-blmth\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.078939 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.078921 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-credential-socket\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.079153 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.079137 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-istio-data\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.079496 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.079389 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bd28b1-844d-4b4f-bed7-542915f85c47-kube-api-access-qs24n" pod "e8bd28b1-844d-4b4f-bed7-542915f85c47" (UID: "e8bd28b1-844d-4b4f-bed7-542915f85c47"). InnerVolumeSpecName "kube-api-access-qs24n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:41.079650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.079632 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-workload-certs\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.079852 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.079835 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f65a98a-1301-4d39-a40f-df19ec451712-istiod-ca-cert\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.080162 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.080147 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:41.080280 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.080266 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4951f605-1e39-41f0-8d0a-2b23791653dd-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:41.080358 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.080347 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:41.080444 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.080434 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bd28b1-844d-4b4f-bed7-542915f85c47-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:41.080525 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.080515 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:41.080605 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.080594 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:41.080684 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.080675 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:41.080846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.080750 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:41.080953 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.080938 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d5lm\" (UniqueName: \"kubernetes.io/projected/4951f605-1e39-41f0-8d0a-2b23791653dd-kube-api-access-7d5lm\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:41.081044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.081033 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs24n\" (UniqueName: \"kubernetes.io/projected/e8bd28b1-844d-4b4f-bed7-542915f85c47-kube-api-access-qs24n\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:41.093292 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.093251 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-envoy" pod "e8bd28b1-844d-4b4f-bed7-542915f85c47" (UID: "e8bd28b1-844d-4b4f-bed7-542915f85c47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:41.095122 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.095095 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-podinfo" pod "e8bd28b1-844d-4b4f-bed7-542915f85c47" (UID: "e8bd28b1-844d-4b4f-bed7-542915f85c47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:47:41.098616 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.098595 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:41.182962 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.182871 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f65a98a-1301-4d39-a40f-df19ec451712-istio-podinfo\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.182962 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.182923 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-workload-socket\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.182962 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.182960 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f65a98a-1301-4d39-a40f-df19ec451712-istio-token\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.183173 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.183016 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blmth\" (UniqueName: \"kubernetes.io/projected/1f65a98a-1301-4d39-a40f-df19ec451712-kube-api-access-blmth\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.183355 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.183331 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-credential-socket\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.183602 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.183575 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-istio-data\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.183919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.183612 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-credential-socket\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.183919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.183637 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-workload-certs\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.183919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.183411 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-workload-socket\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.183919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.183703 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f65a98a-1301-4d39-a40f-df19ec451712-istiod-ca-cert\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.183919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.183893 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-workload-certs\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.184236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.183931 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-istio-envoy\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.184236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.184068 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:41.184236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.184085 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bd28b1-844d-4b4f-bed7-542915f85c47-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:41.184236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.184110 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-istio-data\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.184449 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.184356 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f65a98a-1301-4d39-a40f-df19ec451712-istiod-ca-cert\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.185889 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.185866 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f65a98a-1301-4d39-a40f-df19ec451712-istio-podinfo\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.186565 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.186545 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-istio-envoy\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.209446 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.209419 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f65a98a-1301-4d39-a40f-df19ec451712-istio-token\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.210797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.210779 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blmth\" (UniqueName: \"kubernetes.io/projected/1f65a98a-1301-4d39-a40f-df19ec451712-kube-api-access-blmth\") pod \"e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.227243 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.227208 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:41.230816 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.230795 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt"] Oct 30 16:47:41.231637 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.231622 6202 cpu_manager.go:414] "containerMap: removing container" podUID="e8bd28b1-844d-4b4f-bed7-542915f85c47" containerName="istio-proxy" Oct 30 16:47:41.231681 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.231639 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8bd28b1-844d-4b4f-bed7-542915f85c47" containerName="istio-proxy" Oct 30 16:47:41.231788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.231776 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8bd28b1-844d-4b4f-bed7-542915f85c47" containerName="istio-proxy" Oct 30 16:47:41.235584 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.235570 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt" Oct 30 16:47:41.237303 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.237285 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:41.257599 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.257566 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt"] Oct 30 16:47:41.259991 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.259960 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ca5971\"" Oct 30 16:47:41.260194 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.260177 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ca5971-dockercfg-8t4b9\"" Oct 30 16:47:41.367799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.365804 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qtw6\" (UniqueName: \"kubernetes.io/projected/20ddef6a-15ca-42e0-960f-bff2270c35e6-kube-api-access-2qtw6\") pod \"payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt\" (UID: \"20ddef6a-15ca-42e0-960f-bff2270c35e6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt" Oct 30 16:47:41.367799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.366181 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ddef6a-15ca-42e0-960f-bff2270c35e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt\" (UID: \"20ddef6a-15ca-42e0-960f-bff2270c35e6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt" Oct 30 16:47:41.497327 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.497232 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ddef6a-15ca-42e0-960f-bff2270c35e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt\" (UID: \"20ddef6a-15ca-42e0-960f-bff2270c35e6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt" Oct 30 16:47:41.497505 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.497343 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qtw6\" (UniqueName: \"kubernetes.io/projected/20ddef6a-15ca-42e0-960f-bff2270c35e6-kube-api-access-2qtw6\") pod \"payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt\" (UID: \"20ddef6a-15ca-42e0-960f-bff2270c35e6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt" Oct 30 16:47:41.518676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.498436 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ddef6a-15ca-42e0-960f-bff2270c35e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt\" (UID: \"20ddef6a-15ca-42e0-960f-bff2270c35e6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt" Oct 30 16:47:41.518676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.499174 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:41.651627 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.651592 6202 generic.go:356] "Generic (PLEG): container finished" podID="4951f605-1e39-41f0-8d0a-2b23791653dd" containerID="3a3887b53d5fade58db199dcd2945e18802feea2160f6b4ace28bf945edca57d" exitCode=0 Oct 30 16:47:41.652266 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.651962 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" event={"ID":"4951f605-1e39-41f0-8d0a-2b23791653dd","Type":"ContainerDied","Data":"3a3887b53d5fade58db199dcd2945e18802feea2160f6b4ace28bf945edca57d"} Oct 30 16:47:41.652418 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.652299 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" event={"ID":"4951f605-1e39-41f0-8d0a-2b23791653dd","Type":"ContainerDied","Data":"edb430413b4f6d10a8cc4bf625d558978e7734064b74c362c4be8dd7a57495a4"} Oct 30 16:47:41.652418 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.652339 6202 scope.go:122] "RemoveContainer" containerID="3a3887b53d5fade58db199dcd2945e18802feea2160f6b4ace28bf945edca57d" Oct 30 16:47:41.652549 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.652003 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp" Oct 30 16:47:41.664341 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.663963 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl"] Oct 30 16:47:41.694034 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.694002 6202 generic.go:356] "Generic (PLEG): container finished" podID="80a5af6e-f20f-4745-83d3-fd5df7661cc3" containerID="c9d2dd04ff1419c63bdaeab4082fabb27a4ac267999a9a240987d8b00f65d8d8" exitCode=0 Oct 30 16:47:41.694495 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.694476 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" Oct 30 16:47:41.708480 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.708449 6202 generic.go:356] "Generic (PLEG): container finished" podID="e8bd28b1-844d-4b4f-bed7-542915f85c47" containerID="491f6cb6efb758dd1b31f9909e957fa419b32c053ee233695b38cc2ebf184f8e" exitCode=0 Oct 30 16:47:41.708817 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.708745 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" Oct 30 16:47:41.725556 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.725526 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" event={"ID":"80a5af6e-f20f-4745-83d3-fd5df7661cc3","Type":"ContainerDied","Data":"c9d2dd04ff1419c63bdaeab4082fabb27a4ac267999a9a240987d8b00f65d8d8"} Oct 30 16:47:41.725745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.725680 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl" Oct 30 16:47:41.725967 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.725747 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg" event={"ID":"80a5af6e-f20f-4745-83d3-fd5df7661cc3","Type":"ContainerDied","Data":"de3c101c1a1919aa9b7d04091c643230cdc5fdc623f8bf04a9624013e5ddd31b"} Oct 30 16:47:41.726127 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.726095 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" event={"ID":"9f4d8936-b54c-43f0-acfb-6edc1871e260","Type":"ContainerStarted","Data":"3880465c88ec60d0a7262a495135080e159ca9ed7d1a8eac4f2a040184997b06"} Oct 30 16:47:41.730313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.727560 6202 scope.go:122] "RemoveContainer" containerID="3a3887b53d5fade58db199dcd2945e18802feea2160f6b4ace28bf945edca57d" Oct 30 16:47:41.730313 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:47:41.727986 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a3887b53d5fade58db199dcd2945e18802feea2160f6b4ace28bf945edca57d\": container with ID starting with 3a3887b53d5fade58db199dcd2945e18802feea2160f6b4ace28bf945edca57d not found: ID does not exist" containerID="3a3887b53d5fade58db199dcd2945e18802feea2160f6b4ace28bf945edca57d" Oct 30 16:47:41.730313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.728022 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a3887b53d5fade58db199dcd2945e18802feea2160f6b4ace28bf945edca57d"} err="failed to get container status \"3a3887b53d5fade58db199dcd2945e18802feea2160f6b4ace28bf945edca57d\": rpc error: code = NotFound desc = could not find container \"3a3887b53d5fade58db199dcd2945e18802feea2160f6b4ace28bf945edca57d\": container with ID starting with 3a3887b53d5fade58db199dcd2945e18802feea2160f6b4ace28bf945edca57d not found: ID does not exist" Oct 30 16:47:41.730313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.728044 6202 scope.go:122] "RemoveContainer" containerID="c9d2dd04ff1419c63bdaeab4082fabb27a4ac267999a9a240987d8b00f65d8d8" Oct 30 16:47:41.731226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.726275 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" event={"ID":"e8bd28b1-844d-4b4f-bed7-542915f85c47","Type":"ContainerDied","Data":"491f6cb6efb758dd1b31f9909e957fa419b32c053ee233695b38cc2ebf184f8e"} Oct 30 16:47:41.731398 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.731382 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg" event={"ID":"e8bd28b1-844d-4b4f-bed7-542915f85c47","Type":"ContainerDied","Data":"56ebd316f465301d95789716ea22e8ce211f5b7fbd011103bff8d60a2350490e"} Oct 30 16:47:41.735062 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.732247 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc27f86-485d-4aed-9aa0-5b5f064bb50d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl\" (UID: \"4dc27f86-485d-4aed-9aa0-5b5f064bb50d\") " pod="openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl" Oct 30 16:47:41.735062 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.732281 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4ffm\" (UniqueName: \"kubernetes.io/projected/4dc27f86-485d-4aed-9aa0-5b5f064bb50d-kube-api-access-h4ffm\") pod \"payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl\" (UID: \"4dc27f86-485d-4aed-9aa0-5b5f064bb50d\") " pod="openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl" Oct 30 16:47:41.746436 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.746409 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl"] Oct 30 16:47:41.761698 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.752632 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qtw6\" (UniqueName: \"kubernetes.io/projected/20ddef6a-15ca-42e0-960f-bff2270c35e6-kube-api-access-2qtw6\") pod \"payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt\" (UID: \"20ddef6a-15ca-42e0-960f-bff2270c35e6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt" Oct 30 16:47:41.842831 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.841941 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc27f86-485d-4aed-9aa0-5b5f064bb50d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl\" (UID: \"4dc27f86-485d-4aed-9aa0-5b5f064bb50d\") " pod="openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl" Oct 30 16:47:41.842831 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.841989 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4ffm\" (UniqueName: \"kubernetes.io/projected/4dc27f86-485d-4aed-9aa0-5b5f064bb50d-kube-api-access-h4ffm\") pod \"payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl\" (UID: \"4dc27f86-485d-4aed-9aa0-5b5f064bb50d\") " pod="openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl" Oct 30 16:47:41.844963 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.843839 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc27f86-485d-4aed-9aa0-5b5f064bb50d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl\" (UID: \"4dc27f86-485d-4aed-9aa0-5b5f064bb50d\") " pod="openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl" Oct 30 16:47:41.870376 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.870336 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt" Oct 30 16:47:41.876161 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.874896 6202 scope.go:122] "RemoveContainer" containerID="c9d2dd04ff1419c63bdaeab4082fabb27a4ac267999a9a240987d8b00f65d8d8" Oct 30 16:47:41.880913 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:47:41.879453 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d2dd04ff1419c63bdaeab4082fabb27a4ac267999a9a240987d8b00f65d8d8\": container with ID starting with c9d2dd04ff1419c63bdaeab4082fabb27a4ac267999a9a240987d8b00f65d8d8 not found: ID does not exist" containerID="c9d2dd04ff1419c63bdaeab4082fabb27a4ac267999a9a240987d8b00f65d8d8" Oct 30 16:47:41.880913 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.879496 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d2dd04ff1419c63bdaeab4082fabb27a4ac267999a9a240987d8b00f65d8d8"} err="failed to get container status \"c9d2dd04ff1419c63bdaeab4082fabb27a4ac267999a9a240987d8b00f65d8d8\": rpc error: code = NotFound desc = could not find container \"c9d2dd04ff1419c63bdaeab4082fabb27a4ac267999a9a240987d8b00f65d8d8\": container with ID starting with c9d2dd04ff1419c63bdaeab4082fabb27a4ac267999a9a240987d8b00f65d8d8 not found: ID does not exist" Oct 30 16:47:41.880913 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.879522 6202 scope.go:122] "RemoveContainer" containerID="491f6cb6efb758dd1b31f9909e957fa419b32c053ee233695b38cc2ebf184f8e" Oct 30 16:47:41.907331 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.907302 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4ffm\" (UniqueName: \"kubernetes.io/projected/4dc27f86-485d-4aed-9aa0-5b5f064bb50d-kube-api-access-h4ffm\") pod \"payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl\" (UID: \"4dc27f86-485d-4aed-9aa0-5b5f064bb50d\") " pod="openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl" Oct 30 16:47:41.915020 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.914989 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp"] Oct 30 16:47:41.916947 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.916926 6202 scope.go:122] "RemoveContainer" containerID="491f6cb6efb758dd1b31f9909e957fa419b32c053ee233695b38cc2ebf184f8e" Oct 30 16:47:41.917365 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:47:41.917336 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491f6cb6efb758dd1b31f9909e957fa419b32c053ee233695b38cc2ebf184f8e\": container with ID starting with 491f6cb6efb758dd1b31f9909e957fa419b32c053ee233695b38cc2ebf184f8e not found: ID does not exist" containerID="491f6cb6efb758dd1b31f9909e957fa419b32c053ee233695b38cc2ebf184f8e" Oct 30 16:47:41.917443 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.917379 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491f6cb6efb758dd1b31f9909e957fa419b32c053ee233695b38cc2ebf184f8e"} err="failed to get container status \"491f6cb6efb758dd1b31f9909e957fa419b32c053ee233695b38cc2ebf184f8e\": rpc error: code = NotFound desc = could not find container \"491f6cb6efb758dd1b31f9909e957fa419b32c053ee233695b38cc2ebf184f8e\": container with ID starting with 491f6cb6efb758dd1b31f9909e957fa419b32c053ee233695b38cc2ebf184f8e not found: ID does not exist" Oct 30 16:47:41.920387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.920367 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-bmrxp"] Oct 30 16:47:41.995796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:41.995671 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd"] Oct 30 16:47:42.009501 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.008506 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:47:42.009501 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.008599 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:47:42.009501 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.008641 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:47:42.030921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.022560 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg"] Oct 30 16:47:42.030921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.029922 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-trhsg"] Oct 30 16:47:42.102220 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.102173 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg"] Oct 30 16:47:42.105803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.105484 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-wh8bg"] Oct 30 16:47:42.136836 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.136814 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl" Oct 30 16:47:42.310262 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.309995 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt"] Oct 30 16:47:42.418444 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.418413 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4951f605-1e39-41f0-8d0a-2b23791653dd" path="/var/lib/kubelet/pods/4951f605-1e39-41f0-8d0a-2b23791653dd/volumes" Oct 30 16:47:42.422840 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.422815 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80a5af6e-f20f-4745-83d3-fd5df7661cc3" path="/var/lib/kubelet/pods/80a5af6e-f20f-4745-83d3-fd5df7661cc3/volumes" Oct 30 16:47:42.423688 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.423669 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8bd28b1-844d-4b4f-bed7-542915f85c47" path="/var/lib/kubelet/pods/e8bd28b1-844d-4b4f-bed7-542915f85c47/volumes" Oct 30 16:47:42.535707 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:47:42.535647 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc27f86_485d_4aed_9aa0_5b5f064bb50d.slice/crio-68470a23e8fd08611fe1c6e1f49a00985d7fa8a2068f6769db98471f7e5665ca WatchSource:0}: Error finding container 68470a23e8fd08611fe1c6e1f49a00985d7fa8a2068f6769db98471f7e5665ca: Status 404 returned error can't find the container with id 68470a23e8fd08611fe1c6e1f49a00985d7fa8a2068f6769db98471f7e5665ca Oct 30 16:47:42.545795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.538740 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl"] Oct 30 16:47:42.662519 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.662493 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:47:42.662863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.662841 6202 patch_prober.go:29] interesting pod/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 16:47:42.663019 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.662996 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" podUID="9f4d8936-b54c-43f0-acfb-6edc1871e260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 16:47:42.718999 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.718970 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl" event={"ID":"4dc27f86-485d-4aed-9aa0-5b5f064bb50d","Type":"ContainerStarted","Data":"b93ae686814da77860e01b1f8d8ae574045b40844140a471fa732f95a1f84dbd"} Oct 30 16:47:42.719148 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.719029 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl" event={"ID":"4dc27f86-485d-4aed-9aa0-5b5f064bb50d","Type":"ContainerStarted","Data":"68470a23e8fd08611fe1c6e1f49a00985d7fa8a2068f6769db98471f7e5665ca"} Oct 30 16:47:42.719148 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.719118 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl" Oct 30 16:47:42.722952 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.722927 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt" event={"ID":"20ddef6a-15ca-42e0-960f-bff2270c35e6","Type":"ContainerStarted","Data":"3ba0b4156aa7913ef9d958d1abbd4e7e31a42fe2b7f385662bc4f210a8216037"} Oct 30 16:47:42.723066 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.722976 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt" Oct 30 16:47:42.723066 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.723000 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt" event={"ID":"20ddef6a-15ca-42e0-960f-bff2270c35e6","Type":"ContainerStarted","Data":"2298cc2d5c42c447470fbddd005f1ff95b743640f03f8782dfd00c9a663ca0fe"} Oct 30 16:47:42.726841 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.726817 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" event={"ID":"1f65a98a-1301-4d39-a40f-df19ec451712","Type":"ContainerStarted","Data":"805947c6ef9ee84b32d05c60294d386532b0a2401cdd92a499ee8a53249cf625"} Oct 30 16:47:42.726930 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.726864 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" event={"ID":"1f65a98a-1301-4d39-a40f-df19ec451712","Type":"ContainerStarted","Data":"97cb6202b36412ce941bc91ab8440dbf9a15736d0c1458b08340c51d5503bc10"} Oct 30 16:47:42.815863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.815732 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl" podStartSLOduration=1.8157181 podStartE2EDuration="1.8157181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:42.774164395 +0000 UTC m=+1576.484619138" watchObservedRunningTime="2026-10-30 16:47:42.8157181 +0000 UTC m=+1576.526172844" Oct 30 16:47:42.818308 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.818265 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt" podStartSLOduration=1.818255868 podStartE2EDuration="1.818255868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:42.814263704 +0000 UTC m=+1576.524718449" watchObservedRunningTime="2026-10-30 16:47:42.818255868 +0000 UTC m=+1576.528710612" Oct 30 16:47:42.873927 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:42.864134 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" podStartSLOduration=2.864115868 podStartE2EDuration="2.864115868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:42.86120415 +0000 UTC m=+1576.571658897" watchObservedRunningTime="2026-10-30 16:47:42.864115868 +0000 UTC m=+1576.574570613" Oct 30 16:47:43.237490 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.237462 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:43.263093 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.263062 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:43.314751 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.314720 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:43.674907 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.674867 6202 patch_prober.go:29] interesting pod/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 16:47:43.675075 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.674934 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" podUID="9f4d8936-b54c-43f0-acfb-6edc1871e260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 16:47:43.687660 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.687636 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" Oct 30 16:47:43.731608 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.731577 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/082aec51-71a4-48f7-994b-bd6cc7e05e49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/082aec51-71a4-48f7-994b-bd6cc7e05e49-plugins-config-volume\") pod \"082aec51-71a4-48f7-994b-bd6cc7e05e49\" (UID: \"082aec51-71a4-48f7-994b-bd6cc7e05e49\") " Oct 30 16:47:43.731835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.731815 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/082aec51-71a4-48f7-994b-bd6cc7e05e49-kube-api-access-95b28\" (UniqueName: \"kubernetes.io/projected/082aec51-71a4-48f7-994b-bd6cc7e05e49-kube-api-access-95b28\") pod \"082aec51-71a4-48f7-994b-bd6cc7e05e49\" (UID: \"082aec51-71a4-48f7-994b-bd6cc7e05e49\") " Oct 30 16:47:43.732056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.731998 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082aec51-71a4-48f7-994b-bd6cc7e05e49-plugins-config-volume" pod "082aec51-71a4-48f7-994b-bd6cc7e05e49" (UID: "082aec51-71a4-48f7-994b-bd6cc7e05e49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:43.732429 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.732406 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:43.732820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.732796 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/082aec51-71a4-48f7-994b-bd6cc7e05e49-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:43.736815 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.736746 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/082aec51-71a4-48f7-994b-bd6cc7e05e49-kube-api-access-95b28" pod "082aec51-71a4-48f7-994b-bd6cc7e05e49" (UID: "082aec51-71a4-48f7-994b-bd6cc7e05e49"). InnerVolumeSpecName "kube-api-access-95b28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:43.782642 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.782488 6202 generic.go:356] "Generic (PLEG): container finished" podID="082aec51-71a4-48f7-994b-bd6cc7e05e49" containerID="13b333e57b4d0b695a71b2f305149001ba6a71e90d696c494a5aef8ea53a4b3b" exitCode=0 Oct 30 16:47:43.783608 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.783526 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" Oct 30 16:47:43.790163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.790046 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" event={"ID":"082aec51-71a4-48f7-994b-bd6cc7e05e49","Type":"ContainerDied","Data":"13b333e57b4d0b695a71b2f305149001ba6a71e90d696c494a5aef8ea53a4b3b"} Oct 30 16:47:43.790163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.790096 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk" event={"ID":"082aec51-71a4-48f7-994b-bd6cc7e05e49","Type":"ContainerDied","Data":"39275d4c7c62a4fd5d4c609cb4efb908164656c7ce096bec6911a1423f23d003"} Oct 30 16:47:43.790163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.790119 6202 scope.go:122] "RemoveContainer" containerID="13b333e57b4d0b695a71b2f305149001ba6a71e90d696c494a5aef8ea53a4b3b" Oct 30 16:47:43.791201 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.791030 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:43.791867 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.791851 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:47:43.804719 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.804703 6202 scope.go:122] "RemoveContainer" containerID="13b333e57b4d0b695a71b2f305149001ba6a71e90d696c494a5aef8ea53a4b3b" Oct 30 16:47:43.805021 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:47:43.804997 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b333e57b4d0b695a71b2f305149001ba6a71e90d696c494a5aef8ea53a4b3b\": container with ID starting with 13b333e57b4d0b695a71b2f305149001ba6a71e90d696c494a5aef8ea53a4b3b not found: ID does not exist" containerID="13b333e57b4d0b695a71b2f305149001ba6a71e90d696c494a5aef8ea53a4b3b" Oct 30 16:47:43.805081 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.805028 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b333e57b4d0b695a71b2f305149001ba6a71e90d696c494a5aef8ea53a4b3b"} err="failed to get container status \"13b333e57b4d0b695a71b2f305149001ba6a71e90d696c494a5aef8ea53a4b3b\": rpc error: code = NotFound desc = could not find container \"13b333e57b4d0b695a71b2f305149001ba6a71e90d696c494a5aef8ea53a4b3b\": container with ID starting with 13b333e57b4d0b695a71b2f305149001ba6a71e90d696c494a5aef8ea53a4b3b not found: ID does not exist" Oct 30 16:47:43.834516 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.834489 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95b28\" (UniqueName: \"kubernetes.io/projected/082aec51-71a4-48f7-994b-bd6cc7e05e49-kube-api-access-95b28\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:47:43.879098 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.879065 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk"] Oct 30 16:47:43.885438 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:43.885411 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-w65jk"] Oct 30 16:47:44.107814 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:47:44.107742 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:47:44.421260 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:44.421179 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="082aec51-71a4-48f7-994b-bd6cc7e05e49" path="/var/lib/kubelet/pods/082aec51-71a4-48f7-994b-bd6cc7e05e49/volumes" Oct 30 16:47:44.664331 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:44.663854 6202 patch_prober.go:29] interesting pod/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 16:47:44.664331 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:44.663916 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" podUID="9f4d8936-b54c-43f0-acfb-6edc1871e260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 16:47:45.076333 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:45.076288 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:45.659918 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:45.659884 6202 patch_prober.go:29] interesting pod/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 16:47:45.660225 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:45.660195 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" podUID="9f4d8936-b54c-43f0-acfb-6edc1871e260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 16:47:46.035261 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:46.035180 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52"] Oct 30 16:47:46.070882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:46.070845 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f747af2-c7k52"] Oct 30 16:47:46.413090 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:46.413059 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60339ae6-3e91-4370-9ea1-a7542d77486b" path="/var/lib/kubelet/pods/60339ae6-3e91-4370-9ea1-a7542d77486b/volumes" Oct 30 16:47:46.669852 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:46.658880 6202 patch_prober.go:29] interesting pod/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 16:47:46.669852 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:46.658947 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" podUID="9f4d8936-b54c-43f0-acfb-6edc1871e260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 16:47:47.471792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:47.470951 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:47.764552 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:47.764139 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:47:47.765518 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:47.765498 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:47:47.766539 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:47.766500 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:47:49.571793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:49.570855 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:47:49.932491 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:49.932378 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:50.042910 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:50.042858 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:47:50.102897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:50.102851 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:47:50.104254 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:50.104224 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 16:47:50.219837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:50.218842 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:47:51.728675 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:51.728638 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:53.415888 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:53.415849 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:53.787573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:53.787484 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl" Oct 30 16:47:53.793882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:53.793858 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt" Oct 30 16:47:55.089955 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:55.089922 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:55.402871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:55.402790 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:55.622006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:55.621966 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:55.851591 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:55.851548 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:56.902791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:56.902329 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:57.044532 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.044429 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:57.190838 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.190685 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:57.324025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.323977 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:57.355197 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.355165 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn"] Oct 30 16:47:57.356454 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.356434 6202 cpu_manager.go:414] "containerMap: removing container" podUID="082aec51-71a4-48f7-994b-bd6cc7e05e49" containerName="payload-pre-processing" Oct 30 16:47:57.356522 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.356458 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="082aec51-71a4-48f7-994b-bd6cc7e05e49" containerName="payload-pre-processing" Oct 30 16:47:57.356705 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.356693 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="082aec51-71a4-48f7-994b-bd6cc7e05e49" containerName="payload-pre-processing" Oct 30 16:47:57.361120 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.361101 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" Oct 30 16:47:57.379888 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.379861 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn"] Oct 30 16:47:57.385298 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.385267 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-529bc7-dockercfg-v2gs5\"" Oct 30 16:47:57.389910 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.389874 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-529bc7\"" Oct 30 16:47:57.421311 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.421269 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcea868-43f0-4503-9f06-04bb9db08e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn\" (UID: \"1dcea868-43f0-4503-9f06-04bb9db08e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" Oct 30 16:47:57.421548 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.421529 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7lb9\" (UniqueName: \"kubernetes.io/projected/1dcea868-43f0-4503-9f06-04bb9db08e9b-kube-api-access-x7lb9\") pod \"payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn\" (UID: \"1dcea868-43f0-4503-9f06-04bb9db08e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" Oct 30 16:47:57.465523 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.465420 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw"] Oct 30 16:47:57.529817 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.528299 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcea868-43f0-4503-9f06-04bb9db08e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn\" (UID: \"1dcea868-43f0-4503-9f06-04bb9db08e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" Oct 30 16:47:57.529817 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.528383 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7lb9\" (UniqueName: \"kubernetes.io/projected/1dcea868-43f0-4503-9f06-04bb9db08e9b-kube-api-access-x7lb9\") pod \"payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn\" (UID: \"1dcea868-43f0-4503-9f06-04bb9db08e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" Oct 30 16:47:57.557780 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.557662 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcea868-43f0-4503-9f06-04bb9db08e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn\" (UID: \"1dcea868-43f0-4503-9f06-04bb9db08e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" Oct 30 16:47:57.561687 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.561653 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw"] Oct 30 16:47:57.561890 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.561804 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" Oct 30 16:47:57.563969 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.563936 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7lb9\" (UniqueName: \"kubernetes.io/projected/1dcea868-43f0-4503-9f06-04bb9db08e9b-kube-api-access-x7lb9\") pod \"payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn\" (UID: \"1dcea868-43f0-4503-9f06-04bb9db08e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" Oct 30 16:47:57.609279 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.608917 6202 patch_prober.go:29] interesting pod/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:47:57.609279 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.608977 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" podUID="8109a3b3-6567-49bc-b409-58309ac8bb62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:47:57.644221 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.641744 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4a192c-bbec-4ec7-98c8-378faf14b730-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw\" (UID: \"4e4a192c-bbec-4ec7-98c8-378faf14b730\") " pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" Oct 30 16:47:57.644221 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.642064 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm9v2\" (UniqueName: \"kubernetes.io/projected/4e4a192c-bbec-4ec7-98c8-378faf14b730-kube-api-access-fm9v2\") pod \"payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw\" (UID: \"4e4a192c-bbec-4ec7-98c8-378faf14b730\") " pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" Oct 30 16:47:57.692133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.692092 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" Oct 30 16:47:57.767382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.767149 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4a192c-bbec-4ec7-98c8-378faf14b730-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw\" (UID: \"4e4a192c-bbec-4ec7-98c8-378faf14b730\") " pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" Oct 30 16:47:57.778340 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.759903 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4a192c-bbec-4ec7-98c8-378faf14b730-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw\" (UID: \"4e4a192c-bbec-4ec7-98c8-378faf14b730\") " pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" Oct 30 16:47:57.778746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.778723 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm9v2\" (UniqueName: \"kubernetes.io/projected/4e4a192c-bbec-4ec7-98c8-378faf14b730-kube-api-access-fm9v2\") pod \"payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw\" (UID: \"4e4a192c-bbec-4ec7-98c8-378faf14b730\") " pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" Oct 30 16:47:57.817779 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:57.816325 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm9v2\" (UniqueName: \"kubernetes.io/projected/4e4a192c-bbec-4ec7-98c8-378faf14b730-kube-api-access-fm9v2\") pod \"payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw\" (UID: \"4e4a192c-bbec-4ec7-98c8-378faf14b730\") " pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" Oct 30 16:47:58.013554 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:58.013466 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" Oct 30 16:47:58.104797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:58.102140 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn"] Oct 30 16:47:58.349903 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:47:58.349866 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4a192c_bbec_4ec7_98c8_378faf14b730.slice/crio-3f419a2527c65ae9de14b29a7d9a27096e1a62904ea9eb0e082a78af1ef7e078 WatchSource:0}: Error finding container 3f419a2527c65ae9de14b29a7d9a27096e1a62904ea9eb0e082a78af1ef7e078: Status 404 returned error can't find the container with id 3f419a2527c65ae9de14b29a7d9a27096e1a62904ea9eb0e082a78af1ef7e078 Oct 30 16:47:58.350458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:58.350434 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw"] Oct 30 16:47:58.646904 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:58.646813 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:59.014715 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:59.014629 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:59.117019 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:59.113060 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" event={"ID":"4e4a192c-bbec-4ec7-98c8-378faf14b730","Type":"ContainerStarted","Data":"7e9d2f40a50c1f5ab0e3783f71432301bfb88d55447cc5427dccaa71058bd56a"} Oct 30 16:47:59.117019 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:59.113127 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" event={"ID":"4e4a192c-bbec-4ec7-98c8-378faf14b730","Type":"ContainerStarted","Data":"3f419a2527c65ae9de14b29a7d9a27096e1a62904ea9eb0e082a78af1ef7e078"} Oct 30 16:47:59.117019 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:59.114276 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" Oct 30 16:47:59.119672 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:59.119644 6202 generic.go:356] "Generic (PLEG): container finished" podID="8109a3b3-6567-49bc-b409-58309ac8bb62" containerID="5029ec767242166c92c33ab92d6d4013c2b092d235cdc5307e775c193f1a8b6f" exitCode=0 Oct 30 16:47:59.120798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:59.119804 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" event={"ID":"8109a3b3-6567-49bc-b409-58309ac8bb62","Type":"ContainerDied","Data":"5029ec767242166c92c33ab92d6d4013c2b092d235cdc5307e775c193f1a8b6f"} Oct 30 16:47:59.120798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:59.120462 6202 scope.go:122] "RemoveContainer" containerID="5029ec767242166c92c33ab92d6d4013c2b092d235cdc5307e775c193f1a8b6f" Oct 30 16:47:59.122887 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:59.122849 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" event={"ID":"1dcea868-43f0-4503-9f06-04bb9db08e9b","Type":"ContainerStarted","Data":"a5e9283318dd1a34b10715db8bd65156a41ca264f1de1b3a20bcf5516f24490c"} Oct 30 16:47:59.123006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:59.122900 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" event={"ID":"1dcea868-43f0-4503-9f06-04bb9db08e9b","Type":"ContainerStarted","Data":"ecf474154abd90e57e583c923f1bfe299fbb80ba2c68c4f3d1b27ed8e959ccba"} Oct 30 16:47:59.123006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:59.122940 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" Oct 30 16:47:59.125032 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:59.124997 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:47:59.125178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:59.125092 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:47:59.125178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:59.125144 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:47:59.165104 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:59.165023 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" podStartSLOduration=2.165004186 podStartE2EDuration="2.165004186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:59.159345496 +0000 UTC m=+1592.869800244" watchObservedRunningTime="2026-10-30 16:47:59.165004186 +0000 UTC m=+1592.875458933" Oct 30 16:47:59.199924 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:59.199880 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" podStartSLOduration=2.199865419 podStartE2EDuration="2.199865419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:59.199360586 +0000 UTC m=+1592.909815330" watchObservedRunningTime="2026-10-30 16:47:59.199865419 +0000 UTC m=+1592.910320204" Oct 30 16:47:59.327056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:59.327017 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:59.826263 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:47:59.826228 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:00.083876 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:00.077947 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:00.170133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:00.170078 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" event={"ID":"8109a3b3-6567-49bc-b409-58309ac8bb62","Type":"ContainerStarted","Data":"00e14e56c4aea6b822e7592e983f035c6a4efb8415eb95cd32014af78ddde82b"} Oct 30 16:48:00.631357 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:00.631321 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:48:00.632290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:00.632262 6202 patch_prober.go:29] interesting pod/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:48:00.632431 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:00.632319 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" podUID="8109a3b3-6567-49bc-b409-58309ac8bb62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 16:48:00.956354 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:00.956220 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:01.631787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:01.631738 6202 patch_prober.go:29] interesting pod/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:48:01.632266 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:01.631807 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" podUID="8109a3b3-6567-49bc-b409-58309ac8bb62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 16:48:02.631994 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:02.631962 6202 patch_prober.go:29] interesting pod/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:48:02.632444 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:02.632016 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" podUID="8109a3b3-6567-49bc-b409-58309ac8bb62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 16:48:02.871194 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:02.871139 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:03.631629 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:03.631590 6202 patch_prober.go:29] interesting pod/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:48:03.631843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:03.631654 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" podUID="8109a3b3-6567-49bc-b409-58309ac8bb62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 16:48:04.399912 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:04.399877 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:04.631401 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:04.631365 6202 patch_prober.go:29] interesting pod/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:48:04.631566 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:04.631415 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" podUID="8109a3b3-6567-49bc-b409-58309ac8bb62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 16:48:05.630991 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:05.630952 6202 patch_prober.go:29] interesting pod/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:48:05.631485 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:05.631000 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" podUID="8109a3b3-6567-49bc-b409-58309ac8bb62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 16:48:05.954975 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:05.954883 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:06.636750 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:06.636721 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:48:06.637217 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:06.637201 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:48:06.637886 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:06.637863 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:48:07.753205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:07.753170 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:09.314620 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:09.314578 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:10.174643 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:10.174612 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" Oct 30 16:48:11.190782 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:11.190722 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:11.207972 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:11.207943 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" Oct 30 16:48:13.069725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:13.069119 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:13.694343 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:13.694309 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:13.904583 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:13.904543 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm"] Oct 30 16:48:13.916830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:13.916795 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-vkshm"] Oct 30 16:48:13.917065 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:13.917045 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:14.122490 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:48:14.122450 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:48:14.414159 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:14.414085 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98c04758-eb28-4b22-bd68-ea1eebae6834" path="/var/lib/kubelet/pods/98c04758-eb28-4b22-bd68-ea1eebae6834/volumes" Oct 30 16:48:15.081336 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:15.081301 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:15.625447 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:15.625370 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:17.369859 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:17.369825 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:17.495356 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:17.495314 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:17.615004 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:17.614963 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:19.174044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:19.174009 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:20.758662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:20.758618 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:22.450232 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:22.450202 6202 generic.go:356] "Generic (PLEG): container finished" podID="dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" containerID="99c07b28fa71a9da8c59c9aa59e2b90745a025152b2d1d22e4ebce6fbcc070f7" exitCode=0 Oct 30 16:48:22.450705 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:22.450280 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" event={"ID":"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b","Type":"ContainerDied","Data":"99c07b28fa71a9da8c59c9aa59e2b90745a025152b2d1d22e4ebce6fbcc070f7"} Oct 30 16:48:22.450832 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:22.450814 6202 scope.go:122] "RemoveContainer" containerID="99c07b28fa71a9da8c59c9aa59e2b90745a025152b2d1d22e4ebce6fbcc070f7" Oct 30 16:48:22.452786 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:22.452729 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:48:22.452885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:22.452842 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:48:22.452944 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:22.452882 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:48:22.462380 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:22.462357 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:22.725843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:22.724466 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:22.905794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:22.902572 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:48:23.480115 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:23.480077 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" event={"ID":"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b","Type":"ContainerStarted","Data":"63a1cf4ca3660b65bd553fb6a36a4f490b5a3b30966b4f61f869f0dde5fb7cc4"} Oct 30 16:48:23.865550 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:23.864201 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:48:23.874793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:23.867876 6202 patch_prober.go:29] interesting pod/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:48:23.874793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:23.867980 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" podUID="dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 16:48:24.123985 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:24.123886 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:24.866803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:24.866186 6202 patch_prober.go:29] interesting pod/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:48:24.866803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:24.866253 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" podUID="dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 16:48:25.032211 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:25.032169 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc"] Oct 30 16:48:25.074903 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:25.074861 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6e7ef6-9j7tc"] Oct 30 16:48:25.670466 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:25.670423 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:25.868406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:25.868367 6202 patch_prober.go:29] interesting pod/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:48:25.880821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:25.880784 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" podUID="dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 16:48:26.413425 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:26.413393 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84dc0c14-6e5e-4c72-90d2-db9ca7d42711" path="/var/lib/kubelet/pods/84dc0c14-6e5e-4c72-90d2-db9ca7d42711/volumes" Oct 30 16:48:26.513501 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:26.513465 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:26.864639 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:26.864171 6202 patch_prober.go:29] interesting pod/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:48:26.864639 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:26.864239 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" podUID="dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 16:48:27.370687 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:27.370653 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:27.867964 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:27.867917 6202 patch_prober.go:29] interesting pod/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:48:27.868274 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:27.868240 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" podUID="dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 16:48:28.863926 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:28.863866 6202 patch_prober.go:29] interesting pod/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:48:28.864448 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:28.863931 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" podUID="dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 16:48:28.903922 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:28.903876 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:29.868858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:29.868825 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:48:29.869359 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:29.869209 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:48:29.870100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:29.870066 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:48:30.405083 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:30.405025 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:30.786152 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:30.786069 6202 scope.go:122] "RemoveContainer" containerID="bea80a946f03ce22338c17d69bfabaa053bdd828fca4ec934cc0adfde6f95ca4" Oct 30 16:48:30.797631 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:30.797610 6202 scope.go:122] "RemoveContainer" containerID="677fbbf12c35a66f1612c9310a6246ad838c755b6bf403c7b12d7812cdc10039" Oct 30 16:48:30.807992 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:30.807959 6202 scope.go:122] "RemoveContainer" containerID="23f462464cd1994eadf9e94f43b20228dc12cfd18d66481de09abf9a948ff015" Oct 30 16:48:30.818350 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:30.818329 6202 scope.go:122] "RemoveContainer" containerID="1c64539f30147156f9f4d83670100659ebce0eae0423adb486ed74be2468faee" Oct 30 16:48:30.829121 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:30.829101 6202 scope.go:122] "RemoveContainer" containerID="12b1915199cfbcb372a889fd514085107177b2d2d3d10f0e1c40e6a160871007" Oct 30 16:48:30.839005 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:30.838984 6202 scope.go:122] "RemoveContainer" containerID="079d0b337c49150fe916672a6e70a18b0fe42e27afff84f4c68761cff665bbac" Oct 30 16:48:30.848696 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:30.848668 6202 scope.go:122] "RemoveContainer" containerID="00ba40e387d45078730fea9b84522c0e099644976b55d1a2295b1b609ca2a3fd" Oct 30 16:48:32.044454 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:32.044415 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:32.173719 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:32.173597 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:34.082083 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:34.082044 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:34.209047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:34.208975 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:35.031525 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:35.031491 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k"] Oct 30 16:48:35.043940 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:35.043904 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63cc690-xl49k"] Oct 30 16:48:35.690873 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:35.690709 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:35.794409 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:35.794362 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md"] Oct 30 16:48:35.805566 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:35.805535 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md" Oct 30 16:48:35.815858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:35.815832 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md"] Oct 30 16:48:35.821121 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:35.821098 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:35.822936 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:35.822917 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-407eab\"" Oct 30 16:48:35.823029 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:35.822993 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-407eab-dockercfg-nzk6w\"" Oct 30 16:48:35.858729 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:35.858696 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz"] Oct 30 16:48:35.864822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:35.863976 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz" Oct 30 16:48:35.902229 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:35.895620 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz"] Oct 30 16:48:35.907745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:35.907714 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc9c129-4290-4f6b-b02f-e59979426e5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz\" (UID: \"efc9c129-4290-4f6b-b02f-e59979426e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz" Oct 30 16:48:35.907987 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:35.907903 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6215914d-d1a2-45e0-a811-469d418061df-plugins-config-volume\") pod \"payload-processing-e2e-api-b-407eab-66f8cb4786-g92md\" (UID: \"6215914d-d1a2-45e0-a811-469d418061df\") " pod="openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md" Oct 30 16:48:35.908110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:35.908085 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqz8q\" (UniqueName: \"kubernetes.io/projected/6215914d-d1a2-45e0-a811-469d418061df-kube-api-access-vqz8q\") pod \"payload-processing-e2e-api-b-407eab-66f8cb4786-g92md\" (UID: \"6215914d-d1a2-45e0-a811-469d418061df\") " pod="openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md" Oct 30 16:48:35.908304 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:35.908221 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfm94\" (UniqueName: \"kubernetes.io/projected/efc9c129-4290-4f6b-b02f-e59979426e5a-kube-api-access-qfm94\") pod \"payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz\" (UID: \"efc9c129-4290-4f6b-b02f-e59979426e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz" Oct 30 16:48:35.947929 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:35.947823 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:36.012853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:36.012812 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6215914d-d1a2-45e0-a811-469d418061df-plugins-config-volume\") pod \"payload-processing-e2e-api-b-407eab-66f8cb4786-g92md\" (UID: \"6215914d-d1a2-45e0-a811-469d418061df\") " pod="openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md" Oct 30 16:48:36.030949 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:36.013973 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6215914d-d1a2-45e0-a811-469d418061df-plugins-config-volume\") pod \"payload-processing-e2e-api-b-407eab-66f8cb4786-g92md\" (UID: \"6215914d-d1a2-45e0-a811-469d418061df\") " pod="openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md" Oct 30 16:48:36.031326 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:36.031304 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqz8q\" (UniqueName: \"kubernetes.io/projected/6215914d-d1a2-45e0-a811-469d418061df-kube-api-access-vqz8q\") pod \"payload-processing-e2e-api-b-407eab-66f8cb4786-g92md\" (UID: \"6215914d-d1a2-45e0-a811-469d418061df\") " pod="openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md" Oct 30 16:48:36.031825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:36.031803 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfm94\" (UniqueName: \"kubernetes.io/projected/efc9c129-4290-4f6b-b02f-e59979426e5a-kube-api-access-qfm94\") pod \"payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz\" (UID: \"efc9c129-4290-4f6b-b02f-e59979426e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz" Oct 30 16:48:36.032395 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:36.032374 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc9c129-4290-4f6b-b02f-e59979426e5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz\" (UID: \"efc9c129-4290-4f6b-b02f-e59979426e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz" Oct 30 16:48:36.033393 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:36.033279 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc9c129-4290-4f6b-b02f-e59979426e5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz\" (UID: \"efc9c129-4290-4f6b-b02f-e59979426e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz" Oct 30 16:48:36.072464 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:36.072429 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfm94\" (UniqueName: \"kubernetes.io/projected/efc9c129-4290-4f6b-b02f-e59979426e5a-kube-api-access-qfm94\") pod \"payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz\" (UID: \"efc9c129-4290-4f6b-b02f-e59979426e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz" Oct 30 16:48:36.090178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:36.088792 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqz8q\" (UniqueName: \"kubernetes.io/projected/6215914d-d1a2-45e0-a811-469d418061df-kube-api-access-vqz8q\") pod \"payload-processing-e2e-api-b-407eab-66f8cb4786-g92md\" (UID: \"6215914d-d1a2-45e0-a811-469d418061df\") " pod="openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md" Oct 30 16:48:36.137089 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:36.137055 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md" Oct 30 16:48:36.205157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:36.205063 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz" Oct 30 16:48:36.417957 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:36.417926 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="294ae9a7-cf8b-46c4-abd3-77b7f2254b0a" path="/var/lib/kubelet/pods/294ae9a7-cf8b-46c4-abd3-77b7f2254b0a/volumes" Oct 30 16:48:36.581222 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:36.581193 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md"] Oct 30 16:48:36.583863 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:48:36.583827 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6215914d_d1a2_45e0_a811_469d418061df.slice/crio-c7c75dba8d28a062ddb5ac9ee6752f8c962d39030c3fb136004539a4071f41ad WatchSource:0}: Error finding container c7c75dba8d28a062ddb5ac9ee6752f8c962d39030c3fb136004539a4071f41ad: Status 404 returned error can't find the container with id c7c75dba8d28a062ddb5ac9ee6752f8c962d39030c3fb136004539a4071f41ad Oct 30 16:48:36.621424 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:36.621400 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz"] Oct 30 16:48:36.623225 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:48:36.623201 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc9c129_4290_4f6b_b02f_e59979426e5a.slice/crio-c3f38daa6ebf91e07ff9da72968eb99611d964bb01f63af9c26dce52ab7cdb9d WatchSource:0}: Error finding container c3f38daa6ebf91e07ff9da72968eb99611d964bb01f63af9c26dce52ab7cdb9d: Status 404 returned error can't find the container with id c3f38daa6ebf91e07ff9da72968eb99611d964bb01f63af9c26dce52ab7cdb9d Oct 30 16:48:36.657423 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:36.657372 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md" event={"ID":"6215914d-d1a2-45e0-a811-469d418061df","Type":"ContainerStarted","Data":"c7c75dba8d28a062ddb5ac9ee6752f8c962d39030c3fb136004539a4071f41ad"} Oct 30 16:48:36.659672 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:36.659644 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz" event={"ID":"efc9c129-4290-4f6b-b02f-e59979426e5a","Type":"ContainerStarted","Data":"c3f38daa6ebf91e07ff9da72968eb99611d964bb01f63af9c26dce52ab7cdb9d"} Oct 30 16:48:37.669276 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:37.669239 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz" event={"ID":"efc9c129-4290-4f6b-b02f-e59979426e5a","Type":"ContainerStarted","Data":"519916685c6cb622a42d1901cc2805ecc39ec51f878ba6242bbb35e739cc973e"} Oct 30 16:48:37.669843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:37.669307 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz" Oct 30 16:48:37.671053 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:37.671026 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md" event={"ID":"6215914d-d1a2-45e0-a811-469d418061df","Type":"ContainerStarted","Data":"6fed0e73f163a4ffd9a0a69432e797e7e0de53724fed5f533e91d1d993e914c6"} Oct 30 16:48:37.671189 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:37.671145 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md" Oct 30 16:48:37.711165 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:37.711120 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz" podStartSLOduration=2.711109074 podStartE2EDuration="2.711109074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:37.708960264 +0000 UTC m=+1631.419415008" watchObservedRunningTime="2026-10-30 16:48:37.711109074 +0000 UTC m=+1631.421563817" Oct 30 16:48:37.750473 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:37.750424 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md" podStartSLOduration=2.750411411 podStartE2EDuration="2.750411411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:37.748215116 +0000 UTC m=+1631.458669861" watchObservedRunningTime="2026-10-30 16:48:37.750411411 +0000 UTC m=+1631.460866145" Oct 30 16:48:38.065827 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:38.065725 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:38.979150 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:38.979112 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:39.812810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:39.812755 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:40.099041 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:40.099004 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg"] Oct 30 16:48:40.104428 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:40.104406 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg" Oct 30 16:48:40.124316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:40.124290 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg"] Oct 30 16:48:40.130271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:40.130243 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:48:40.130450 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:40.130434 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:48:40.130520 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:40.130505 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:48:40.130950 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:40.130932 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:48:40.237562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:40.237530 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5d3534-9495-47b5-a171-f3d4bc384eb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-968df4-pmrvg\" (UID: \"8e5d3534-9495-47b5-a171-f3d4bc384eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg" Oct 30 16:48:40.237808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:40.237595 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5d3534-9495-47b5-a171-f3d4bc384eb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-968df4-pmrvg\" (UID: \"8e5d3534-9495-47b5-a171-f3d4bc384eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg" Oct 30 16:48:40.237808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:40.237738 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-849s5\" (UniqueName: \"kubernetes.io/projected/8e5d3534-9495-47b5-a171-f3d4bc384eb1-kube-api-access-849s5\") pod \"maas-api-revoke-keys-e2e-api-a-968df4-pmrvg\" (UID: \"8e5d3534-9495-47b5-a171-f3d4bc384eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg" Oct 30 16:48:40.339324 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:40.339283 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5d3534-9495-47b5-a171-f3d4bc384eb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-968df4-pmrvg\" (UID: \"8e5d3534-9495-47b5-a171-f3d4bc384eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg" Oct 30 16:48:40.339542 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:40.339372 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5d3534-9495-47b5-a171-f3d4bc384eb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-968df4-pmrvg\" (UID: \"8e5d3534-9495-47b5-a171-f3d4bc384eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg" Oct 30 16:48:40.339542 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:40.339498 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-849s5\" (UniqueName: \"kubernetes.io/projected/8e5d3534-9495-47b5-a171-f3d4bc384eb1-kube-api-access-849s5\") pod \"maas-api-revoke-keys-e2e-api-a-968df4-pmrvg\" (UID: \"8e5d3534-9495-47b5-a171-f3d4bc384eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg" Oct 30 16:48:40.340190 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:40.340154 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5d3534-9495-47b5-a171-f3d4bc384eb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-968df4-pmrvg\" (UID: \"8e5d3534-9495-47b5-a171-f3d4bc384eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg" Oct 30 16:48:40.342331 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:40.342303 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5d3534-9495-47b5-a171-f3d4bc384eb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-968df4-pmrvg\" (UID: \"8e5d3534-9495-47b5-a171-f3d4bc384eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg" Oct 30 16:48:40.362727 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:40.362665 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-849s5\" (UniqueName: \"kubernetes.io/projected/8e5d3534-9495-47b5-a171-f3d4bc384eb1-kube-api-access-849s5\") pod \"maas-api-revoke-keys-e2e-api-a-968df4-pmrvg\" (UID: \"8e5d3534-9495-47b5-a171-f3d4bc384eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg" Oct 30 16:48:40.427848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:40.427821 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg" Oct 30 16:48:40.739209 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:48:40.739162 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5d3534_9495_47b5_a171_f3d4bc384eb1.slice/crio-c52e4b38974018277428dfbbb2534c9539ba9ca1dc54b4998dd3f3870808090c WatchSource:0}: Error finding container c52e4b38974018277428dfbbb2534c9539ba9ca1dc54b4998dd3f3870808090c: Status 404 returned error can't find the container with id c52e4b38974018277428dfbbb2534c9539ba9ca1dc54b4998dd3f3870808090c Oct 30 16:48:40.740349 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:40.740320 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg"] Oct 30 16:48:41.620634 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:41.620568 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:41.709533 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:41.709499 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg" event={"ID":"8e5d3534-9495-47b5-a171-f3d4bc384eb1","Type":"ContainerStarted","Data":"a802f2d979da69e617142f7ca05762877607a7c8db1027a8e07ea49106864bd8"} Oct 30 16:48:41.709685 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:41.709553 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg" event={"ID":"8e5d3534-9495-47b5-a171-f3d4bc384eb1","Type":"ContainerStarted","Data":"c52e4b38974018277428dfbbb2534c9539ba9ca1dc54b4998dd3f3870808090c"} Oct 30 16:48:41.757819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:41.757282 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg" podStartSLOduration=1.7572650840000001 podStartE2EDuration="1.757265084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:41.753705612 +0000 UTC m=+1635.464160382" watchObservedRunningTime="2026-10-30 16:48:41.757265084 +0000 UTC m=+1635.467719827" Oct 30 16:48:42.718946 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:42.718912 6202 generic.go:356] "Generic (PLEG): container finished" podID="8e5d3534-9495-47b5-a171-f3d4bc384eb1" containerID="a802f2d979da69e617142f7ca05762877607a7c8db1027a8e07ea49106864bd8" exitCode=0 Oct 30 16:48:42.719374 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:42.718993 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg" event={"ID":"8e5d3534-9495-47b5-a171-f3d4bc384eb1","Type":"ContainerDied","Data":"a802f2d979da69e617142f7ca05762877607a7c8db1027a8e07ea49106864bd8"} Oct 30 16:48:43.418016 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:43.417976 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:43.910029 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:43.910004 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg" Oct 30 16:48:44.015466 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:44.015384 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e5d3534-9495-47b5-a171-f3d4bc384eb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5d3534-9495-47b5-a171-f3d4bc384eb1-maas-api-openshift-ca\") pod \"8e5d3534-9495-47b5-a171-f3d4bc384eb1\" (UID: \"8e5d3534-9495-47b5-a171-f3d4bc384eb1\") " Oct 30 16:48:44.015466 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:44.015424 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e5d3534-9495-47b5-a171-f3d4bc384eb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5d3534-9495-47b5-a171-f3d4bc384eb1-maas-api-xks-ca\") pod \"8e5d3534-9495-47b5-a171-f3d4bc384eb1\" (UID: \"8e5d3534-9495-47b5-a171-f3d4bc384eb1\") " Oct 30 16:48:44.015651 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:44.015632 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5d3534-9495-47b5-a171-f3d4bc384eb1-kube-api-access-849s5\" (UniqueName: \"kubernetes.io/projected/8e5d3534-9495-47b5-a171-f3d4bc384eb1-kube-api-access-849s5\") pod \"8e5d3534-9495-47b5-a171-f3d4bc384eb1\" (UID: \"8e5d3534-9495-47b5-a171-f3d4bc384eb1\") " Oct 30 16:48:44.015847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:44.015821 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5d3534-9495-47b5-a171-f3d4bc384eb1-maas-api-openshift-ca" pod "8e5d3534-9495-47b5-a171-f3d4bc384eb1" (UID: "8e5d3534-9495-47b5-a171-f3d4bc384eb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:44.016568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:44.016540 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5d3534-9495-47b5-a171-f3d4bc384eb1-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:48:44.017956 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:44.017928 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5d3534-9495-47b5-a171-f3d4bc384eb1-kube-api-access-849s5" pod "8e5d3534-9495-47b5-a171-f3d4bc384eb1" (UID: "8e5d3534-9495-47b5-a171-f3d4bc384eb1"). InnerVolumeSpecName "kube-api-access-849s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:44.018041 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:44.017956 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e5d3534-9495-47b5-a171-f3d4bc384eb1-maas-api-xks-ca" pod "8e5d3534-9495-47b5-a171-f3d4bc384eb1" (UID: "8e5d3534-9495-47b5-a171-f3d4bc384eb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:48:44.073554 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:48:44.073516 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:48:44.118498 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:44.118471 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5d3534-9495-47b5-a171-f3d4bc384eb1-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:48:44.118498 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:44.118498 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-849s5\" (UniqueName: \"kubernetes.io/projected/8e5d3534-9495-47b5-a171-f3d4bc384eb1-kube-api-access-849s5\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:48:44.736935 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:44.736864 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg" Oct 30 16:48:44.736935 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:44.736904 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg" event={"ID":"8e5d3534-9495-47b5-a171-f3d4bc384eb1","Type":"ContainerDied","Data":"c52e4b38974018277428dfbbb2534c9539ba9ca1dc54b4998dd3f3870808090c"} Oct 30 16:48:44.737122 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:44.736961 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c52e4b38974018277428dfbbb2534c9539ba9ca1dc54b4998dd3f3870808090c" Oct 30 16:48:45.138525 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:45.138489 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:46.684259 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:46.684207 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:48.472664 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:48.472618 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:48.586062 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:48.586022 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:48.683131 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:48.683100 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md" Oct 30 16:48:48.683311 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:48.683166 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz" Oct 30 16:48:50.167164 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:50.167132 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:50.501184 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:50.501097 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt"] Oct 30 16:48:50.536011 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:50.535980 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm"] Oct 30 16:48:50.639816 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:50.639761 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:50.757182 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:50.757047 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:52.012307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:52.012273 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:52.132040 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:52.132012 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:52.488737 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:52.488597 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:52.869202 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:52.869168 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:52.981110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:52.981070 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:53.048558 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.048517 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6"] Oct 30 16:48:53.050002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.049974 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8e5d3534-9495-47b5-a171-f3d4bc384eb1" containerName="revoke-keys" Oct 30 16:48:53.050002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.050001 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5d3534-9495-47b5-a171-f3d4bc384eb1" containerName="revoke-keys" Oct 30 16:48:53.050278 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.050261 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5d3534-9495-47b5-a171-f3d4bc384eb1" containerName="revoke-keys" Oct 30 16:48:53.055332 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.055311 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6" Oct 30 16:48:53.071541 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.071514 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6"] Oct 30 16:48:53.128850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.105283 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b8af660d\"" Oct 30 16:48:53.186804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.129040 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b8af660d-dockercfg-p6rlm\"" Oct 30 16:48:53.186804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.130782 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872ede25-b9c3-43a9-a242-88381d4c8e74-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6\" (UID: \"872ede25-b9c3-43a9-a242-88381d4c8e74\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6" Oct 30 16:48:53.186804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.130828 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6f4g\" (UniqueName: \"kubernetes.io/projected/872ede25-b9c3-43a9-a242-88381d4c8e74-kube-api-access-k6f4g\") pod \"payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6\" (UID: \"872ede25-b9c3-43a9-a242-88381d4c8e74\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6" Oct 30 16:48:53.264793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.233399 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872ede25-b9c3-43a9-a242-88381d4c8e74-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6\" (UID: \"872ede25-b9c3-43a9-a242-88381d4c8e74\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6" Oct 30 16:48:53.264793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.233452 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6f4g\" (UniqueName: \"kubernetes.io/projected/872ede25-b9c3-43a9-a242-88381d4c8e74-kube-api-access-k6f4g\") pod \"payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6\" (UID: \"872ede25-b9c3-43a9-a242-88381d4c8e74\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6" Oct 30 16:48:53.329790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.328017 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872ede25-b9c3-43a9-a242-88381d4c8e74-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6\" (UID: \"872ede25-b9c3-43a9-a242-88381d4c8e74\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6" Oct 30 16:48:53.341845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.339449 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6f4g\" (UniqueName: \"kubernetes.io/projected/872ede25-b9c3-43a9-a242-88381d4c8e74-kube-api-access-k6f4g\") pod \"payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6\" (UID: \"872ede25-b9c3-43a9-a242-88381d4c8e74\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6" Oct 30 16:48:53.376390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.376357 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6" Oct 30 16:48:53.568192 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.568152 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6"] Oct 30 16:48:53.575066 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.575038 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6" Oct 30 16:48:53.590240 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.590019 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:48:53.590240 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.590144 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:48:53.592918 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.592897 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6"] Oct 30 16:48:53.599044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.598985 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:48:53.599044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.598994 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:48:53.663120 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.663036 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c849ce-219a-48c0-8646-071636dc498a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a130e2-zcpv6\" (UID: \"19c849ce-219a-48c0-8646-071636dc498a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6" Oct 30 16:48:53.663120 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.663093 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jvkt\" (UniqueName: \"kubernetes.io/projected/19c849ce-219a-48c0-8646-071636dc498a-kube-api-access-4jvkt\") pod \"maas-api-revoke-sub-61a130e2-zcpv6\" (UID: \"19c849ce-219a-48c0-8646-071636dc498a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6" Oct 30 16:48:53.663567 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.663537 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c849ce-219a-48c0-8646-071636dc498a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a130e2-zcpv6\" (UID: \"19c849ce-219a-48c0-8646-071636dc498a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6" Oct 30 16:48:53.694624 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.693809 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6"] Oct 30 16:48:53.694975 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:48:53.694946 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872ede25_b9c3_43a9_a242_88381d4c8e74.slice/crio-4529bf22832fdc77aca74ad53016107860b301dfc7385a618a40de71488ac7b0 WatchSource:0}: Error finding container 4529bf22832fdc77aca74ad53016107860b301dfc7385a618a40de71488ac7b0: Status 404 returned error can't find the container with id 4529bf22832fdc77aca74ad53016107860b301dfc7385a618a40de71488ac7b0 Oct 30 16:48:53.765557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.765528 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c849ce-219a-48c0-8646-071636dc498a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a130e2-zcpv6\" (UID: \"19c849ce-219a-48c0-8646-071636dc498a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6" Oct 30 16:48:53.765696 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.765586 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c849ce-219a-48c0-8646-071636dc498a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a130e2-zcpv6\" (UID: \"19c849ce-219a-48c0-8646-071636dc498a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6" Oct 30 16:48:53.765696 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.765617 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jvkt\" (UniqueName: \"kubernetes.io/projected/19c849ce-219a-48c0-8646-071636dc498a-kube-api-access-4jvkt\") pod \"maas-api-revoke-sub-61a130e2-zcpv6\" (UID: \"19c849ce-219a-48c0-8646-071636dc498a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6" Oct 30 16:48:53.766359 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.766333 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c849ce-219a-48c0-8646-071636dc498a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a130e2-zcpv6\" (UID: \"19c849ce-219a-48c0-8646-071636dc498a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6" Oct 30 16:48:53.768191 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.768160 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c849ce-219a-48c0-8646-071636dc498a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a130e2-zcpv6\" (UID: \"19c849ce-219a-48c0-8646-071636dc498a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6" Oct 30 16:48:53.788928 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.788904 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jvkt\" (UniqueName: \"kubernetes.io/projected/19c849ce-219a-48c0-8646-071636dc498a-kube-api-access-4jvkt\") pod \"maas-api-revoke-sub-61a130e2-zcpv6\" (UID: \"19c849ce-219a-48c0-8646-071636dc498a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6" Oct 30 16:48:53.846910 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.846821 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6" event={"ID":"872ede25-b9c3-43a9-a242-88381d4c8e74","Type":"ContainerStarted","Data":"e8a87859cd9d22de6a25137cc3ceaf8e40d6e01e97291f29dc3b3771e94284de"} Oct 30 16:48:53.846910 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.846917 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6" event={"ID":"872ede25-b9c3-43a9-a242-88381d4c8e74","Type":"ContainerStarted","Data":"4529bf22832fdc77aca74ad53016107860b301dfc7385a618a40de71488ac7b0"} Oct 30 16:48:53.846910 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.846945 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6" Oct 30 16:48:53.886950 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.886901 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6" podStartSLOduration=0.886887499 podStartE2EDuration="886.887499ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:53.885639364 +0000 UTC m=+1647.596094109" watchObservedRunningTime="2026-10-30 16:48:53.886887499 +0000 UTC m=+1647.597342244" Oct 30 16:48:53.897281 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:53.897255 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6" Oct 30 16:48:54.201530 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:48:54.201495 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c849ce_219a_48c0_8646_071636dc498a.slice/crio-c3c6f0a2afbec34585f33229fa55d50350631a8b1f0e55a6c3fa091d26857dd2 WatchSource:0}: Error finding container c3c6f0a2afbec34585f33229fa55d50350631a8b1f0e55a6c3fa091d26857dd2: Status 404 returned error can't find the container with id c3c6f0a2afbec34585f33229fa55d50350631a8b1f0e55a6c3fa091d26857dd2 Oct 30 16:48:54.201944 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:54.201707 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6"] Oct 30 16:48:54.792113 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:54.792035 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:54.859843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:54.859756 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6" event={"ID":"19c849ce-219a-48c0-8646-071636dc498a","Type":"ContainerStarted","Data":"87ccf2487430f92873e24ccd858631799acb3e30db5ed76f3b6c23fae9e2daa8"} Oct 30 16:48:54.860026 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:54.859892 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6" event={"ID":"19c849ce-219a-48c0-8646-071636dc498a","Type":"ContainerStarted","Data":"c3c6f0a2afbec34585f33229fa55d50350631a8b1f0e55a6c3fa091d26857dd2"} Oct 30 16:48:54.904960 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:54.904901 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6" podStartSLOduration=1.9048835990000001 podStartE2EDuration="1.904883599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:54.900074413 +0000 UTC m=+1648.610529157" watchObservedRunningTime="2026-10-30 16:48:54.904883599 +0000 UTC m=+1648.615338349" Oct 30 16:48:55.502187 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.502104 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt" podUID="adf56f97-c26b-4f47-bb3d-8d1b43b2b800" containerName="payload-processing" containerID="cri-o://6d0421e1730cf350f4038ff3c5f0d11596896aa27c0c0b8271fe777e5c8716ca" gracePeriod=25 Oct 30 16:48:55.536781 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.536732 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm" podUID="fcd634aa-9f6b-4341-a572-2e24161d692f" containerName="payload-pre-processing" containerID="cri-o://c69d96d9c51661fef2168e7fde6fa329cb6f133d08bec5effda6d3c5253a1a4b" gracePeriod=25 Oct 30 16:48:55.856163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.856131 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt" Oct 30 16:48:55.871198 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.871168 6202 generic.go:356] "Generic (PLEG): container finished" podID="19c849ce-219a-48c0-8646-071636dc498a" containerID="87ccf2487430f92873e24ccd858631799acb3e30db5ed76f3b6c23fae9e2daa8" exitCode=0 Oct 30 16:48:55.871369 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.871257 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6" event={"ID":"19c849ce-219a-48c0-8646-071636dc498a","Type":"ContainerDied","Data":"87ccf2487430f92873e24ccd858631799acb3e30db5ed76f3b6c23fae9e2daa8"} Oct 30 16:48:55.873048 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.873023 6202 generic.go:356] "Generic (PLEG): container finished" podID="adf56f97-c26b-4f47-bb3d-8d1b43b2b800" containerID="6d0421e1730cf350f4038ff3c5f0d11596896aa27c0c0b8271fe777e5c8716ca" exitCode=0 Oct 30 16:48:55.873151 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.873069 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt" Oct 30 16:48:55.873151 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.873102 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt" event={"ID":"adf56f97-c26b-4f47-bb3d-8d1b43b2b800","Type":"ContainerDied","Data":"6d0421e1730cf350f4038ff3c5f0d11596896aa27c0c0b8271fe777e5c8716ca"} Oct 30 16:48:55.873260 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.873158 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt" event={"ID":"adf56f97-c26b-4f47-bb3d-8d1b43b2b800","Type":"ContainerDied","Data":"639e55d8478582a418c3f726474bd3d852c8247bd8b2deec28c84341fad768ae"} Oct 30 16:48:55.873260 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.873191 6202 scope.go:122] "RemoveContainer" containerID="6d0421e1730cf350f4038ff3c5f0d11596896aa27c0c0b8271fe777e5c8716ca" Oct 30 16:48:55.875258 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.875211 6202 generic.go:356] "Generic (PLEG): container finished" podID="fcd634aa-9f6b-4341-a572-2e24161d692f" containerID="c69d96d9c51661fef2168e7fde6fa329cb6f133d08bec5effda6d3c5253a1a4b" exitCode=0 Oct 30 16:48:55.875472 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.875398 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm" event={"ID":"fcd634aa-9f6b-4341-a572-2e24161d692f","Type":"ContainerDied","Data":"c69d96d9c51661fef2168e7fde6fa329cb6f133d08bec5effda6d3c5253a1a4b"} Oct 30 16:48:55.891276 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.891254 6202 scope.go:122] "RemoveContainer" containerID="6d0421e1730cf350f4038ff3c5f0d11596896aa27c0c0b8271fe777e5c8716ca" Oct 30 16:48:55.891592 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:48:55.891571 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0421e1730cf350f4038ff3c5f0d11596896aa27c0c0b8271fe777e5c8716ca\": container with ID starting with 6d0421e1730cf350f4038ff3c5f0d11596896aa27c0c0b8271fe777e5c8716ca not found: ID does not exist" containerID="6d0421e1730cf350f4038ff3c5f0d11596896aa27c0c0b8271fe777e5c8716ca" Oct 30 16:48:55.891669 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.891605 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0421e1730cf350f4038ff3c5f0d11596896aa27c0c0b8271fe777e5c8716ca"} err="failed to get container status \"6d0421e1730cf350f4038ff3c5f0d11596896aa27c0c0b8271fe777e5c8716ca\": rpc error: code = NotFound desc = could not find container \"6d0421e1730cf350f4038ff3c5f0d11596896aa27c0c0b8271fe777e5c8716ca\": container with ID starting with 6d0421e1730cf350f4038ff3c5f0d11596896aa27c0c0b8271fe777e5c8716ca not found: ID does not exist" Oct 30 16:48:55.894409 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.894387 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm" Oct 30 16:48:55.903516 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.903494 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adf56f97-c26b-4f47-bb3d-8d1b43b2b800-kube-api-access-xgvv4\" (UniqueName: \"kubernetes.io/projected/adf56f97-c26b-4f47-bb3d-8d1b43b2b800-kube-api-access-xgvv4\") pod \"adf56f97-c26b-4f47-bb3d-8d1b43b2b800\" (UID: \"adf56f97-c26b-4f47-bb3d-8d1b43b2b800\") " Oct 30 16:48:55.903903 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.903882 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adf56f97-c26b-4f47-bb3d-8d1b43b2b800-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf56f97-c26b-4f47-bb3d-8d1b43b2b800-plugins-config-volume\") pod \"adf56f97-c26b-4f47-bb3d-8d1b43b2b800\" (UID: \"adf56f97-c26b-4f47-bb3d-8d1b43b2b800\") " Oct 30 16:48:55.904255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.904231 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf56f97-c26b-4f47-bb3d-8d1b43b2b800-plugins-config-volume" pod "adf56f97-c26b-4f47-bb3d-8d1b43b2b800" (UID: "adf56f97-c26b-4f47-bb3d-8d1b43b2b800"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:55.904568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.904548 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf56f97-c26b-4f47-bb3d-8d1b43b2b800-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:48:55.906110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:55.906085 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf56f97-c26b-4f47-bb3d-8d1b43b2b800-kube-api-access-xgvv4" pod "adf56f97-c26b-4f47-bb3d-8d1b43b2b800" (UID: "adf56f97-c26b-4f47-bb3d-8d1b43b2b800"). InnerVolumeSpecName "kube-api-access-xgvv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:56.006025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:56.005996 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd634aa-9f6b-4341-a572-2e24161d692f-kube-api-access-9nzks\" (UniqueName: \"kubernetes.io/projected/fcd634aa-9f6b-4341-a572-2e24161d692f-kube-api-access-9nzks\") pod \"fcd634aa-9f6b-4341-a572-2e24161d692f\" (UID: \"fcd634aa-9f6b-4341-a572-2e24161d692f\") " Oct 30 16:48:56.006409 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:56.006387 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcd634aa-9f6b-4341-a572-2e24161d692f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd634aa-9f6b-4341-a572-2e24161d692f-plugins-config-volume\") pod \"fcd634aa-9f6b-4341-a572-2e24161d692f\" (UID: \"fcd634aa-9f6b-4341-a572-2e24161d692f\") " Oct 30 16:48:56.006924 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:56.006846 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcd634aa-9f6b-4341-a572-2e24161d692f-plugins-config-volume" pod "fcd634aa-9f6b-4341-a572-2e24161d692f" (UID: "fcd634aa-9f6b-4341-a572-2e24161d692f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:56.007147 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:56.007115 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgvv4\" (UniqueName: \"kubernetes.io/projected/adf56f97-c26b-4f47-bb3d-8d1b43b2b800-kube-api-access-xgvv4\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:48:56.007147 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:56.007141 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd634aa-9f6b-4341-a572-2e24161d692f-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:48:56.008541 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:56.008511 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd634aa-9f6b-4341-a572-2e24161d692f-kube-api-access-9nzks" pod "fcd634aa-9f6b-4341-a572-2e24161d692f" (UID: "fcd634aa-9f6b-4341-a572-2e24161d692f"). InnerVolumeSpecName "kube-api-access-9nzks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:56.108689 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:56.108653 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nzks\" (UniqueName: \"kubernetes.io/projected/fcd634aa-9f6b-4341-a572-2e24161d692f-kube-api-access-9nzks\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:48:56.221537 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:56.221506 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt"] Oct 30 16:48:56.226367 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:56.226343 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-968df4-c86dd7864-sxkpt"] Oct 30 16:48:56.414642 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:56.414549 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adf56f97-c26b-4f47-bb3d-8d1b43b2b800" path="/var/lib/kubelet/pods/adf56f97-c26b-4f47-bb3d-8d1b43b2b800/volumes" Oct 30 16:48:56.493244 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:56.493210 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:56.885165 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:56.885132 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm" Oct 30 16:48:56.885165 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:56.885141 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm" event={"ID":"fcd634aa-9f6b-4341-a572-2e24161d692f","Type":"ContainerDied","Data":"87cce2b9c88d21f1333faa678388b37413c1ba905dd7832a0cc9df35be2d8a86"} Oct 30 16:48:56.885728 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:56.885229 6202 scope.go:122] "RemoveContainer" containerID="c69d96d9c51661fef2168e7fde6fa329cb6f133d08bec5effda6d3c5253a1a4b" Oct 30 16:48:56.943292 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:56.943260 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm"] Oct 30 16:48:56.955813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:56.955783 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-968df4-5f54586b9f-9n7rm"] Oct 30 16:48:57.100305 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:57.100276 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6" Oct 30 16:48:57.229679 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:57.229576 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c849ce-219a-48c0-8646-071636dc498a-kube-api-access-4jvkt\" (UniqueName: \"kubernetes.io/projected/19c849ce-219a-48c0-8646-071636dc498a-kube-api-access-4jvkt\") pod \"19c849ce-219a-48c0-8646-071636dc498a\" (UID: \"19c849ce-219a-48c0-8646-071636dc498a\") " Oct 30 16:48:57.229889 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:57.229710 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19c849ce-219a-48c0-8646-071636dc498a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c849ce-219a-48c0-8646-071636dc498a-maas-api-xks-ca\") pod \"19c849ce-219a-48c0-8646-071636dc498a\" (UID: \"19c849ce-219a-48c0-8646-071636dc498a\") " Oct 30 16:48:57.230005 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:57.229982 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c849ce-219a-48c0-8646-071636dc498a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c849ce-219a-48c0-8646-071636dc498a-maas-api-openshift-ca\") pod \"19c849ce-219a-48c0-8646-071636dc498a\" (UID: \"19c849ce-219a-48c0-8646-071636dc498a\") " Oct 30 16:48:57.230806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:57.230712 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c849ce-219a-48c0-8646-071636dc498a-maas-api-openshift-ca" pod "19c849ce-219a-48c0-8646-071636dc498a" (UID: "19c849ce-219a-48c0-8646-071636dc498a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:57.231116 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:57.231088 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c849ce-219a-48c0-8646-071636dc498a-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:48:57.232281 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:57.232252 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c849ce-219a-48c0-8646-071636dc498a-maas-api-xks-ca" pod "19c849ce-219a-48c0-8646-071636dc498a" (UID: "19c849ce-219a-48c0-8646-071636dc498a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:48:57.232381 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:57.232283 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c849ce-219a-48c0-8646-071636dc498a-kube-api-access-4jvkt" pod "19c849ce-219a-48c0-8646-071636dc498a" (UID: "19c849ce-219a-48c0-8646-071636dc498a"). InnerVolumeSpecName "kube-api-access-4jvkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:57.332579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:57.332539 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jvkt\" (UniqueName: \"kubernetes.io/projected/19c849ce-219a-48c0-8646-071636dc498a-kube-api-access-4jvkt\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:48:57.332579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:57.332571 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c849ce-219a-48c0-8646-071636dc498a-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:48:57.980891 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:57.980850 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6" event={"ID":"19c849ce-219a-48c0-8646-071636dc498a","Type":"ContainerDied","Data":"c3c6f0a2afbec34585f33229fa55d50350631a8b1f0e55a6c3fa091d26857dd2"} Oct 30 16:48:57.994935 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:57.994880 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3c6f0a2afbec34585f33229fa55d50350631a8b1f0e55a6c3fa091d26857dd2" Oct 30 16:48:57.995152 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:57.995025 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6" Oct 30 16:48:58.427798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:58.423587 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcd634aa-9f6b-4341-a572-2e24161d692f" path="/var/lib/kubelet/pods/fcd634aa-9f6b-4341-a572-2e24161d692f/volumes" Oct 30 16:48:58.441458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:48:58.441419 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:00.163459 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:00.163423 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:00.642597 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:00.642556 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:01.039518 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:01.039439 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv"] Oct 30 16:49:01.049175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:01.049150 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49da4384-cdhkv"] Oct 30 16:49:01.852359 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:01.852316 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:02.199491 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.199414 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8"] Oct 30 16:49:02.200309 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.200292 6202 cpu_manager.go:414] "containerMap: removing container" podUID="adf56f97-c26b-4f47-bb3d-8d1b43b2b800" containerName="payload-processing" Oct 30 16:49:02.200361 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.200312 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adf56f97-c26b-4f47-bb3d-8d1b43b2b800" containerName="payload-processing" Oct 30 16:49:02.200361 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.200347 6202 cpu_manager.go:414] "containerMap: removing container" podUID="19c849ce-219a-48c0-8646-071636dc498a" containerName="revoke-keys" Oct 30 16:49:02.200361 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.200352 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c849ce-219a-48c0-8646-071636dc498a" containerName="revoke-keys" Oct 30 16:49:02.200480 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.200383 6202 cpu_manager.go:414] "containerMap: removing container" podUID="fcd634aa-9f6b-4341-a572-2e24161d692f" containerName="payload-pre-processing" Oct 30 16:49:02.200480 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.200387 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd634aa-9f6b-4341-a572-2e24161d692f" containerName="payload-pre-processing" Oct 30 16:49:02.200559 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.200492 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd634aa-9f6b-4341-a572-2e24161d692f" containerName="payload-pre-processing" Oct 30 16:49:02.200559 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.200504 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="adf56f97-c26b-4f47-bb3d-8d1b43b2b800" containerName="payload-processing" Oct 30 16:49:02.200559 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.200517 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c849ce-219a-48c0-8646-071636dc498a" containerName="revoke-keys" Oct 30 16:49:02.204779 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.204751 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8" Oct 30 16:49:02.223230 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.223205 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8"] Oct 30 16:49:02.229938 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.229917 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:49:02.230099 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.229975 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:49:02.230099 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.229975 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:49:02.232340 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.232322 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:49:02.322007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.321974 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fd66f3-986d-47bf-b83f-15ebdd97f027-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8\" (UID: \"a5fd66f3-986d-47bf-b83f-15ebdd97f027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8" Oct 30 16:49:02.322179 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.322055 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8dtn\" (UniqueName: \"kubernetes.io/projected/a5fd66f3-986d-47bf-b83f-15ebdd97f027-kube-api-access-x8dtn\") pod \"maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8\" (UID: \"a5fd66f3-986d-47bf-b83f-15ebdd97f027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8" Oct 30 16:49:02.322307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.322281 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fd66f3-986d-47bf-b83f-15ebdd97f027-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8\" (UID: \"a5fd66f3-986d-47bf-b83f-15ebdd97f027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8" Oct 30 16:49:02.412432 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.412398 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3" path="/var/lib/kubelet/pods/ed7fb93a-650e-4c3c-9d27-93f2ac7c5ba3/volumes" Oct 30 16:49:02.424005 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.423976 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fd66f3-986d-47bf-b83f-15ebdd97f027-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8\" (UID: \"a5fd66f3-986d-47bf-b83f-15ebdd97f027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8" Oct 30 16:49:02.424147 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.424096 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fd66f3-986d-47bf-b83f-15ebdd97f027-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8\" (UID: \"a5fd66f3-986d-47bf-b83f-15ebdd97f027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8" Oct 30 16:49:02.424218 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.424147 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8dtn\" (UniqueName: \"kubernetes.io/projected/a5fd66f3-986d-47bf-b83f-15ebdd97f027-kube-api-access-x8dtn\") pod \"maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8\" (UID: \"a5fd66f3-986d-47bf-b83f-15ebdd97f027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8" Oct 30 16:49:02.424646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.424622 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fd66f3-986d-47bf-b83f-15ebdd97f027-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8\" (UID: \"a5fd66f3-986d-47bf-b83f-15ebdd97f027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8" Oct 30 16:49:02.426642 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.426620 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fd66f3-986d-47bf-b83f-15ebdd97f027-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8\" (UID: \"a5fd66f3-986d-47bf-b83f-15ebdd97f027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8" Oct 30 16:49:02.447738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.447708 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8dtn\" (UniqueName: \"kubernetes.io/projected/a5fd66f3-986d-47bf-b83f-15ebdd97f027-kube-api-access-x8dtn\") pod \"maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8\" (UID: \"a5fd66f3-986d-47bf-b83f-15ebdd97f027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8" Oct 30 16:49:02.520329 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.520244 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8" Oct 30 16:49:02.834408 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:49:02.834376 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5fd66f3_986d_47bf_b83f_15ebdd97f027.slice/crio-ca1912c9d9a03943f61756e120a3c77e00dfc287a95a258d45954f9b5473bfb0 WatchSource:0}: Error finding container ca1912c9d9a03943f61756e120a3c77e00dfc287a95a258d45954f9b5473bfb0: Status 404 returned error can't find the container with id ca1912c9d9a03943f61756e120a3c77e00dfc287a95a258d45954f9b5473bfb0 Oct 30 16:49:02.834681 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:02.834659 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8"] Oct 30 16:49:03.025244 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:03.025209 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8" event={"ID":"a5fd66f3-986d-47bf-b83f-15ebdd97f027","Type":"ContainerStarted","Data":"ca1912c9d9a03943f61756e120a3c77e00dfc287a95a258d45954f9b5473bfb0"} Oct 30 16:49:03.666965 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:03.666929 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:03.916736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:03.916699 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:04.036406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:04.036320 6202 generic.go:356] "Generic (PLEG): container finished" podID="a5fd66f3-986d-47bf-b83f-15ebdd97f027" containerID="3befbad68f60aba6d6aaeb4103fc58e3f5f2430438055520b1f236284282dc28" exitCode=0 Oct 30 16:49:04.036805 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:04.036406 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8" event={"ID":"a5fd66f3-986d-47bf-b83f-15ebdd97f027","Type":"ContainerDied","Data":"3befbad68f60aba6d6aaeb4103fc58e3f5f2430438055520b1f236284282dc28"} Oct 30 16:49:04.863835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:04.863803 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6" Oct 30 16:49:05.226527 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:05.226498 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8" Oct 30 16:49:05.270671 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:05.270595 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:05.274169 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:05.274135 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5fd66f3-986d-47bf-b83f-15ebdd97f027-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fd66f3-986d-47bf-b83f-15ebdd97f027-maas-api-xks-ca\") pod \"a5fd66f3-986d-47bf-b83f-15ebdd97f027\" (UID: \"a5fd66f3-986d-47bf-b83f-15ebdd97f027\") " Oct 30 16:49:05.274309 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:05.274266 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5fd66f3-986d-47bf-b83f-15ebdd97f027-kube-api-access-x8dtn\" (UniqueName: \"kubernetes.io/projected/a5fd66f3-986d-47bf-b83f-15ebdd97f027-kube-api-access-x8dtn\") pod \"a5fd66f3-986d-47bf-b83f-15ebdd97f027\" (UID: \"a5fd66f3-986d-47bf-b83f-15ebdd97f027\") " Oct 30 16:49:05.274503 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:05.274485 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5fd66f3-986d-47bf-b83f-15ebdd97f027-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fd66f3-986d-47bf-b83f-15ebdd97f027-maas-api-openshift-ca\") pod \"a5fd66f3-986d-47bf-b83f-15ebdd97f027\" (UID: \"a5fd66f3-986d-47bf-b83f-15ebdd97f027\") " Oct 30 16:49:05.275194 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:05.275164 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5fd66f3-986d-47bf-b83f-15ebdd97f027-maas-api-openshift-ca" pod "a5fd66f3-986d-47bf-b83f-15ebdd97f027" (UID: "a5fd66f3-986d-47bf-b83f-15ebdd97f027"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:05.275558 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:05.275536 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fd66f3-986d-47bf-b83f-15ebdd97f027-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:49:05.277689 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:05.277664 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5fd66f3-986d-47bf-b83f-15ebdd97f027-maas-api-xks-ca" pod "a5fd66f3-986d-47bf-b83f-15ebdd97f027" (UID: "a5fd66f3-986d-47bf-b83f-15ebdd97f027"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:49:05.277964 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:05.277941 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fd66f3-986d-47bf-b83f-15ebdd97f027-kube-api-access-x8dtn" pod "a5fd66f3-986d-47bf-b83f-15ebdd97f027" (UID: "a5fd66f3-986d-47bf-b83f-15ebdd97f027"). InnerVolumeSpecName "kube-api-access-x8dtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:05.381585 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:05.381467 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fd66f3-986d-47bf-b83f-15ebdd97f027-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:49:05.381585 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:05.381503 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8dtn\" (UniqueName: \"kubernetes.io/projected/a5fd66f3-986d-47bf-b83f-15ebdd97f027-kube-api-access-x8dtn\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:49:06.061536 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:06.061489 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8" event={"ID":"a5fd66f3-986d-47bf-b83f-15ebdd97f027","Type":"ContainerDied","Data":"ca1912c9d9a03943f61756e120a3c77e00dfc287a95a258d45954f9b5473bfb0"} Oct 30 16:49:06.061746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:06.061552 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca1912c9d9a03943f61756e120a3c77e00dfc287a95a258d45954f9b5473bfb0" Oct 30 16:49:06.061746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:06.061508 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8" Oct 30 16:49:06.981493 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:06.981447 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:08.476053 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:08.476020 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:09.979294 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:09.979254 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:10.527130 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:10.527098 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:11.610740 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:11.610707 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:12.596106 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:12.596069 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj"] Oct 30 16:49:12.738754 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:12.738718 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:12.854044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:12.852996 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:13.208662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:13.208564 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:13.334790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:13.332144 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:13.455125 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:13.455087 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:14.196712 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:49:14.196669 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:49:15.544794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:15.533221 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:17.107075 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:17.107038 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:17.314335 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:17.314303 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:17.596936 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:17.596888 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj" podUID="1516689f-6030-4980-9440-ed9d4e41a310" containerName="payload-processing" containerID="cri-o://fcfe02fbf7e049f56495b5aa2027c7d97e928911c93e905966d0fa81dd306b46" gracePeriod=25 Oct 30 16:49:17.626808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:17.626778 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:17.961063 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:17.955036 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj" Oct 30 16:49:17.981254 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:17.981086 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1516689f-6030-4980-9440-ed9d4e41a310-kube-api-access-jc29h\" (UniqueName: \"kubernetes.io/projected/1516689f-6030-4980-9440-ed9d4e41a310-kube-api-access-jc29h\") pod \"1516689f-6030-4980-9440-ed9d4e41a310\" (UID: \"1516689f-6030-4980-9440-ed9d4e41a310\") " Oct 30 16:49:17.981254 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:17.981140 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1516689f-6030-4980-9440-ed9d4e41a310-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1516689f-6030-4980-9440-ed9d4e41a310-plugins-config-volume\") pod \"1516689f-6030-4980-9440-ed9d4e41a310\" (UID: \"1516689f-6030-4980-9440-ed9d4e41a310\") " Oct 30 16:49:17.981668 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:17.981639 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1516689f-6030-4980-9440-ed9d4e41a310-plugins-config-volume" pod "1516689f-6030-4980-9440-ed9d4e41a310" (UID: "1516689f-6030-4980-9440-ed9d4e41a310"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:17.982569 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:17.982304 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1516689f-6030-4980-9440-ed9d4e41a310-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:49:17.989805 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:17.988949 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1516689f-6030-4980-9440-ed9d4e41a310-kube-api-access-jc29h" pod "1516689f-6030-4980-9440-ed9d4e41a310" (UID: "1516689f-6030-4980-9440-ed9d4e41a310"). InnerVolumeSpecName "kube-api-access-jc29h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:18.084849 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:18.083750 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc29h\" (UniqueName: \"kubernetes.io/projected/1516689f-6030-4980-9440-ed9d4e41a310-kube-api-access-jc29h\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:49:18.092424 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:18.092389 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:18.331804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:18.327805 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:18.364792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:18.364513 6202 generic.go:356] "Generic (PLEG): container finished" podID="1516689f-6030-4980-9440-ed9d4e41a310" containerID="fcfe02fbf7e049f56495b5aa2027c7d97e928911c93e905966d0fa81dd306b46" exitCode=0 Oct 30 16:49:18.364792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:18.364661 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj" event={"ID":"1516689f-6030-4980-9440-ed9d4e41a310","Type":"ContainerDied","Data":"fcfe02fbf7e049f56495b5aa2027c7d97e928911c93e905966d0fa81dd306b46"} Oct 30 16:49:18.364792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:18.364713 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj" event={"ID":"1516689f-6030-4980-9440-ed9d4e41a310","Type":"ContainerDied","Data":"690b432c42d1f05c6cf375f49da516d1fc66577351c057780490ad1ce76148b3"} Oct 30 16:49:18.364792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:18.364786 6202 scope.go:122] "RemoveContainer" containerID="fcfe02fbf7e049f56495b5aa2027c7d97e928911c93e905966d0fa81dd306b46" Oct 30 16:49:18.365174 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:18.364998 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj" Oct 30 16:49:18.467951 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:18.466109 6202 scope.go:122] "RemoveContainer" containerID="fcfe02fbf7e049f56495b5aa2027c7d97e928911c93e905966d0fa81dd306b46" Oct 30 16:49:18.467951 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:49:18.466642 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcfe02fbf7e049f56495b5aa2027c7d97e928911c93e905966d0fa81dd306b46\": container with ID starting with fcfe02fbf7e049f56495b5aa2027c7d97e928911c93e905966d0fa81dd306b46 not found: ID does not exist" containerID="fcfe02fbf7e049f56495b5aa2027c7d97e928911c93e905966d0fa81dd306b46" Oct 30 16:49:18.467951 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:18.466695 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcfe02fbf7e049f56495b5aa2027c7d97e928911c93e905966d0fa81dd306b46"} err="failed to get container status \"fcfe02fbf7e049f56495b5aa2027c7d97e928911c93e905966d0fa81dd306b46\": rpc error: code = NotFound desc = could not find container \"fcfe02fbf7e049f56495b5aa2027c7d97e928911c93e905966d0fa81dd306b46\": container with ID starting with fcfe02fbf7e049f56495b5aa2027c7d97e928911c93e905966d0fa81dd306b46 not found: ID does not exist" Oct 30 16:49:18.569828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:18.556822 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj"] Oct 30 16:49:18.572530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:18.572495 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b315d4-7c4d87887b-q6vgj"] Oct 30 16:49:19.093992 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:19.093955 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:20.412398 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:20.412357 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1516689f-6030-4980-9440-ed9d4e41a310" path="/var/lib/kubelet/pods/1516689f-6030-4980-9440-ed9d4e41a310/volumes" Oct 30 16:49:20.798307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:20.798211 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:22.404039 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:22.403994 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:22.520816 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:22.520715 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:23.991644 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:23.991607 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:24.081783 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.081732 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92"] Oct 30 16:49:24.082802 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.082759 6202 cpu_manager.go:414] "containerMap: removing container" podUID="1516689f-6030-4980-9440-ed9d4e41a310" containerName="payload-processing" Oct 30 16:49:24.082802 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.082802 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1516689f-6030-4980-9440-ed9d4e41a310" containerName="payload-processing" Oct 30 16:49:24.082990 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.082841 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a5fd66f3-986d-47bf-b83f-15ebdd97f027" containerName="revoke-keys" Oct 30 16:49:24.082990 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.082850 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5fd66f3-986d-47bf-b83f-15ebdd97f027" containerName="revoke-keys" Oct 30 16:49:24.083127 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.083112 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="1516689f-6030-4980-9440-ed9d4e41a310" containerName="payload-processing" Oct 30 16:49:24.083180 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.083136 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5fd66f3-986d-47bf-b83f-15ebdd97f027" containerName="revoke-keys" Oct 30 16:49:24.088665 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.088644 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" Oct 30 16:49:24.105982 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.105957 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3ee682\"" Oct 30 16:49:24.117783 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.116845 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c"] Oct 30 16:49:24.123219 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.123196 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92"] Oct 30 16:49:24.123493 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.123475 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" Oct 30 16:49:24.143260 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.143199 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3ee682-dockercfg-hchgq\"" Oct 30 16:49:24.150036 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.148445 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:24.157561 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.157506 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c"] Oct 30 16:49:24.215537 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.215491 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15c9a03-dd7f-4323-a9a5-83b76cd584ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92\" (UID: \"c15c9a03-dd7f-4323-a9a5-83b76cd584ff\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" Oct 30 16:49:24.215965 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.215929 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2t2p\" (UniqueName: \"kubernetes.io/projected/c15c9a03-dd7f-4323-a9a5-83b76cd584ff-kube-api-access-c2t2p\") pod \"payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92\" (UID: \"c15c9a03-dd7f-4323-a9a5-83b76cd584ff\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" Oct 30 16:49:24.216306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.216279 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3b37b0-b774-4172-917e-b887db4df343-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c\" (UID: \"bc3b37b0-b774-4172-917e-b887db4df343\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" Oct 30 16:49:24.216530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.216509 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz8np\" (UniqueName: \"kubernetes.io/projected/bc3b37b0-b774-4172-917e-b887db4df343-kube-api-access-zz8np\") pod \"payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c\" (UID: \"bc3b37b0-b774-4172-917e-b887db4df343\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" Oct 30 16:49:24.318627 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.318517 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2t2p\" (UniqueName: \"kubernetes.io/projected/c15c9a03-dd7f-4323-a9a5-83b76cd584ff-kube-api-access-c2t2p\") pod \"payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92\" (UID: \"c15c9a03-dd7f-4323-a9a5-83b76cd584ff\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" Oct 30 16:49:24.318975 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.318950 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3b37b0-b774-4172-917e-b887db4df343-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c\" (UID: \"bc3b37b0-b774-4172-917e-b887db4df343\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" Oct 30 16:49:24.319136 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.319121 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz8np\" (UniqueName: \"kubernetes.io/projected/bc3b37b0-b774-4172-917e-b887db4df343-kube-api-access-zz8np\") pod \"payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c\" (UID: \"bc3b37b0-b774-4172-917e-b887db4df343\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" Oct 30 16:49:24.319557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.319520 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15c9a03-dd7f-4323-a9a5-83b76cd584ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92\" (UID: \"c15c9a03-dd7f-4323-a9a5-83b76cd584ff\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" Oct 30 16:49:24.324975 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.320611 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15c9a03-dd7f-4323-a9a5-83b76cd584ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92\" (UID: \"c15c9a03-dd7f-4323-a9a5-83b76cd584ff\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" Oct 30 16:49:24.326062 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.326011 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3b37b0-b774-4172-917e-b887db4df343-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c\" (UID: \"bc3b37b0-b774-4172-917e-b887db4df343\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" Oct 30 16:49:24.439722 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.437999 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz8np\" (UniqueName: \"kubernetes.io/projected/bc3b37b0-b774-4172-917e-b887db4df343-kube-api-access-zz8np\") pod \"payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c\" (UID: \"bc3b37b0-b774-4172-917e-b887db4df343\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" Oct 30 16:49:24.468323 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.468291 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" Oct 30 16:49:24.485787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.470138 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2t2p\" (UniqueName: \"kubernetes.io/projected/c15c9a03-dd7f-4323-a9a5-83b76cd584ff-kube-api-access-c2t2p\") pod \"payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92\" (UID: \"c15c9a03-dd7f-4323-a9a5-83b76cd584ff\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" Oct 30 16:49:24.734828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.734756 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" Oct 30 16:49:24.817118 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.817090 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c"] Oct 30 16:49:24.817262 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:49:24.817230 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3b37b0_b774_4172_917e_b887db4df343.slice/crio-7e182be153e7d8f0b959e4f9937554fffa38cb7675662c7aebb0843ad0d62fb1 WatchSource:0}: Error finding container 7e182be153e7d8f0b959e4f9937554fffa38cb7675662c7aebb0843ad0d62fb1: Status 404 returned error can't find the container with id 7e182be153e7d8f0b959e4f9937554fffa38cb7675662c7aebb0843ad0d62fb1 Oct 30 16:49:24.916984 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:24.916573 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:25.066443 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:25.066410 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92"] Oct 30 16:49:25.071910 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:49:25.071836 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15c9a03_dd7f_4323_a9a5_83b76cd584ff.slice/crio-658f17bb7081dfe924352a03e204865d6c77dcd0e691b88e84851e48464d77e7 WatchSource:0}: Error finding container 658f17bb7081dfe924352a03e204865d6c77dcd0e691b88e84851e48464d77e7: Status 404 returned error can't find the container with id 658f17bb7081dfe924352a03e204865d6c77dcd0e691b88e84851e48464d77e7 Oct 30 16:49:25.503210 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:25.503126 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" event={"ID":"c15c9a03-dd7f-4323-a9a5-83b76cd584ff","Type":"ContainerStarted","Data":"52aa1db6ba7d73b3dfb90dce6d83b02ea3f119aae4a3cc9df28d98385a2d0ee9"} Oct 30 16:49:25.503210 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:25.503189 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" Oct 30 16:49:25.503461 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:25.503213 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" event={"ID":"c15c9a03-dd7f-4323-a9a5-83b76cd584ff","Type":"ContainerStarted","Data":"658f17bb7081dfe924352a03e204865d6c77dcd0e691b88e84851e48464d77e7"} Oct 30 16:49:25.504833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:25.504799 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" event={"ID":"bc3b37b0-b774-4172-917e-b887db4df343","Type":"ContainerStarted","Data":"6ab1d6f25ac64ddc8136793275db54e2108a4aa4e808450c16d1557e2ee6b533"} Oct 30 16:49:25.504972 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:25.504855 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" event={"ID":"bc3b37b0-b774-4172-917e-b887db4df343","Type":"ContainerStarted","Data":"7e182be153e7d8f0b959e4f9937554fffa38cb7675662c7aebb0843ad0d62fb1"} Oct 30 16:49:25.504972 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:25.504899 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" Oct 30 16:49:25.548987 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:25.548927 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" podStartSLOduration=1.5489097219999999 podStartE2EDuration="1.548909722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:25.542080924 +0000 UTC m=+1679.252535668" watchObservedRunningTime="2026-10-30 16:49:25.548909722 +0000 UTC m=+1679.259364466" Oct 30 16:49:25.584301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:25.584240 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" podStartSLOduration=1.584222503 podStartE2EDuration="1.584222503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:25.58147979 +0000 UTC m=+1679.291934537" watchObservedRunningTime="2026-10-30 16:49:25.584222503 +0000 UTC m=+1679.294677248" Oct 30 16:49:25.794835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:25.794759 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:26.518395 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:26.518355 6202 generic.go:356] "Generic (PLEG): container finished" podID="1f65a98a-1301-4d39-a40f-df19ec451712" containerID="805947c6ef9ee84b32d05c60294d386532b0a2401cdd92a499ee8a53249cf625" exitCode=0 Oct 30 16:49:26.518909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:26.518427 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" event={"ID":"1f65a98a-1301-4d39-a40f-df19ec451712","Type":"ContainerDied","Data":"805947c6ef9ee84b32d05c60294d386532b0a2401cdd92a499ee8a53249cf625"} Oct 30 16:49:26.519043 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:26.519023 6202 scope.go:122] "RemoveContainer" containerID="805947c6ef9ee84b32d05c60294d386532b0a2401cdd92a499ee8a53249cf625" Oct 30 16:49:26.521455 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:26.521421 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:49:26.521570 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:26.521548 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:49:26.521623 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:26.521611 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:49:26.558352 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:26.558325 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:27.529372 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:27.529338 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" event={"ID":"1f65a98a-1301-4d39-a40f-df19ec451712","Type":"ContainerStarted","Data":"9b0e8e1e1ed81010e4e9aeb1debbb374f13786c5259e9c340974190a659f39bf"} Oct 30 16:49:28.068817 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:28.068288 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:28.242484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:28.237830 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:49:28.258789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:28.250937 6202 patch_prober.go:29] interesting pod/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 16:49:28.258789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:28.251023 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" podUID="1f65a98a-1301-4d39-a40f-df19ec451712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 16:49:28.791704 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:28.791665 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:49:29.245790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:29.245186 6202 patch_prober.go:29] interesting pod/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 16:49:29.245790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:29.245260 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" podUID="1f65a98a-1301-4d39-a40f-df19ec451712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 16:49:29.628572 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:29.628520 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:29.775791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:29.775370 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:29.888761 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:29.888666 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:30.237680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:30.237594 6202 patch_prober.go:29] interesting pod/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 16:49:30.237680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:30.237652 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" podUID="1f65a98a-1301-4d39-a40f-df19ec451712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 16:49:30.314638 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:30.314602 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:30.430049 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:30.430016 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:30.962794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:30.959335 6202 scope.go:122] "RemoveContainer" containerID="2c62167b38fee54b8f6e5d7487fedfb8f7a94dfca5314cd66ab76a631909bb3e" Oct 30 16:49:31.003473 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:31.003426 6202 scope.go:122] "RemoveContainer" containerID="dbc6035f6fdaccccc0e6f0ea8bfe6161f0fceec635660f3fe6177a739a723655" Oct 30 16:49:31.026702 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:31.026002 6202 scope.go:122] "RemoveContainer" containerID="4183ca445e22e0e5bb78b17f76eeb6b75e3f2fcda0a99274cab8ef05f75114a7" Oct 30 16:49:31.080797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:31.078926 6202 scope.go:122] "RemoveContainer" containerID="4e04dae1366e2ee822f5ff3f2683a49c71bf70caeca3ca03a32e02a1ec1cf361" Oct 30 16:49:31.254849 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:31.254145 6202 patch_prober.go:29] interesting pod/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 16:49:31.254849 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:31.254204 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" podUID="1f65a98a-1301-4d39-a40f-df19ec451712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 16:49:31.361841 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:31.361806 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:32.250899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:32.250490 6202 patch_prober.go:29] interesting pod/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 16:49:32.250899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:32.250580 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" podUID="1f65a98a-1301-4d39-a40f-df19ec451712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 16:49:32.892543 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:32.892505 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:33.303965 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:33.303862 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:49:33.306272 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:33.306252 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:49:34.725163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:34.720577 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:36.214984 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:36.214948 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:36.522812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:36.522721 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" Oct 30 16:49:36.522812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:36.522810 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" Oct 30 16:49:37.771999 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:37.771963 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:39.093993 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:39.093963 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:39.204970 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:39.204931 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:39.720161 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:39.720131 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:42.233623 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:42.233581 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:44.070716 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:49:44.070661 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:49:44.154226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:44.154190 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:45.583279 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:45.583246 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:45.983050 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:45.982974 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:46.100030 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:46.099953 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:46.223899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:46.223845 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:47.837181 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:47.837151 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:49.630950 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:49.630897 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:49.980433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:49.980355 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:50.098359 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:50.098325 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:51.440485 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:51.440447 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:51.686410 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:51.686371 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:51.792362 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:51.792275 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs"] Oct 30 16:49:51.796909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:51.796890 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" Oct 30 16:49:51.816561 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:51.816530 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:51.817501 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:51.817478 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs"] Oct 30 16:49:51.824241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:51.824217 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ff1189-dockercfg-d2rk2\"" Oct 30 16:49:51.824342 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:51.824290 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ff1189\"" Oct 30 16:49:51.865785 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:51.865734 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c79977c-90b4-4d76-b6b4-b24aaf3df92b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs\" (UID: \"4c79977c-90b4-4d76-b6b4-b24aaf3df92b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" Oct 30 16:49:51.866187 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:51.866168 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7btft\" (UniqueName: \"kubernetes.io/projected/4c79977c-90b4-4d76-b6b4-b24aaf3df92b-kube-api-access-7btft\") pod \"payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs\" (UID: \"4c79977c-90b4-4d76-b6b4-b24aaf3df92b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" Oct 30 16:49:51.978239 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:51.978194 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c79977c-90b4-4d76-b6b4-b24aaf3df92b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs\" (UID: \"4c79977c-90b4-4d76-b6b4-b24aaf3df92b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" Oct 30 16:49:52.124796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:52.123973 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7btft\" (UniqueName: \"kubernetes.io/projected/4c79977c-90b4-4d76-b6b4-b24aaf3df92b-kube-api-access-7btft\") pod \"payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs\" (UID: \"4c79977c-90b4-4d76-b6b4-b24aaf3df92b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" Oct 30 16:49:52.124796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:52.002677 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c79977c-90b4-4d76-b6b4-b24aaf3df92b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs\" (UID: \"4c79977c-90b4-4d76-b6b4-b24aaf3df92b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" Oct 30 16:49:52.182023 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:52.181252 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7btft\" (UniqueName: \"kubernetes.io/projected/4c79977c-90b4-4d76-b6b4-b24aaf3df92b-kube-api-access-7btft\") pod \"payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs\" (UID: \"4c79977c-90b4-4d76-b6b4-b24aaf3df92b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" Oct 30 16:49:52.412079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:52.411985 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" Oct 30 16:49:52.734148 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:52.734106 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs"] Oct 30 16:49:52.734487 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:49:52.734240 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c79977c_90b4_4d76_b6b4_b24aaf3df92b.slice/crio-d2bd4463af7327ca3a3e1cbdc3bd4a7bf2e66997d1ba23ea224c3edac4cc895e WatchSource:0}: Error finding container d2bd4463af7327ca3a3e1cbdc3bd4a7bf2e66997d1ba23ea224c3edac4cc895e: Status 404 returned error can't find the container with id d2bd4463af7327ca3a3e1cbdc3bd4a7bf2e66997d1ba23ea224c3edac4cc895e Oct 30 16:49:52.905062 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:52.905024 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" event={"ID":"4c79977c-90b4-4d76-b6b4-b24aaf3df92b","Type":"ContainerStarted","Data":"81e406a2b0c9d3c01725f0c6a3a92c20be48ca8983296dac27219f16e047ab15"} Oct 30 16:49:52.905247 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:52.905099 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" event={"ID":"4c79977c-90b4-4d76-b6b4-b24aaf3df92b","Type":"ContainerStarted","Data":"d2bd4463af7327ca3a3e1cbdc3bd4a7bf2e66997d1ba23ea224c3edac4cc895e"} Oct 30 16:49:52.905247 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:52.905125 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" Oct 30 16:49:52.945452 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:52.945404 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" podStartSLOduration=1.945388129 podStartE2EDuration="1.945388129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:52.944966115 +0000 UTC m=+1706.655420864" watchObservedRunningTime="2026-10-30 16:49:52.945388129 +0000 UTC m=+1706.655842873" Oct 30 16:49:53.461059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:53.461012 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:55.327178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:55.327137 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:55.729486 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:55.729384 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:55.846265 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:55.846209 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:57.182497 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:57.182456 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:58.880683 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:49:58.880626 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:00.528282 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:00.528249 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:00.709791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:00.709728 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:02.119168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.119136 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb"] Oct 30 16:50:02.125946 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.125917 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:02.126103 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.126081 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb" Oct 30 16:50:02.143301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.143270 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb"] Oct 30 16:50:02.143524 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.143506 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:50:02.149061 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.149034 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:50:02.153152 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.153124 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:50:02.153440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.153419 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:50:02.226800 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.226339 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0775c0d-b1be-472c-aa0a-7a17d8352b28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb\" (UID: \"d0775c0d-b1be-472c-aa0a-7a17d8352b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb" Oct 30 16:50:02.226800 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.226570 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0775c0d-b1be-472c-aa0a-7a17d8352b28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb\" (UID: \"d0775c0d-b1be-472c-aa0a-7a17d8352b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb" Oct 30 16:50:02.226800 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.226644 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fzrr\" (UniqueName: \"kubernetes.io/projected/d0775c0d-b1be-472c-aa0a-7a17d8352b28-kube-api-access-8fzrr\") pod \"maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb\" (UID: \"d0775c0d-b1be-472c-aa0a-7a17d8352b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb" Oct 30 16:50:02.248540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.248509 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:02.331850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.331813 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0775c0d-b1be-472c-aa0a-7a17d8352b28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb\" (UID: \"d0775c0d-b1be-472c-aa0a-7a17d8352b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb" Oct 30 16:50:02.332050 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.331907 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fzrr\" (UniqueName: \"kubernetes.io/projected/d0775c0d-b1be-472c-aa0a-7a17d8352b28-kube-api-access-8fzrr\") pod \"maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb\" (UID: \"d0775c0d-b1be-472c-aa0a-7a17d8352b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb" Oct 30 16:50:02.332187 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.332169 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0775c0d-b1be-472c-aa0a-7a17d8352b28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb\" (UID: \"d0775c0d-b1be-472c-aa0a-7a17d8352b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb" Oct 30 16:50:02.332841 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.332758 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0775c0d-b1be-472c-aa0a-7a17d8352b28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb\" (UID: \"d0775c0d-b1be-472c-aa0a-7a17d8352b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb" Oct 30 16:50:02.334624 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.334599 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0775c0d-b1be-472c-aa0a-7a17d8352b28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb\" (UID: \"d0775c0d-b1be-472c-aa0a-7a17d8352b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb" Oct 30 16:50:02.366812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.366750 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fzrr\" (UniqueName: \"kubernetes.io/projected/d0775c0d-b1be-472c-aa0a-7a17d8352b28-kube-api-access-8fzrr\") pod \"maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb\" (UID: \"d0775c0d-b1be-472c-aa0a-7a17d8352b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb" Oct 30 16:50:02.443081 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.442994 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb" Oct 30 16:50:02.754207 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:02.754177 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb"] Oct 30 16:50:02.755370 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:50:02.755339 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0775c0d_b1be_472c_aa0a_7a17d8352b28.slice/crio-a6ff2cfcc778cad0369ecfa06ed463f57848a4a2cd697705bb94580472bdadc2 WatchSource:0}: Error finding container a6ff2cfcc778cad0369ecfa06ed463f57848a4a2cd697705bb94580472bdadc2: Status 404 returned error can't find the container with id a6ff2cfcc778cad0369ecfa06ed463f57848a4a2cd697705bb94580472bdadc2 Oct 30 16:50:03.018509 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:03.018422 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb" event={"ID":"d0775c0d-b1be-472c-aa0a-7a17d8352b28","Type":"ContainerStarted","Data":"a6ff2cfcc778cad0369ecfa06ed463f57848a4a2cd697705bb94580472bdadc2"} Oct 30 16:50:03.917411 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:03.917380 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" Oct 30 16:50:04.033952 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:04.033854 6202 generic.go:356] "Generic (PLEG): container finished" podID="d0775c0d-b1be-472c-aa0a-7a17d8352b28" containerID="76b5a30126e26238df544f902cf98f8444366da240f3260dcb711fcf415a89cb" exitCode=0 Oct 30 16:50:04.034131 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:04.034056 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb" event={"ID":"d0775c0d-b1be-472c-aa0a-7a17d8352b28","Type":"ContainerDied","Data":"76b5a30126e26238df544f902cf98f8444366da240f3260dcb711fcf415a89cb"} Oct 30 16:50:04.053587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:04.053555 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:04.793543 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:04.793502 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:05.031076 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:05.031044 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg"] Oct 30 16:50:05.031475 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:05.031253 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:05.044348 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:05.043745 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-968df4-pmrvg"] Oct 30 16:50:05.156325 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:05.156291 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:05.316164 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:05.315177 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb" Oct 30 16:50:05.433967 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:05.433390 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0775c0d-b1be-472c-aa0a-7a17d8352b28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0775c0d-b1be-472c-aa0a-7a17d8352b28-maas-api-xks-ca\") pod \"d0775c0d-b1be-472c-aa0a-7a17d8352b28\" (UID: \"d0775c0d-b1be-472c-aa0a-7a17d8352b28\") " Oct 30 16:50:05.433967 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:05.433547 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0775c0d-b1be-472c-aa0a-7a17d8352b28-kube-api-access-8fzrr\" (UniqueName: \"kubernetes.io/projected/d0775c0d-b1be-472c-aa0a-7a17d8352b28-kube-api-access-8fzrr\") pod \"d0775c0d-b1be-472c-aa0a-7a17d8352b28\" (UID: \"d0775c0d-b1be-472c-aa0a-7a17d8352b28\") " Oct 30 16:50:05.433967 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:05.433694 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0775c0d-b1be-472c-aa0a-7a17d8352b28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0775c0d-b1be-472c-aa0a-7a17d8352b28-maas-api-openshift-ca\") pod \"d0775c0d-b1be-472c-aa0a-7a17d8352b28\" (UID: \"d0775c0d-b1be-472c-aa0a-7a17d8352b28\") " Oct 30 16:50:05.459789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:05.436787 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0775c0d-b1be-472c-aa0a-7a17d8352b28-maas-api-openshift-ca" pod "d0775c0d-b1be-472c-aa0a-7a17d8352b28" (UID: "d0775c0d-b1be-472c-aa0a-7a17d8352b28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:05.510872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:05.509058 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0775c0d-b1be-472c-aa0a-7a17d8352b28-maas-api-xks-ca" pod "d0775c0d-b1be-472c-aa0a-7a17d8352b28" (UID: "d0775c0d-b1be-472c-aa0a-7a17d8352b28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:05.523457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:05.523422 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0775c0d-b1be-472c-aa0a-7a17d8352b28-kube-api-access-8fzrr" pod "d0775c0d-b1be-472c-aa0a-7a17d8352b28" (UID: "d0775c0d-b1be-472c-aa0a-7a17d8352b28"). InnerVolumeSpecName "kube-api-access-8fzrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:05.537988 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:05.537901 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0775c0d-b1be-472c-aa0a-7a17d8352b28-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:05.537988 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:05.537937 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fzrr\" (UniqueName: \"kubernetes.io/projected/d0775c0d-b1be-472c-aa0a-7a17d8352b28-kube-api-access-8fzrr\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:05.537988 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:05.537953 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0775c0d-b1be-472c-aa0a-7a17d8352b28-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:05.672163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:05.672129 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:06.112379 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:06.109894 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb" event={"ID":"d0775c0d-b1be-472c-aa0a-7a17d8352b28","Type":"ContainerDied","Data":"a6ff2cfcc778cad0369ecfa06ed463f57848a4a2cd697705bb94580472bdadc2"} Oct 30 16:50:06.112379 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:06.110000 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6ff2cfcc778cad0369ecfa06ed463f57848a4a2cd697705bb94580472bdadc2" Oct 30 16:50:06.112379 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:06.110080 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb" Oct 30 16:50:06.415961 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:06.415875 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e5d3534-9495-47b5-a171-f3d4bc384eb1" path="/var/lib/kubelet/pods/8e5d3534-9495-47b5-a171-f3d4bc384eb1/volumes" Oct 30 16:50:07.191123 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:07.191092 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:08.354844 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:08.354808 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:08.837396 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:08.837364 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:10.456798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:10.455715 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:11.761359 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:11.761324 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:12.405790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:12.405179 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:12.518615 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:12.518574 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:12.592899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:12.592869 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8"] Oct 30 16:50:12.606469 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:12.606440 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b315d4-jcdv8"] Oct 30 16:50:12.647104 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:12.647064 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:12.782902 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:12.782866 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:12.860429 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:12.856364 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl"] Oct 30 16:50:12.860429 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:12.856633 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" podUID="d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" containerName="istio-proxy" containerID="cri-o://b38baebc14951408e8c0c3a27e9c584d29f4b1ec5695936fa47a2de688cdffc9" gracePeriod=30 Oct 30 16:50:12.924798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:12.924706 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:13.163274 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.150295 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:13.316917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.316880 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9"] Oct 30 16:50:13.459738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.458595 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:13.520829 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.520791 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg"] Oct 30 16:50:13.522218 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.522196 6202 cpu_manager.go:414] "containerMap: removing container" podUID="d0775c0d-b1be-472c-aa0a-7a17d8352b28" containerName="revoke-keys" Oct 30 16:50:13.522282 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.522223 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0775c0d-b1be-472c-aa0a-7a17d8352b28" containerName="revoke-keys" Oct 30 16:50:13.522501 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.522485 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0775c0d-b1be-472c-aa0a-7a17d8352b28" containerName="revoke-keys" Oct 30 16:50:13.530010 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.529983 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg" Oct 30 16:50:13.568140 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.562431 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg"] Oct 30 16:50:13.568140 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.562621 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:50:13.568140 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.562735 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:50:13.568140 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.562852 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:50:13.568140 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.562935 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:50:13.580850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.580629 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:13.749852 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.720133 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95bc524-7472-4b9a-a0a5-2f12d396170a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg\" (UID: \"e95bc524-7472-4b9a-a0a5-2f12d396170a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg" Oct 30 16:50:13.755996 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.750345 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95bc524-7472-4b9a-a0a5-2f12d396170a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg\" (UID: \"e95bc524-7472-4b9a-a0a5-2f12d396170a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg" Oct 30 16:50:13.756435 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.756409 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grcbt\" (UniqueName: \"kubernetes.io/projected/e95bc524-7472-4b9a-a0a5-2f12d396170a-kube-api-access-grcbt\") pod \"maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg\" (UID: \"e95bc524-7472-4b9a-a0a5-2f12d396170a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg" Oct 30 16:50:13.857957 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.857916 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95bc524-7472-4b9a-a0a5-2f12d396170a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg\" (UID: \"e95bc524-7472-4b9a-a0a5-2f12d396170a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg" Oct 30 16:50:13.858424 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.858073 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95bc524-7472-4b9a-a0a5-2f12d396170a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg\" (UID: \"e95bc524-7472-4b9a-a0a5-2f12d396170a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg" Oct 30 16:50:13.858424 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.858191 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grcbt\" (UniqueName: \"kubernetes.io/projected/e95bc524-7472-4b9a-a0a5-2f12d396170a-kube-api-access-grcbt\") pod \"maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg\" (UID: \"e95bc524-7472-4b9a-a0a5-2f12d396170a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg" Oct 30 16:50:13.858781 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.858744 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95bc524-7472-4b9a-a0a5-2f12d396170a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg\" (UID: \"e95bc524-7472-4b9a-a0a5-2f12d396170a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg" Oct 30 16:50:13.860567 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.860546 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95bc524-7472-4b9a-a0a5-2f12d396170a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg\" (UID: \"e95bc524-7472-4b9a-a0a5-2f12d396170a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg" Oct 30 16:50:13.882741 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.882701 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grcbt\" (UniqueName: \"kubernetes.io/projected/e95bc524-7472-4b9a-a0a5-2f12d396170a-kube-api-access-grcbt\") pod \"maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg\" (UID: \"e95bc524-7472-4b9a-a0a5-2f12d396170a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg" Oct 30 16:50:13.887540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:13.887517 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg" Oct 30 16:50:14.447639 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:50:14.447576 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:50:14.482192 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:14.482159 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5fd66f3-986d-47bf-b83f-15ebdd97f027" path="/var/lib/kubelet/pods/a5fd66f3-986d-47bf-b83f-15ebdd97f027/volumes" Oct 30 16:50:14.482526 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:14.482511 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg"] Oct 30 16:50:15.257654 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:15.257614 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:15.345563 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:15.344965 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg" event={"ID":"e95bc524-7472-4b9a-a0a5-2f12d396170a","Type":"ContainerStarted","Data":"d52a1feebf490c9bcb620547c8683b2e2323e3482f7f1f698149bc207a75a3dd"} Oct 30 16:50:15.345563 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:15.345055 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg" event={"ID":"e95bc524-7472-4b9a-a0a5-2f12d396170a","Type":"ContainerStarted","Data":"81e07a196db0875ac2ef3699cfdea7e6bf262ec5ccc2ae5668e8686588f61b2a"} Oct 30 16:50:15.401081 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:15.401027 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg" podStartSLOduration=2.401011827 podStartE2EDuration="2.401011827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:15.395125126 +0000 UTC m=+1729.105579872" watchObservedRunningTime="2026-10-30 16:50:15.401011827 +0000 UTC m=+1729.111466570" Oct 30 16:50:16.361815 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:16.360061 6202 generic.go:356] "Generic (PLEG): container finished" podID="e95bc524-7472-4b9a-a0a5-2f12d396170a" containerID="d52a1feebf490c9bcb620547c8683b2e2323e3482f7f1f698149bc207a75a3dd" exitCode=0 Oct 30 16:50:16.361815 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:16.360113 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg" event={"ID":"e95bc524-7472-4b9a-a0a5-2f12d396170a","Type":"ContainerDied","Data":"d52a1feebf490c9bcb620547c8683b2e2323e3482f7f1f698149bc207a75a3dd"} Oct 30 16:50:17.015533 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:17.015487 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:17.557009 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:17.556978 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg" Oct 30 16:50:17.717332 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:17.717235 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e95bc524-7472-4b9a-a0a5-2f12d396170a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95bc524-7472-4b9a-a0a5-2f12d396170a-maas-api-xks-ca\") pod \"e95bc524-7472-4b9a-a0a5-2f12d396170a\" (UID: \"e95bc524-7472-4b9a-a0a5-2f12d396170a\") " Oct 30 16:50:17.717332 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:17.717306 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e95bc524-7472-4b9a-a0a5-2f12d396170a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95bc524-7472-4b9a-a0a5-2f12d396170a-maas-api-openshift-ca\") pod \"e95bc524-7472-4b9a-a0a5-2f12d396170a\" (UID: \"e95bc524-7472-4b9a-a0a5-2f12d396170a\") " Oct 30 16:50:17.717548 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:17.717414 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e95bc524-7472-4b9a-a0a5-2f12d396170a-kube-api-access-grcbt\" (UniqueName: \"kubernetes.io/projected/e95bc524-7472-4b9a-a0a5-2f12d396170a-kube-api-access-grcbt\") pod \"e95bc524-7472-4b9a-a0a5-2f12d396170a\" (UID: \"e95bc524-7472-4b9a-a0a5-2f12d396170a\") " Oct 30 16:50:17.717755 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:17.717729 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95bc524-7472-4b9a-a0a5-2f12d396170a-maas-api-openshift-ca" pod "e95bc524-7472-4b9a-a0a5-2f12d396170a" (UID: "e95bc524-7472-4b9a-a0a5-2f12d396170a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:17.718105 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:17.718085 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95bc524-7472-4b9a-a0a5-2f12d396170a-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:17.719554 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:17.719531 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e95bc524-7472-4b9a-a0a5-2f12d396170a-maas-api-xks-ca" pod "e95bc524-7472-4b9a-a0a5-2f12d396170a" (UID: "e95bc524-7472-4b9a-a0a5-2f12d396170a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:17.719722 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:17.719699 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95bc524-7472-4b9a-a0a5-2f12d396170a-kube-api-access-grcbt" pod "e95bc524-7472-4b9a-a0a5-2f12d396170a" (UID: "e95bc524-7472-4b9a-a0a5-2f12d396170a"). InnerVolumeSpecName "kube-api-access-grcbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:17.819466 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:17.819431 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95bc524-7472-4b9a-a0a5-2f12d396170a-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:17.819466 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:17.819461 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grcbt\" (UniqueName: \"kubernetes.io/projected/e95bc524-7472-4b9a-a0a5-2f12d396170a-kube-api-access-grcbt\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:18.153491 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.153464 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:50:18.224242 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.224210 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-token\" (UniqueName: \"kubernetes.io/projected/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-token\") pod \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " Oct 30 16:50:18.224425 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.224283 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-podinfo\") pod \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " Oct 30 16:50:18.224425 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.224305 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-credential-socket\") pod \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " Oct 30 16:50:18.224425 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.224340 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-kube-api-access-97pn2\" (UniqueName: \"kubernetes.io/projected/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-kube-api-access-97pn2\") pod \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " Oct 30 16:50:18.224425 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.224385 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-envoy\") pod \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " Oct 30 16:50:18.225038 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.224460 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-workload-socket\") pod \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " Oct 30 16:50:18.225038 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.224490 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-workload-certs\") pod \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " Oct 30 16:50:18.225038 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.224507 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-data\") pod \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " Oct 30 16:50:18.225038 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.224543 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istiod-ca-cert\") pod \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\" (UID: \"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48\") " Oct 30 16:50:18.225038 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.224673 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-credential-socket" pod "d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" (UID: "d0f45a80-37c0-47fe-95d2-ab1f6da1cd48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:18.225038 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.224872 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-data" pod "d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" (UID: "d0f45a80-37c0-47fe-95d2-ab1f6da1cd48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:18.225038 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.224884 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-workload-socket" pod "d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" (UID: "d0f45a80-37c0-47fe-95d2-ab1f6da1cd48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:18.225575 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.225181 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istiod-ca-cert" pod "d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" (UID: "d0f45a80-37c0-47fe-95d2-ab1f6da1cd48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:18.225575 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.225418 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:18.225575 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.225439 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:18.225575 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.225453 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:18.225575 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.225465 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:18.225872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.225618 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-workload-certs" pod "d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" (UID: "d0f45a80-37c0-47fe-95d2-ab1f6da1cd48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:18.226960 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.226942 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-envoy" pod "d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" (UID: "d0f45a80-37c0-47fe-95d2-ab1f6da1cd48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:18.227253 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.227228 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-podinfo" pod "d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" (UID: "d0f45a80-37c0-47fe-95d2-ab1f6da1cd48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:18.227348 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.227287 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-token" pod "d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" (UID: "d0f45a80-37c0-47fe-95d2-ab1f6da1cd48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:18.227598 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.227579 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-kube-api-access-97pn2" pod "d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" (UID: "d0f45a80-37c0-47fe-95d2-ab1f6da1cd48"). InnerVolumeSpecName "kube-api-access-97pn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:18.318187 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.318090 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9" podUID="f816f8bb-62e1-4fa0-abd1-37671f959fa2" containerName="payload-processing" containerID="cri-o://41e25693bf7f48434ffaf03c1950f11a1b1e83bcf5b50bd710af411f88dccc9e" gracePeriod=25 Oct 30 16:50:18.326523 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.326498 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:18.326604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.326526 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:18.326604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.326534 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:18.326604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.326542 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97pn2\" (UniqueName: \"kubernetes.io/projected/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-kube-api-access-97pn2\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:18.326604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.326552 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:18.380427 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.380394 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg" Oct 30 16:50:18.380611 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.380395 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg" event={"ID":"e95bc524-7472-4b9a-a0a5-2f12d396170a","Type":"ContainerDied","Data":"81e07a196db0875ac2ef3699cfdea7e6bf262ec5ccc2ae5668e8686588f61b2a"} Oct 30 16:50:18.380611 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.380566 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81e07a196db0875ac2ef3699cfdea7e6bf262ec5ccc2ae5668e8686588f61b2a" Oct 30 16:50:18.382444 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.382417 6202 generic.go:356] "Generic (PLEG): container finished" podID="d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" containerID="b38baebc14951408e8c0c3a27e9c584d29f4b1ec5695936fa47a2de688cdffc9" exitCode=0 Oct 30 16:50:18.382561 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.382483 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" Oct 30 16:50:18.382561 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.382506 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" event={"ID":"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48","Type":"ContainerDied","Data":"b38baebc14951408e8c0c3a27e9c584d29f4b1ec5695936fa47a2de688cdffc9"} Oct 30 16:50:18.382646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.382563 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl" event={"ID":"d0f45a80-37c0-47fe-95d2-ab1f6da1cd48","Type":"ContainerDied","Data":"bc83259142479de6ee5aa819d429f39ee793101c04c5ecb38f7fb45741171180"} Oct 30 16:50:18.382646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.382593 6202 scope.go:122] "RemoveContainer" containerID="b38baebc14951408e8c0c3a27e9c584d29f4b1ec5695936fa47a2de688cdffc9" Oct 30 16:50:18.432347 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.432326 6202 scope.go:122] "RemoveContainer" containerID="d52da4f3cfbaf5adab63dc37792286a479d63af462dd30f0a734ca29f4e0c7cc" Oct 30 16:50:18.443488 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.443460 6202 scope.go:122] "RemoveContainer" containerID="b38baebc14951408e8c0c3a27e9c584d29f4b1ec5695936fa47a2de688cdffc9" Oct 30 16:50:18.443834 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:50:18.443811 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38baebc14951408e8c0c3a27e9c584d29f4b1ec5695936fa47a2de688cdffc9\": container with ID starting with b38baebc14951408e8c0c3a27e9c584d29f4b1ec5695936fa47a2de688cdffc9 not found: ID does not exist" containerID="b38baebc14951408e8c0c3a27e9c584d29f4b1ec5695936fa47a2de688cdffc9" Oct 30 16:50:18.443928 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.443843 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38baebc14951408e8c0c3a27e9c584d29f4b1ec5695936fa47a2de688cdffc9"} err="failed to get container status \"b38baebc14951408e8c0c3a27e9c584d29f4b1ec5695936fa47a2de688cdffc9\": rpc error: code = NotFound desc = could not find container \"b38baebc14951408e8c0c3a27e9c584d29f4b1ec5695936fa47a2de688cdffc9\": container with ID starting with b38baebc14951408e8c0c3a27e9c584d29f4b1ec5695936fa47a2de688cdffc9 not found: ID does not exist" Oct 30 16:50:18.443928 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.443862 6202 scope.go:122] "RemoveContainer" containerID="d52da4f3cfbaf5adab63dc37792286a479d63af462dd30f0a734ca29f4e0c7cc" Oct 30 16:50:18.444170 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:50:18.444147 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d52da4f3cfbaf5adab63dc37792286a479d63af462dd30f0a734ca29f4e0c7cc\": container with ID starting with d52da4f3cfbaf5adab63dc37792286a479d63af462dd30f0a734ca29f4e0c7cc not found: ID does not exist" containerID="d52da4f3cfbaf5adab63dc37792286a479d63af462dd30f0a734ca29f4e0c7cc" Oct 30 16:50:18.444228 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.444179 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d52da4f3cfbaf5adab63dc37792286a479d63af462dd30f0a734ca29f4e0c7cc"} err="failed to get container status \"d52da4f3cfbaf5adab63dc37792286a479d63af462dd30f0a734ca29f4e0c7cc\": rpc error: code = NotFound desc = could not find container \"d52da4f3cfbaf5adab63dc37792286a479d63af462dd30f0a734ca29f4e0c7cc\": container with ID starting with d52da4f3cfbaf5adab63dc37792286a479d63af462dd30f0a734ca29f4e0c7cc not found: ID does not exist" Oct 30 16:50:18.463502 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.463472 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl"] Oct 30 16:50:18.470572 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.470549 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-968df4-openshift-default-56cf45cb95-zxvxl"] Oct 30 16:50:18.601963 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.601911 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:18.666789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.666503 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9" Oct 30 16:50:18.736959 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.736916 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f816f8bb-62e1-4fa0-abd1-37671f959fa2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f816f8bb-62e1-4fa0-abd1-37671f959fa2-plugins-config-volume\") pod \"f816f8bb-62e1-4fa0-abd1-37671f959fa2\" (UID: \"f816f8bb-62e1-4fa0-abd1-37671f959fa2\") " Oct 30 16:50:18.737180 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.737010 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f816f8bb-62e1-4fa0-abd1-37671f959fa2-kube-api-access-vb8bp\" (UniqueName: \"kubernetes.io/projected/f816f8bb-62e1-4fa0-abd1-37671f959fa2-kube-api-access-vb8bp\") pod \"f816f8bb-62e1-4fa0-abd1-37671f959fa2\" (UID: \"f816f8bb-62e1-4fa0-abd1-37671f959fa2\") " Oct 30 16:50:18.751884 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.741674 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f816f8bb-62e1-4fa0-abd1-37671f959fa2-kube-api-access-vb8bp" pod "f816f8bb-62e1-4fa0-abd1-37671f959fa2" (UID: "f816f8bb-62e1-4fa0-abd1-37671f959fa2"). InnerVolumeSpecName "kube-api-access-vb8bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:18.751884 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.743101 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f816f8bb-62e1-4fa0-abd1-37671f959fa2-plugins-config-volume" pod "f816f8bb-62e1-4fa0-abd1-37671f959fa2" (UID: "f816f8bb-62e1-4fa0-abd1-37671f959fa2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:18.844053 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.843949 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f816f8bb-62e1-4fa0-abd1-37671f959fa2-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:18.844053 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:18.843999 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb8bp\" (UniqueName: \"kubernetes.io/projected/f816f8bb-62e1-4fa0-abd1-37671f959fa2-kube-api-access-vb8bp\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:19.392950 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:19.392917 6202 generic.go:356] "Generic (PLEG): container finished" podID="f816f8bb-62e1-4fa0-abd1-37671f959fa2" containerID="41e25693bf7f48434ffaf03c1950f11a1b1e83bcf5b50bd710af411f88dccc9e" exitCode=0 Oct 30 16:50:19.393165 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:19.392970 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9" Oct 30 16:50:19.393165 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:19.393006 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9" event={"ID":"f816f8bb-62e1-4fa0-abd1-37671f959fa2","Type":"ContainerDied","Data":"41e25693bf7f48434ffaf03c1950f11a1b1e83bcf5b50bd710af411f88dccc9e"} Oct 30 16:50:19.393165 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:19.393065 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9" event={"ID":"f816f8bb-62e1-4fa0-abd1-37671f959fa2","Type":"ContainerDied","Data":"b5bdf45c45a1f0f18ae1cb08cb79efbe6d2a39bafb61601ea339969a02b75bc8"} Oct 30 16:50:19.393165 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:19.393093 6202 scope.go:122] "RemoveContainer" containerID="41e25693bf7f48434ffaf03c1950f11a1b1e83bcf5b50bd710af411f88dccc9e" Oct 30 16:50:19.404744 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:19.404725 6202 scope.go:122] "RemoveContainer" containerID="41e25693bf7f48434ffaf03c1950f11a1b1e83bcf5b50bd710af411f88dccc9e" Oct 30 16:50:19.405068 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:50:19.405047 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e25693bf7f48434ffaf03c1950f11a1b1e83bcf5b50bd710af411f88dccc9e\": container with ID starting with 41e25693bf7f48434ffaf03c1950f11a1b1e83bcf5b50bd710af411f88dccc9e not found: ID does not exist" containerID="41e25693bf7f48434ffaf03c1950f11a1b1e83bcf5b50bd710af411f88dccc9e" Oct 30 16:50:19.405151 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:19.405076 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e25693bf7f48434ffaf03c1950f11a1b1e83bcf5b50bd710af411f88dccc9e"} err="failed to get container status \"41e25693bf7f48434ffaf03c1950f11a1b1e83bcf5b50bd710af411f88dccc9e\": rpc error: code = NotFound desc = could not find container \"41e25693bf7f48434ffaf03c1950f11a1b1e83bcf5b50bd710af411f88dccc9e\": container with ID starting with 41e25693bf7f48434ffaf03c1950f11a1b1e83bcf5b50bd710af411f88dccc9e not found: ID does not exist" Oct 30 16:50:19.447314 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:19.447284 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9"] Oct 30 16:50:19.458460 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:19.458434 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-00fd1a-686899595d-zxsg9"] Oct 30 16:50:19.955625 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:19.955587 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:20.073099 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:20.073054 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:20.198479 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:20.198438 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:20.302747 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:20.302660 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w"] Oct 30 16:50:20.302955 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:20.302926 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" podUID="9f4d8936-b54c-43f0-acfb-6edc1871e260" containerName="istio-proxy" containerID="cri-o://3880465c88ec60d0a7262a495135080e159ca9ed7d1a8eac4f2a040184997b06" gracePeriod=30 Oct 30 16:50:20.355804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:20.351793 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:20.593793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:20.577855 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" path="/var/lib/kubelet/pods/d0f45a80-37c0-47fe-95d2-ab1f6da1cd48/volumes" Oct 30 16:50:20.595490 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:20.595457 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f816f8bb-62e1-4fa0-abd1-37671f959fa2" path="/var/lib/kubelet/pods/f816f8bb-62e1-4fa0-abd1-37671f959fa2/volumes" Oct 30 16:50:20.738360 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:20.738326 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb"] Oct 30 16:50:20.738605 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:20.738577 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" podUID="ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" containerName="istio-proxy" containerID="cri-o://f1490f678d537ce07537b3b72cee7adcd65ddff0cc9c7996e18aa926d57fc4d9" gracePeriod=30 Oct 30 16:50:21.993208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:21.993163 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:23.550243 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:23.550129 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:23.987535 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:23.987498 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx"] Oct 30 16:50:24.022698 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:24.022659 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf"] Oct 30 16:50:24.123195 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:24.123158 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:24.238738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:24.238645 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:25.315253 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.315221 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:25.625156 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.625121 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:50:25.659618 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.659587 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-token\" (UniqueName: \"kubernetes.io/projected/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-token\") pod \"9f4d8936-b54c-43f0-acfb-6edc1871e260\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " Oct 30 16:50:25.659825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.659651 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-credential-socket\") pod \"9f4d8936-b54c-43f0-acfb-6edc1871e260\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " Oct 30 16:50:25.659825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.659717 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4d8936-b54c-43f0-acfb-6edc1871e260-kube-api-access-f8qvp\" (UniqueName: \"kubernetes.io/projected/9f4d8936-b54c-43f0-acfb-6edc1871e260-kube-api-access-f8qvp\") pod \"9f4d8936-b54c-43f0-acfb-6edc1871e260\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " Oct 30 16:50:25.659825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.659789 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f4d8936-b54c-43f0-acfb-6edc1871e260-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4d8936-b54c-43f0-acfb-6edc1871e260-istiod-ca-cert\") pod \"9f4d8936-b54c-43f0-acfb-6edc1871e260\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " Oct 30 16:50:25.660071 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.659868 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-podinfo\") pod \"9f4d8936-b54c-43f0-acfb-6edc1871e260\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " Oct 30 16:50:25.660071 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.660007 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-workload-socket\") pod \"9f4d8936-b54c-43f0-acfb-6edc1871e260\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " Oct 30 16:50:25.660204 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.660131 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-data\") pod \"9f4d8936-b54c-43f0-acfb-6edc1871e260\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " Oct 30 16:50:25.660204 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.660168 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-credential-socket" pod "9f4d8936-b54c-43f0-acfb-6edc1871e260" (UID: "9f4d8936-b54c-43f0-acfb-6edc1871e260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:25.660204 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.660198 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-workload-certs\") pod \"9f4d8936-b54c-43f0-acfb-6edc1871e260\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " Oct 30 16:50:25.660391 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.660298 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-envoy\") pod \"9f4d8936-b54c-43f0-acfb-6edc1871e260\" (UID: \"9f4d8936-b54c-43f0-acfb-6edc1871e260\") " Oct 30 16:50:25.660560 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.660412 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4d8936-b54c-43f0-acfb-6edc1871e260-istiod-ca-cert" pod "9f4d8936-b54c-43f0-acfb-6edc1871e260" (UID: "9f4d8936-b54c-43f0-acfb-6edc1871e260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:25.661345 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.661318 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4d8936-b54c-43f0-acfb-6edc1871e260-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:25.661345 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.661343 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:25.661535 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.660473 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-workload-socket" pod "9f4d8936-b54c-43f0-acfb-6edc1871e260" (UID: "9f4d8936-b54c-43f0-acfb-6edc1871e260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:25.661726 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.661703 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-workload-certs" pod "9f4d8936-b54c-43f0-acfb-6edc1871e260" (UID: "9f4d8936-b54c-43f0-acfb-6edc1871e260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:25.661833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.661792 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-data" pod "9f4d8936-b54c-43f0-acfb-6edc1871e260" (UID: "9f4d8936-b54c-43f0-acfb-6edc1871e260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:25.663881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.663854 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-envoy" pod "9f4d8936-b54c-43f0-acfb-6edc1871e260" (UID: "9f4d8936-b54c-43f0-acfb-6edc1871e260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:25.663881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.663867 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-podinfo" pod "9f4d8936-b54c-43f0-acfb-6edc1871e260" (UID: "9f4d8936-b54c-43f0-acfb-6edc1871e260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:25.664054 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.663899 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-token" pod "9f4d8936-b54c-43f0-acfb-6edc1871e260" (UID: "9f4d8936-b54c-43f0-acfb-6edc1871e260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:25.664209 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.664191 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4d8936-b54c-43f0-acfb-6edc1871e260-kube-api-access-f8qvp" pod "9f4d8936-b54c-43f0-acfb-6edc1871e260" (UID: "9f4d8936-b54c-43f0-acfb-6edc1871e260"). InnerVolumeSpecName "kube-api-access-f8qvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:25.673236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.673212 6202 generic.go:356] "Generic (PLEG): container finished" podID="9f4d8936-b54c-43f0-acfb-6edc1871e260" containerID="3880465c88ec60d0a7262a495135080e159ca9ed7d1a8eac4f2a040184997b06" exitCode=0 Oct 30 16:50:25.673343 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.673267 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" Oct 30 16:50:25.673343 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.673295 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" event={"ID":"9f4d8936-b54c-43f0-acfb-6edc1871e260","Type":"ContainerDied","Data":"3880465c88ec60d0a7262a495135080e159ca9ed7d1a8eac4f2a040184997b06"} Oct 30 16:50:25.673443 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.673352 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w" event={"ID":"9f4d8936-b54c-43f0-acfb-6edc1871e260","Type":"ContainerDied","Data":"6451d4e409297182e57978ae41b9b6f71495da38c5b4c64adf6bc8cf88898f38"} Oct 30 16:50:25.673443 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.673383 6202 scope.go:122] "RemoveContainer" containerID="3880465c88ec60d0a7262a495135080e159ca9ed7d1a8eac4f2a040184997b06" Oct 30 16:50:25.709033 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.709008 6202 scope.go:122] "RemoveContainer" containerID="02e9dd07c45d190160cf388a4b085dff2107acc4eeded43bc3b55a1a39b826bb" Oct 30 16:50:25.720951 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.720880 6202 scope.go:122] "RemoveContainer" containerID="3880465c88ec60d0a7262a495135080e159ca9ed7d1a8eac4f2a040184997b06" Oct 30 16:50:25.721326 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:50:25.721294 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3880465c88ec60d0a7262a495135080e159ca9ed7d1a8eac4f2a040184997b06\": container with ID starting with 3880465c88ec60d0a7262a495135080e159ca9ed7d1a8eac4f2a040184997b06 not found: ID does not exist" containerID="3880465c88ec60d0a7262a495135080e159ca9ed7d1a8eac4f2a040184997b06" Oct 30 16:50:25.721476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.721337 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3880465c88ec60d0a7262a495135080e159ca9ed7d1a8eac4f2a040184997b06"} err="failed to get container status \"3880465c88ec60d0a7262a495135080e159ca9ed7d1a8eac4f2a040184997b06\": rpc error: code = NotFound desc = could not find container \"3880465c88ec60d0a7262a495135080e159ca9ed7d1a8eac4f2a040184997b06\": container with ID starting with 3880465c88ec60d0a7262a495135080e159ca9ed7d1a8eac4f2a040184997b06 not found: ID does not exist" Oct 30 16:50:25.721476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.721361 6202 scope.go:122] "RemoveContainer" containerID="02e9dd07c45d190160cf388a4b085dff2107acc4eeded43bc3b55a1a39b826bb" Oct 30 16:50:25.721659 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:50:25.721641 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e9dd07c45d190160cf388a4b085dff2107acc4eeded43bc3b55a1a39b826bb\": container with ID starting with 02e9dd07c45d190160cf388a4b085dff2107acc4eeded43bc3b55a1a39b826bb not found: ID does not exist" containerID="02e9dd07c45d190160cf388a4b085dff2107acc4eeded43bc3b55a1a39b826bb" Oct 30 16:50:25.721726 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.721667 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e9dd07c45d190160cf388a4b085dff2107acc4eeded43bc3b55a1a39b826bb"} err="failed to get container status \"02e9dd07c45d190160cf388a4b085dff2107acc4eeded43bc3b55a1a39b826bb\": rpc error: code = NotFound desc = could not find container \"02e9dd07c45d190160cf388a4b085dff2107acc4eeded43bc3b55a1a39b826bb\": container with ID starting with 02e9dd07c45d190160cf388a4b085dff2107acc4eeded43bc3b55a1a39b826bb not found: ID does not exist" Oct 30 16:50:25.744752 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.744722 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w"] Oct 30 16:50:25.750276 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.750249 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b315d4-openshift-default-698f5cfcc5-qb59w"] Oct 30 16:50:25.763671 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.763143 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:25.763671 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.763171 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:25.763671 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.763185 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:25.763671 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.763197 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:25.763671 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.763209 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8qvp\" (UniqueName: \"kubernetes.io/projected/9f4d8936-b54c-43f0-acfb-6edc1871e260-kube-api-access-f8qvp\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:25.763671 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.763222 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4d8936-b54c-43f0-acfb-6edc1871e260-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:25.763671 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:25.763236 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4d8936-b54c-43f0-acfb-6edc1871e260-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.017701 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.017670 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:50:26.067617 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.067581 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-data\") pod \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " Oct 30 16:50:26.067818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.067704 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-envoy\") pod \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " Oct 30 16:50:26.067913 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.067837 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-credential-socket\") pod \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " Oct 30 16:50:26.067913 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.067884 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-data" pod "ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" (UID: "ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.068055 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.067894 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istiod-ca-cert\") pod \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " Oct 30 16:50:26.068193 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.068152 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-podinfo\") pod \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " Oct 30 16:50:26.068294 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.068193 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-workload-certs\") pod \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " Oct 30 16:50:26.068294 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.068230 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-kube-api-access-rt26z\" (UniqueName: \"kubernetes.io/projected/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-kube-api-access-rt26z\") pod \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " Oct 30 16:50:26.068419 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.068289 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istiod-ca-cert" pod "ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" (UID: "ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:26.068419 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.068305 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-token\" (UniqueName: \"kubernetes.io/projected/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-token\") pod \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " Oct 30 16:50:26.068525 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.068451 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-workload-socket\") pod \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\" (UID: \"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6\") " Oct 30 16:50:26.068525 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.068486 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-credential-socket" pod "ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" (UID: "ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.069052 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.069015 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-workload-socket" pod "ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" (UID: "ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.069268 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.069230 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-workload-certs" pod "ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" (UID: "ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.070375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.069917 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.070375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.069941 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.070375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.069959 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.070375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.069973 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.070375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.069985 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.071414 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.071385 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-kube-api-access-rt26z" pod "ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" (UID: "ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6"). InnerVolumeSpecName "kube-api-access-rt26z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:26.071668 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.071637 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-podinfo" pod "ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" (UID: "ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:26.073278 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.073253 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-token" pod "ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" (UID: "ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:26.073611 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.073584 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-envoy" pod "ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" (UID: "ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.171032 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.170905 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.171032 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.170934 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.171032 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.170943 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt26z\" (UniqueName: \"kubernetes.io/projected/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-kube-api-access-rt26z\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.171032 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.170952 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.415695 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.415665 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f4d8936-b54c-43f0-acfb-6edc1871e260" path="/var/lib/kubelet/pods/9f4d8936-b54c-43f0-acfb-6edc1871e260/volumes" Oct 30 16:50:26.683755 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.683649 6202 generic.go:356] "Generic (PLEG): container finished" podID="ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" containerID="f1490f678d537ce07537b3b72cee7adcd65ddff0cc9c7996e18aa926d57fc4d9" exitCode=0 Oct 30 16:50:26.683755 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.683717 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" Oct 30 16:50:26.683755 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.683730 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" event={"ID":"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6","Type":"ContainerDied","Data":"f1490f678d537ce07537b3b72cee7adcd65ddff0cc9c7996e18aa926d57fc4d9"} Oct 30 16:50:26.684075 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.683831 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb" event={"ID":"ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6","Type":"ContainerDied","Data":"9419b449f9cdd53ac2de07f10c4d05a7ee979a5c70aaae6c08571c5f7e7e239e"} Oct 30 16:50:26.684075 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.683870 6202 scope.go:122] "RemoveContainer" containerID="f1490f678d537ce07537b3b72cee7adcd65ddff0cc9c7996e18aa926d57fc4d9" Oct 30 16:50:26.695747 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.695727 6202 scope.go:122] "RemoveContainer" containerID="f1490f678d537ce07537b3b72cee7adcd65ddff0cc9c7996e18aa926d57fc4d9" Oct 30 16:50:26.696090 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:50:26.696063 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1490f678d537ce07537b3b72cee7adcd65ddff0cc9c7996e18aa926d57fc4d9\": container with ID starting with f1490f678d537ce07537b3b72cee7adcd65ddff0cc9c7996e18aa926d57fc4d9 not found: ID does not exist" containerID="f1490f678d537ce07537b3b72cee7adcd65ddff0cc9c7996e18aa926d57fc4d9" Oct 30 16:50:26.696198 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.696100 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1490f678d537ce07537b3b72cee7adcd65ddff0cc9c7996e18aa926d57fc4d9"} err="failed to get container status \"f1490f678d537ce07537b3b72cee7adcd65ddff0cc9c7996e18aa926d57fc4d9\": rpc error: code = NotFound desc = could not find container \"f1490f678d537ce07537b3b72cee7adcd65ddff0cc9c7996e18aa926d57fc4d9\": container with ID starting with f1490f678d537ce07537b3b72cee7adcd65ddff0cc9c7996e18aa926d57fc4d9 not found: ID does not exist" Oct 30 16:50:26.739715 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.739687 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb"] Oct 30 16:50:26.753373 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:26.753343 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-w2mwb"] Oct 30 16:50:27.174572 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:27.174530 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:28.411936 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:28.411902 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" path="/var/lib/kubelet/pods/ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6/volumes" Oct 30 16:50:28.735033 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:28.734950 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:28.988944 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:28.988846 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx" podUID="4830c6cf-3abd-485a-87ee-6b094461b5ae" containerName="payload-processing" containerID="cri-o://e4aa1fc29c5328b22696e1a48555f4126f624559bf4670ac1f1a6df5377bea30" gracePeriod=25 Oct 30 16:50:29.023645 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.023610 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf" podUID="b37d7321-2cef-42aa-80ff-515da8f8a2df" containerName="payload-pre-processing" containerID="cri-o://aed1c1229a836144f60af4b5cac1836ed16f675491117e0748b7d41cc62e733d" gracePeriod=25 Oct 30 16:50:29.360207 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.360180 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx" Oct 30 16:50:29.364320 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.364298 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf" Oct 30 16:50:29.419555 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.419519 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37d7321-2cef-42aa-80ff-515da8f8a2df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37d7321-2cef-42aa-80ff-515da8f8a2df-plugins-config-volume\") pod \"b37d7321-2cef-42aa-80ff-515da8f8a2df\" (UID: \"b37d7321-2cef-42aa-80ff-515da8f8a2df\") " Oct 30 16:50:29.420094 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.419633 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4830c6cf-3abd-485a-87ee-6b094461b5ae-kube-api-access-jxrsp\" (UniqueName: \"kubernetes.io/projected/4830c6cf-3abd-485a-87ee-6b094461b5ae-kube-api-access-jxrsp\") pod \"4830c6cf-3abd-485a-87ee-6b094461b5ae\" (UID: \"4830c6cf-3abd-485a-87ee-6b094461b5ae\") " Oct 30 16:50:29.420094 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.419685 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37d7321-2cef-42aa-80ff-515da8f8a2df-kube-api-access-wq9c9\" (UniqueName: \"kubernetes.io/projected/b37d7321-2cef-42aa-80ff-515da8f8a2df-kube-api-access-wq9c9\") pod \"b37d7321-2cef-42aa-80ff-515da8f8a2df\" (UID: \"b37d7321-2cef-42aa-80ff-515da8f8a2df\") " Oct 30 16:50:29.420094 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.419842 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4830c6cf-3abd-485a-87ee-6b094461b5ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4830c6cf-3abd-485a-87ee-6b094461b5ae-plugins-config-volume\") pod \"4830c6cf-3abd-485a-87ee-6b094461b5ae\" (UID: \"4830c6cf-3abd-485a-87ee-6b094461b5ae\") " Oct 30 16:50:29.420094 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.419875 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37d7321-2cef-42aa-80ff-515da8f8a2df-plugins-config-volume" pod "b37d7321-2cef-42aa-80ff-515da8f8a2df" (UID: "b37d7321-2cef-42aa-80ff-515da8f8a2df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:29.420297 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.420233 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4830c6cf-3abd-485a-87ee-6b094461b5ae-plugins-config-volume" pod "4830c6cf-3abd-485a-87ee-6b094461b5ae" (UID: "4830c6cf-3abd-485a-87ee-6b094461b5ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:29.420410 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.420393 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4830c6cf-3abd-485a-87ee-6b094461b5ae-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:29.420467 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.420412 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37d7321-2cef-42aa-80ff-515da8f8a2df-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:29.422503 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.422474 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4830c6cf-3abd-485a-87ee-6b094461b5ae-kube-api-access-jxrsp" pod "4830c6cf-3abd-485a-87ee-6b094461b5ae" (UID: "4830c6cf-3abd-485a-87ee-6b094461b5ae"). InnerVolumeSpecName "kube-api-access-jxrsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:29.422637 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.422569 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37d7321-2cef-42aa-80ff-515da8f8a2df-kube-api-access-wq9c9" pod "b37d7321-2cef-42aa-80ff-515da8f8a2df" (UID: "b37d7321-2cef-42aa-80ff-515da8f8a2df"). InnerVolumeSpecName "kube-api-access-wq9c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:29.522355 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.522316 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxrsp\" (UniqueName: \"kubernetes.io/projected/4830c6cf-3abd-485a-87ee-6b094461b5ae-kube-api-access-jxrsp\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:29.522355 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.522344 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq9c9\" (UniqueName: \"kubernetes.io/projected/b37d7321-2cef-42aa-80ff-515da8f8a2df-kube-api-access-wq9c9\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:29.712099 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.712065 6202 generic.go:356] "Generic (PLEG): container finished" podID="b37d7321-2cef-42aa-80ff-515da8f8a2df" containerID="aed1c1229a836144f60af4b5cac1836ed16f675491117e0748b7d41cc62e733d" exitCode=0 Oct 30 16:50:29.712284 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.712128 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf" Oct 30 16:50:29.712284 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.712155 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf" event={"ID":"b37d7321-2cef-42aa-80ff-515da8f8a2df","Type":"ContainerDied","Data":"aed1c1229a836144f60af4b5cac1836ed16f675491117e0748b7d41cc62e733d"} Oct 30 16:50:29.712284 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.712206 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf" event={"ID":"b37d7321-2cef-42aa-80ff-515da8f8a2df","Type":"ContainerDied","Data":"48a9bcc0effdf9bb748bd184fc859d814009d6f71ee509f5aa434e3aff45d9f6"} Oct 30 16:50:29.712284 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.712235 6202 scope.go:122] "RemoveContainer" containerID="aed1c1229a836144f60af4b5cac1836ed16f675491117e0748b7d41cc62e733d" Oct 30 16:50:29.713817 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.713790 6202 generic.go:356] "Generic (PLEG): container finished" podID="4830c6cf-3abd-485a-87ee-6b094461b5ae" containerID="e4aa1fc29c5328b22696e1a48555f4126f624559bf4670ac1f1a6df5377bea30" exitCode=0 Oct 30 16:50:29.713926 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.713833 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx" event={"ID":"4830c6cf-3abd-485a-87ee-6b094461b5ae","Type":"ContainerDied","Data":"e4aa1fc29c5328b22696e1a48555f4126f624559bf4670ac1f1a6df5377bea30"} Oct 30 16:50:29.713926 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.713866 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx" Oct 30 16:50:29.713926 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.713878 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx" event={"ID":"4830c6cf-3abd-485a-87ee-6b094461b5ae","Type":"ContainerDied","Data":"46eee561a20f5fe23d1a31c37a6147251fcd854c3b0d3a77c27fdf70d3a84973"} Oct 30 16:50:29.725977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.725962 6202 scope.go:122] "RemoveContainer" containerID="aed1c1229a836144f60af4b5cac1836ed16f675491117e0748b7d41cc62e733d" Oct 30 16:50:29.726228 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:50:29.726209 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aed1c1229a836144f60af4b5cac1836ed16f675491117e0748b7d41cc62e733d\": container with ID starting with aed1c1229a836144f60af4b5cac1836ed16f675491117e0748b7d41cc62e733d not found: ID does not exist" containerID="aed1c1229a836144f60af4b5cac1836ed16f675491117e0748b7d41cc62e733d" Oct 30 16:50:29.726316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.726240 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aed1c1229a836144f60af4b5cac1836ed16f675491117e0748b7d41cc62e733d"} err="failed to get container status \"aed1c1229a836144f60af4b5cac1836ed16f675491117e0748b7d41cc62e733d\": rpc error: code = NotFound desc = could not find container \"aed1c1229a836144f60af4b5cac1836ed16f675491117e0748b7d41cc62e733d\": container with ID starting with aed1c1229a836144f60af4b5cac1836ed16f675491117e0748b7d41cc62e733d not found: ID does not exist" Oct 30 16:50:29.726316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.726263 6202 scope.go:122] "RemoveContainer" containerID="e4aa1fc29c5328b22696e1a48555f4126f624559bf4670ac1f1a6df5377bea30" Oct 30 16:50:29.736496 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.736479 6202 scope.go:122] "RemoveContainer" containerID="e4aa1fc29c5328b22696e1a48555f4126f624559bf4670ac1f1a6df5377bea30" Oct 30 16:50:29.736801 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:50:29.736756 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4aa1fc29c5328b22696e1a48555f4126f624559bf4670ac1f1a6df5377bea30\": container with ID starting with e4aa1fc29c5328b22696e1a48555f4126f624559bf4670ac1f1a6df5377bea30 not found: ID does not exist" containerID="e4aa1fc29c5328b22696e1a48555f4126f624559bf4670ac1f1a6df5377bea30" Oct 30 16:50:29.736877 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.736811 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4aa1fc29c5328b22696e1a48555f4126f624559bf4670ac1f1a6df5377bea30"} err="failed to get container status \"e4aa1fc29c5328b22696e1a48555f4126f624559bf4670ac1f1a6df5377bea30\": rpc error: code = NotFound desc = could not find container \"e4aa1fc29c5328b22696e1a48555f4126f624559bf4670ac1f1a6df5377bea30\": container with ID starting with e4aa1fc29c5328b22696e1a48555f4126f624559bf4670ac1f1a6df5377bea30 not found: ID does not exist" Oct 30 16:50:29.766809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.766780 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf"] Oct 30 16:50:29.777824 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.777803 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e5674c-5d6f4cd555-7twxf"] Oct 30 16:50:29.801497 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.801475 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx"] Oct 30 16:50:29.805435 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:29.805415 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e5674c-7589b958dd-pbprx"] Oct 30 16:50:30.320308 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.320223 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:30.414080 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.414038 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4830c6cf-3abd-485a-87ee-6b094461b5ae" path="/var/lib/kubelet/pods/4830c6cf-3abd-485a-87ee-6b094461b5ae/volumes" Oct 30 16:50:30.414543 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.414528 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37d7321-2cef-42aa-80ff-515da8f8a2df" path="/var/lib/kubelet/pods/b37d7321-2cef-42aa-80ff-515da8f8a2df/volumes" Oct 30 16:50:30.873308 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.873263 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb"] Oct 30 16:50:30.874112 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874093 6202 cpu_manager.go:414] "containerMap: removing container" podUID="d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" containerName="istio-proxy" Oct 30 16:50:30.874112 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874112 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" containerName="istio-proxy" Oct 30 16:50:30.874271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874136 6202 cpu_manager.go:414] "containerMap: removing container" podUID="d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" containerName="istio-proxy" Oct 30 16:50:30.874271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874142 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" containerName="istio-proxy" Oct 30 16:50:30.874271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874150 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9f4d8936-b54c-43f0-acfb-6edc1871e260" containerName="istio-proxy" Oct 30 16:50:30.874271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874155 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4d8936-b54c-43f0-acfb-6edc1871e260" containerName="istio-proxy" Oct 30 16:50:30.874271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874175 6202 cpu_manager.go:414] "containerMap: removing container" podUID="b37d7321-2cef-42aa-80ff-515da8f8a2df" containerName="payload-pre-processing" Oct 30 16:50:30.874271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874179 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37d7321-2cef-42aa-80ff-515da8f8a2df" containerName="payload-pre-processing" Oct 30 16:50:30.874271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874195 6202 cpu_manager.go:414] "containerMap: removing container" podUID="e95bc524-7472-4b9a-a0a5-2f12d396170a" containerName="revoke-keys" Oct 30 16:50:30.874271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874203 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e95bc524-7472-4b9a-a0a5-2f12d396170a" containerName="revoke-keys" Oct 30 16:50:30.874271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874214 6202 cpu_manager.go:414] "containerMap: removing container" podUID="f816f8bb-62e1-4fa0-abd1-37671f959fa2" containerName="payload-processing" Oct 30 16:50:30.874271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874221 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f816f8bb-62e1-4fa0-abd1-37671f959fa2" containerName="payload-processing" Oct 30 16:50:30.874271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874228 6202 cpu_manager.go:414] "containerMap: removing container" podUID="4830c6cf-3abd-485a-87ee-6b094461b5ae" containerName="payload-processing" Oct 30 16:50:30.874271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874232 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4830c6cf-3abd-485a-87ee-6b094461b5ae" containerName="payload-processing" Oct 30 16:50:30.874271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874246 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9f4d8936-b54c-43f0-acfb-6edc1871e260" containerName="istio-proxy" Oct 30 16:50:30.874271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874251 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4d8936-b54c-43f0-acfb-6edc1871e260" containerName="istio-proxy" Oct 30 16:50:30.874271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874262 6202 cpu_manager.go:414] "containerMap: removing container" podUID="ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" containerName="istio-proxy" Oct 30 16:50:30.874271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874266 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" containerName="istio-proxy" Oct 30 16:50:30.875069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874371 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" containerName="istio-proxy" Oct 30 16:50:30.875069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874378 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4d8936-b54c-43f0-acfb-6edc1871e260" containerName="istio-proxy" Oct 30 16:50:30.875069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874388 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37d7321-2cef-42aa-80ff-515da8f8a2df" containerName="payload-pre-processing" Oct 30 16:50:30.875069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874397 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="e95bc524-7472-4b9a-a0a5-2f12d396170a" containerName="revoke-keys" Oct 30 16:50:30.875069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874403 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="f816f8bb-62e1-4fa0-abd1-37671f959fa2" containerName="payload-processing" Oct 30 16:50:30.875069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874409 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="4830c6cf-3abd-485a-87ee-6b094461b5ae" containerName="payload-processing" Oct 30 16:50:30.875069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874416 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4d8936-b54c-43f0-acfb-6edc1871e260" containerName="istio-proxy" Oct 30 16:50:30.875069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874423 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffcf87ea-7a55-4f30-9349-b50f3f1bf5f6" containerName="istio-proxy" Oct 30 16:50:30.875069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.874430 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f45a80-37c0-47fe-95d2-ab1f6da1cd48" containerName="istio-proxy" Oct 30 16:50:30.879949 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.879920 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb" Oct 30 16:50:30.908612 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.908579 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:50:30.908843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.908649 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:50:30.908979 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.908916 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:50:30.909073 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.908973 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:50:30.915062 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.915037 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb"] Oct 30 16:50:30.942162 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.942127 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065c113c-8979-4fde-871d-3f88dea368b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb\" (UID: \"065c113c-8979-4fde-871d-3f88dea368b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb" Oct 30 16:50:30.942355 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.942193 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9hnd\" (UniqueName: \"kubernetes.io/projected/065c113c-8979-4fde-871d-3f88dea368b3-kube-api-access-w9hnd\") pod \"maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb\" (UID: \"065c113c-8979-4fde-871d-3f88dea368b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb" Oct 30 16:50:30.942449 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:30.942427 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065c113c-8979-4fde-871d-3f88dea368b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb\" (UID: \"065c113c-8979-4fde-871d-3f88dea368b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb" Oct 30 16:50:31.044086 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.044048 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065c113c-8979-4fde-871d-3f88dea368b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb\" (UID: \"065c113c-8979-4fde-871d-3f88dea368b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb" Oct 30 16:50:31.044281 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.044117 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9hnd\" (UniqueName: \"kubernetes.io/projected/065c113c-8979-4fde-871d-3f88dea368b3-kube-api-access-w9hnd\") pod \"maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb\" (UID: \"065c113c-8979-4fde-871d-3f88dea368b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb" Oct 30 16:50:31.044341 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.044328 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065c113c-8979-4fde-871d-3f88dea368b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb\" (UID: \"065c113c-8979-4fde-871d-3f88dea368b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb" Oct 30 16:50:31.044930 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.044900 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065c113c-8979-4fde-871d-3f88dea368b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb\" (UID: \"065c113c-8979-4fde-871d-3f88dea368b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb" Oct 30 16:50:31.046989 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.046960 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065c113c-8979-4fde-871d-3f88dea368b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb\" (UID: \"065c113c-8979-4fde-871d-3f88dea368b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb" Oct 30 16:50:31.068349 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.068320 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9hnd\" (UniqueName: \"kubernetes.io/projected/065c113c-8979-4fde-871d-3f88dea368b3-kube-api-access-w9hnd\") pod \"maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb\" (UID: \"065c113c-8979-4fde-871d-3f88dea368b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb" Oct 30 16:50:31.186833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.186723 6202 scope.go:122] "RemoveContainer" containerID="9f5a6a44502b4bcde7923565e2eb996c22a514eae58bcb07beae428204d57a44" Oct 30 16:50:31.197012 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.196984 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb" Oct 30 16:50:31.197427 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.197405 6202 scope.go:122] "RemoveContainer" containerID="d36ae674b90a266add680a7682208c1a55f3c7044f9a18c43bd62aff6471425b" Oct 30 16:50:31.516121 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:50:31.516086 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065c113c_8979_4fde_871d_3f88dea368b3.slice/crio-5a206ae314babffa024cc2ac48fd0435d9d5642b87cb9b654287a2abcc34bbca WatchSource:0}: Error finding container 5a206ae314babffa024cc2ac48fd0435d9d5642b87cb9b654287a2abcc34bbca: Status 404 returned error can't find the container with id 5a206ae314babffa024cc2ac48fd0435d9d5642b87cb9b654287a2abcc34bbca Oct 30 16:50:31.517637 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.517525 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb"] Oct 30 16:50:31.693411 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.693377 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr"] Oct 30 16:50:31.699185 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.699166 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.721590 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.721560 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr"] Oct 30 16:50:31.760791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.760314 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb" event={"ID":"065c113c-8979-4fde-871d-3f88dea368b3","Type":"ContainerStarted","Data":"5a206ae314babffa024cc2ac48fd0435d9d5642b87cb9b654287a2abcc34bbca"} Oct 30 16:50:31.760791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.760400 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53877150-e074-4622-afb6-eb0699548ecb-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.760791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.760553 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53877150-e074-4622-afb6-eb0699548ecb-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.760791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.760658 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn9td\" (UniqueName: \"kubernetes.io/projected/53877150-e074-4622-afb6-eb0699548ecb-kube-api-access-dn9td\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.760791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.760722 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53877150-e074-4622-afb6-eb0699548ecb-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.760791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.760752 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.761226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.760804 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.761226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.760987 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.761226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.761083 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.761226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.761103 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.863033 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.863004 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53877150-e074-4622-afb6-eb0699548ecb-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.863215 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.863043 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53877150-e074-4622-afb6-eb0699548ecb-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.863215 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.863132 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn9td\" (UniqueName: \"kubernetes.io/projected/53877150-e074-4622-afb6-eb0699548ecb-kube-api-access-dn9td\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.863215 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.863197 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53877150-e074-4622-afb6-eb0699548ecb-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.863411 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.863240 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.863411 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.863283 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.863411 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.863402 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.863563 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.863485 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.863563 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.863516 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.863809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.863781 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53877150-e074-4622-afb6-eb0699548ecb-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.863891 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.863817 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.864205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.864181 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.864592 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.864518 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.864706 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.864538 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.866040 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.866014 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.866167 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.866151 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53877150-e074-4622-afb6-eb0699548ecb-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.886891 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.886862 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53877150-e074-4622-afb6-eb0699548ecb-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:31.887297 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:31.886932 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn9td\" (UniqueName: \"kubernetes.io/projected/53877150-e074-4622-afb6-eb0699548ecb-kube-api-access-dn9td\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:32.015154 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:32.015126 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:32.029214 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:32.029184 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:32.247457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:32.242914 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:32.342826 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:50:32.342789 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53877150_e074_4622_afb6_eb0699548ecb.slice/crio-604831d265cfc2fd712e5e4d3d4d1aa18df5d7af2598284cb00f1cbc90d0ade2 WatchSource:0}: Error finding container 604831d265cfc2fd712e5e4d3d4d1aa18df5d7af2598284cb00f1cbc90d0ade2: Status 404 returned error can't find the container with id 604831d265cfc2fd712e5e4d3d4d1aa18df5d7af2598284cb00f1cbc90d0ade2 Oct 30 16:50:32.343385 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:32.343357 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr"] Oct 30 16:50:32.346484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:32.346447 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:50:32.346598 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:32.346519 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:50:32.346598 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:32.346546 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:50:32.374707 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:32.374672 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:32.771610 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:32.771575 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" event={"ID":"53877150-e074-4622-afb6-eb0699548ecb","Type":"ContainerStarted","Data":"e372756145a9b4243d25c1b2a64eb92d630e8f9bf3706ffa27e7d9a9560e9ef5"} Oct 30 16:50:32.771813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:32.771638 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" event={"ID":"53877150-e074-4622-afb6-eb0699548ecb","Type":"ContainerStarted","Data":"604831d265cfc2fd712e5e4d3d4d1aa18df5d7af2598284cb00f1cbc90d0ade2"} Oct 30 16:50:32.773794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:32.773747 6202 generic.go:356] "Generic (PLEG): container finished" podID="065c113c-8979-4fde-871d-3f88dea368b3" containerID="8da7a577762116c664954402f8c6282459daaa5a3ffedadf2ba95d68e088ed02" exitCode=0 Oct 30 16:50:32.773908 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:32.773834 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb" event={"ID":"065c113c-8979-4fde-871d-3f88dea368b3","Type":"ContainerDied","Data":"8da7a577762116c664954402f8c6282459daaa5a3ffedadf2ba95d68e088ed02"} Oct 30 16:50:32.821943 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:32.821891 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" podStartSLOduration=1.821876133 podStartE2EDuration="1.821876133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:32.818661283 +0000 UTC m=+1746.529116027" watchObservedRunningTime="2026-10-30 16:50:32.821876133 +0000 UTC m=+1746.532330944" Oct 30 16:50:33.015758 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:33.015672 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:33.017908 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:33.017875 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 16:50:33.018038 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:33.017934 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" podUID="53877150-e074-4622-afb6-eb0699548ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 16:50:34.016023 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.015992 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 16:50:34.016357 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.016049 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" podUID="53877150-e074-4622-afb6-eb0699548ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 16:50:34.017079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.017060 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb" Oct 30 16:50:34.168879 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.162888 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/065c113c-8979-4fde-871d-3f88dea368b3-kube-api-access-w9hnd\" (UniqueName: \"kubernetes.io/projected/065c113c-8979-4fde-871d-3f88dea368b3-kube-api-access-w9hnd\") pod \"065c113c-8979-4fde-871d-3f88dea368b3\" (UID: \"065c113c-8979-4fde-871d-3f88dea368b3\") " Oct 30 16:50:34.168879 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.163034 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/065c113c-8979-4fde-871d-3f88dea368b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065c113c-8979-4fde-871d-3f88dea368b3-maas-api-xks-ca\") pod \"065c113c-8979-4fde-871d-3f88dea368b3\" (UID: \"065c113c-8979-4fde-871d-3f88dea368b3\") " Oct 30 16:50:34.168879 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.163067 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/065c113c-8979-4fde-871d-3f88dea368b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065c113c-8979-4fde-871d-3f88dea368b3-maas-api-openshift-ca\") pod \"065c113c-8979-4fde-871d-3f88dea368b3\" (UID: \"065c113c-8979-4fde-871d-3f88dea368b3\") " Oct 30 16:50:34.168879 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.164338 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065c113c-8979-4fde-871d-3f88dea368b3-maas-api-openshift-ca" pod "065c113c-8979-4fde-871d-3f88dea368b3" (UID: "065c113c-8979-4fde-871d-3f88dea368b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:34.175826 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.174297 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/065c113c-8979-4fde-871d-3f88dea368b3-maas-api-xks-ca" pod "065c113c-8979-4fde-871d-3f88dea368b3" (UID: "065c113c-8979-4fde-871d-3f88dea368b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:34.175826 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.174898 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065c113c-8979-4fde-871d-3f88dea368b3-kube-api-access-w9hnd" pod "065c113c-8979-4fde-871d-3f88dea368b3" (UID: "065c113c-8979-4fde-871d-3f88dea368b3"). InnerVolumeSpecName "kube-api-access-w9hnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:34.263867 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.262375 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:34.271049 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.271012 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9hnd\" (UniqueName: \"kubernetes.io/projected/065c113c-8979-4fde-871d-3f88dea368b3-kube-api-access-w9hnd\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:34.271208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.271052 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065c113c-8979-4fde-871d-3f88dea368b3-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:34.271208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.271070 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065c113c-8979-4fde-871d-3f88dea368b3-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:34.436319 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.436288 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:34.800251 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.800166 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb" Oct 30 16:50:34.800251 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.800190 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb" event={"ID":"065c113c-8979-4fde-871d-3f88dea368b3","Type":"ContainerDied","Data":"5a206ae314babffa024cc2ac48fd0435d9d5642b87cb9b654287a2abcc34bbca"} Oct 30 16:50:34.800251 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.800240 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a206ae314babffa024cc2ac48fd0435d9d5642b87cb9b654287a2abcc34bbca" Oct 30 16:50:34.995659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.995612 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh"] Oct 30 16:50:34.997076 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.997043 6202 cpu_manager.go:414] "containerMap: removing container" podUID="065c113c-8979-4fde-871d-3f88dea368b3" containerName="revoke-keys" Oct 30 16:50:34.997076 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.997074 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="065c113c-8979-4fde-871d-3f88dea368b3" containerName="revoke-keys" Oct 30 16:50:34.997339 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:34.997309 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="065c113c-8979-4fde-871d-3f88dea368b3" containerName="revoke-keys" Oct 30 16:50:35.001578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.001555 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.016040 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.016016 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 16:50:35.016370 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.016059 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" podUID="53877150-e074-4622-afb6-eb0699548ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 16:50:35.035402 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.035371 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh"] Oct 30 16:50:35.152792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.151032 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-668047-openshift-default-dockercfg-7bjgw\"" Oct 30 16:50:35.239792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.239326 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:35.259909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.259873 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-workload-socket\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.260437 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.260414 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-data\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.260674 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.260655 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-podinfo\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.260833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.260815 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-envoy\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.261035 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.261018 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkrjw\" (UniqueName: \"kubernetes.io/projected/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-kube-api-access-vkrjw\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.261281 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.261260 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-token\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.275935 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.273544 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istiod-ca-cert\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.282900 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.282868 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-credential-socket\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.295627 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.295589 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-workload-certs\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.415843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.411966 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-credential-socket\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.415843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.412043 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-workload-certs\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.415843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.412086 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-workload-socket\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.415843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.412262 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-data\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.415843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.412335 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-podinfo\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.415843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.412363 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-envoy\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.415843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.412416 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkrjw\" (UniqueName: \"kubernetes.io/projected/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-kube-api-access-vkrjw\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.415843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.412492 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-token\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.415843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.412611 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istiod-ca-cert\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.415843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.413511 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istiod-ca-cert\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.415843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.413823 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-credential-socket\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.421807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.419073 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-workload-certs\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.421807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.419316 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-workload-socket\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.421807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.419551 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-data\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.446791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.439256 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:35.446976 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.446923 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-podinfo\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.467803 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.465062 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-envoy\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.473350 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.473275 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-token\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.515790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.506724 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkrjw\" (UniqueName: \"kubernetes.io/projected/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-kube-api-access-vkrjw\") pod \"e2e-inf-a-668047-openshift-default-c49c74d65-p65gh\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.616683 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.616648 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:35.702100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.701994 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt"] Oct 30 16:50:35.708721 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.708691 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt" Oct 30 16:50:35.727329 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.725295 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:50:35.727329 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.725963 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:50:35.792790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.763457 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt"] Oct 30 16:50:35.796047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.795104 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:50:35.796047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.795154 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:50:35.830354 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.830315 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8af660d-5dswt\" (UID: \"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt" Oct 30 16:50:35.830523 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.830500 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8af660d-5dswt\" (UID: \"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt" Oct 30 16:50:35.830638 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.830622 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m46z\" (UniqueName: \"kubernetes.io/projected/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-kube-api-access-4m46z\") pod \"maas-api-revoke-keys-e2e-mt-b8af660d-5dswt\" (UID: \"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt" Oct 30 16:50:35.932020 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.931967 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m46z\" (UniqueName: \"kubernetes.io/projected/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-kube-api-access-4m46z\") pod \"maas-api-revoke-keys-e2e-mt-b8af660d-5dswt\" (UID: \"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt" Oct 30 16:50:35.932187 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.932125 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8af660d-5dswt\" (UID: \"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt" Oct 30 16:50:35.932252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.932233 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8af660d-5dswt\" (UID: \"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt" Oct 30 16:50:35.932812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.932751 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8af660d-5dswt\" (UID: \"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt" Oct 30 16:50:35.934706 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.934686 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b8af660d-5dswt\" (UID: \"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt" Oct 30 16:50:35.970839 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:35.970553 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m46z\" (UniqueName: \"kubernetes.io/projected/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-kube-api-access-4m46z\") pod \"maas-api-revoke-keys-e2e-mt-b8af660d-5dswt\" (UID: \"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt" Oct 30 16:50:36.003380 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:50:36.003348 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod927cb01d_5133_4ecd_abb7_fa13f1fbe8c5.slice/crio-46d7b357e7c26bcab7253dbc33f1735a851f89c70abe52b5d6c66b0eca0c9d54 WatchSource:0}: Error finding container 46d7b357e7c26bcab7253dbc33f1735a851f89c70abe52b5d6c66b0eca0c9d54: Status 404 returned error can't find the container with id 46d7b357e7c26bcab7253dbc33f1735a851f89c70abe52b5d6c66b0eca0c9d54 Oct 30 16:50:36.004497 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:36.004474 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh"] Oct 30 16:50:36.006365 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:36.006333 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:50:36.006411 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:36.006398 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:50:36.006451 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:36.006429 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:50:36.016313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:36.016286 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 16:50:36.016603 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:36.016341 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" podUID="53877150-e074-4622-afb6-eb0699548ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 16:50:36.031537 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:36.031513 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt" Oct 30 16:50:36.197887 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:36.197637 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m"] Oct 30 16:50:36.198545 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:36.198512 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" podUID="05d8ddc3-7f1c-4c56-a906-3722b011ab90" containerName="istio-proxy" containerID="cri-o://a9a7e14e8000b40aab2c3757c983b7dc0e74029fe2d759aca7b9f39db3448792" gracePeriod=30 Oct 30 16:50:36.488897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:36.488010 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:36.605722 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:36.605601 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt"] Oct 30 16:50:36.608503 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:50:36.608468 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3bcd3df_6929_4c0b_a70b_13abcc4c5d8c.slice/crio-e754a1e72b61b103e309a9484794dff18cd88d06463fea4da11e7359f60f09f9 WatchSource:0}: Error finding container e754a1e72b61b103e309a9484794dff18cd88d06463fea4da11e7359f60f09f9: Status 404 returned error can't find the container with id e754a1e72b61b103e309a9484794dff18cd88d06463fea4da11e7359f60f09f9 Oct 30 16:50:36.819611 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:36.819526 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt" event={"ID":"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c","Type":"ContainerStarted","Data":"e754a1e72b61b103e309a9484794dff18cd88d06463fea4da11e7359f60f09f9"} Oct 30 16:50:36.821301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:36.821271 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" event={"ID":"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5","Type":"ContainerStarted","Data":"c8c0a858d4907d0a55f065d3af3b8eeef108dd178fe3f6e47f43304fbfbdef15"} Oct 30 16:50:36.821433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:36.821331 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" event={"ID":"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5","Type":"ContainerStarted","Data":"46d7b357e7c26bcab7253dbc33f1735a851f89c70abe52b5d6c66b0eca0c9d54"} Oct 30 16:50:36.871792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:36.871719 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" podStartSLOduration=2.8717058189999998 podStartE2EDuration="2.871705819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:36.870070079 +0000 UTC m=+1750.580524820" watchObservedRunningTime="2026-10-30 16:50:36.871705819 +0000 UTC m=+1750.582160585" Oct 30 16:50:37.024822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:37.022752 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 16:50:37.024822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:37.022838 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" podUID="53877150-e074-4622-afb6-eb0699548ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 16:50:37.617702 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:37.617670 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:37.624281 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:37.624257 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:37.831203 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:37.831164 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt" event={"ID":"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c","Type":"ContainerStarted","Data":"00f11cabc6a2810ee94901abd0e45963fb186346725e2c9ccd62984f96bcbde5"} Oct 30 16:50:37.831713 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:37.831689 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:37.832710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:37.832689 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:50:37.935751 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:37.935647 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt" podStartSLOduration=2.9356340640000003 podStartE2EDuration="2.935634064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:37.886930831 +0000 UTC m=+1751.597385574" watchObservedRunningTime="2026-10-30 16:50:37.935634064 +0000 UTC m=+1751.646088806" Oct 30 16:50:38.016349 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:38.016312 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 16:50:38.016501 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:38.016368 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" podUID="53877150-e074-4622-afb6-eb0699548ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 16:50:38.317980 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:38.317897 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:38.842996 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:38.842956 6202 generic.go:356] "Generic (PLEG): container finished" podID="a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c" containerID="00f11cabc6a2810ee94901abd0e45963fb186346725e2c9ccd62984f96bcbde5" exitCode=0 Oct 30 16:50:38.843221 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:38.843054 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt" event={"ID":"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c","Type":"ContainerDied","Data":"00f11cabc6a2810ee94901abd0e45963fb186346725e2c9ccd62984f96bcbde5"} Oct 30 16:50:39.016598 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:39.016562 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 16:50:39.016784 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:39.016619 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" podUID="53877150-e074-4622-afb6-eb0699548ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 16:50:39.250189 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:39.250097 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:39.435598 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:39.435554 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:39.617510 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:39.617477 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:40.020579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.020554 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:40.020998 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.020972 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:40.021800 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.021778 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:50:40.025865 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.025848 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt" Oct 30 16:50:40.057453 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.057424 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-maas-api-xks-ca\") pod \"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c\" (UID: \"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c\") " Oct 30 16:50:40.057599 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.057472 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-maas-api-openshift-ca\") pod \"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c\" (UID: \"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c\") " Oct 30 16:50:40.057599 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.057511 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-kube-api-access-4m46z\" (UniqueName: \"kubernetes.io/projected/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-kube-api-access-4m46z\") pod \"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c\" (UID: \"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c\") " Oct 30 16:50:40.058337 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.057923 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-maas-api-openshift-ca" pod "a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c" (UID: "a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:40.058693 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.058673 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:40.059885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.059859 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-maas-api-xks-ca" pod "a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c" (UID: "a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:40.060371 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.060345 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-kube-api-access-4m46z" pod "a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c" (UID: "a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c"). InnerVolumeSpecName "kube-api-access-4m46z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:40.161448 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.161407 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:40.161448 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.161450 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m46z\" (UniqueName: \"kubernetes.io/projected/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c-kube-api-access-4m46z\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:40.250338 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.250258 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:40.865844 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.865804 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt" Oct 30 16:50:40.865844 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.865818 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt" event={"ID":"a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c","Type":"ContainerDied","Data":"e754a1e72b61b103e309a9484794dff18cd88d06463fea4da11e7359f60f09f9"} Oct 30 16:50:40.866372 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.865895 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e754a1e72b61b103e309a9484794dff18cd88d06463fea4da11e7359f60f09f9" Oct 30 16:50:40.921965 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.921919 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:50:40.922133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:40.921982 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" podUID="05d8ddc3-7f1c-4c56-a906-3722b011ab90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:50:41.039334 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.039297 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:41.151755 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.151681 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf"] Oct 30 16:50:41.152553 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.152537 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c" containerName="revoke-keys" Oct 30 16:50:41.152595 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.152555 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c" containerName="revoke-keys" Oct 30 16:50:41.152714 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.152703 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c" containerName="revoke-keys" Oct 30 16:50:41.157020 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.157002 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" Oct 30 16:50:41.166462 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.166437 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:41.172256 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.172228 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-668047\"" Oct 30 16:50:41.175077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.174669 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf"] Oct 30 16:50:41.178227 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.178195 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxmb9\" (UniqueName: \"kubernetes.io/projected/e3847b39-1962-4f73-abeb-b57ea2a2f8b1-kube-api-access-kxmb9\") pod \"payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf\" (UID: \"e3847b39-1962-4f73-abeb-b57ea2a2f8b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" Oct 30 16:50:41.178398 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.178379 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3847b39-1962-4f73-abeb-b57ea2a2f8b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf\" (UID: \"e3847b39-1962-4f73-abeb-b57ea2a2f8b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" Oct 30 16:50:41.180902 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.180876 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-668047-dockercfg-n7zl8\"" Oct 30 16:50:41.417853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.399729 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxmb9\" (UniqueName: \"kubernetes.io/projected/e3847b39-1962-4f73-abeb-b57ea2a2f8b1-kube-api-access-kxmb9\") pod \"payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf\" (UID: \"e3847b39-1962-4f73-abeb-b57ea2a2f8b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" Oct 30 16:50:41.417853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.400014 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3847b39-1962-4f73-abeb-b57ea2a2f8b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf\" (UID: \"e3847b39-1962-4f73-abeb-b57ea2a2f8b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" Oct 30 16:50:41.417853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.401137 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3847b39-1962-4f73-abeb-b57ea2a2f8b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf\" (UID: \"e3847b39-1962-4f73-abeb-b57ea2a2f8b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" Oct 30 16:50:41.490342 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.488869 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxmb9\" (UniqueName: \"kubernetes.io/projected/e3847b39-1962-4f73-abeb-b57ea2a2f8b1-kube-api-access-kxmb9\") pod \"payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf\" (UID: \"e3847b39-1962-4f73-abeb-b57ea2a2f8b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" Oct 30 16:50:41.496605 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.496572 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j"] Oct 30 16:50:41.508459 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.507314 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j"] Oct 30 16:50:41.508459 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.507431 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" Oct 30 16:50:41.559836 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.558301 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl"] Oct 30 16:50:41.570458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.570421 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt"] Oct 30 16:50:41.611263 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.609526 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00bc805-8ee0-4851-aee7-cfc2856ee609-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j\" (UID: \"d00bc805-8ee0-4851-aee7-cfc2856ee609\") " pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" Oct 30 16:50:41.611263 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.609833 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dgj9\" (UniqueName: \"kubernetes.io/projected/d00bc805-8ee0-4851-aee7-cfc2856ee609-kube-api-access-9dgj9\") pod \"payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j\" (UID: \"d00bc805-8ee0-4851-aee7-cfc2856ee609\") " pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" Oct 30 16:50:41.675885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.675758 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:41.712340 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.712305 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00bc805-8ee0-4851-aee7-cfc2856ee609-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j\" (UID: \"d00bc805-8ee0-4851-aee7-cfc2856ee609\") " pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" Oct 30 16:50:41.712541 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.712522 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dgj9\" (UniqueName: \"kubernetes.io/projected/d00bc805-8ee0-4851-aee7-cfc2856ee609-kube-api-access-9dgj9\") pod \"payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j\" (UID: \"d00bc805-8ee0-4851-aee7-cfc2856ee609\") " pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" Oct 30 16:50:41.713223 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.713202 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00bc805-8ee0-4851-aee7-cfc2856ee609-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j\" (UID: \"d00bc805-8ee0-4851-aee7-cfc2856ee609\") " pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" Oct 30 16:50:41.736361 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.736332 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dgj9\" (UniqueName: \"kubernetes.io/projected/d00bc805-8ee0-4851-aee7-cfc2856ee609-kube-api-access-9dgj9\") pod \"payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j\" (UID: \"d00bc805-8ee0-4851-aee7-cfc2856ee609\") " pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" Oct 30 16:50:41.784046 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.782363 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" Oct 30 16:50:41.841647 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.838154 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" Oct 30 16:50:41.889795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.888692 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:50:41.895796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.894240 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:41.916794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.915856 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-workload-certs\") pod \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " Oct 30 16:50:41.916794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.915950 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-credential-socket\") pod \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " Oct 30 16:50:41.916794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.916071 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-data\") pod \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " Oct 30 16:50:41.916794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.916106 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-podinfo\") pod \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " Oct 30 16:50:41.916794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.916158 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d8ddc3-7f1c-4c56-a906-3722b011ab90-kube-api-access-vr95h\" (UniqueName: \"kubernetes.io/projected/05d8ddc3-7f1c-4c56-a906-3722b011ab90-kube-api-access-vr95h\") pod \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " Oct 30 16:50:41.916794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.916205 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-workload-socket\") pod \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " Oct 30 16:50:41.916794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.916324 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-token\" (UniqueName: \"kubernetes.io/projected/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-token\") pod \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " Oct 30 16:50:41.916794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.916369 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istiod-ca-cert\") pod \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " Oct 30 16:50:41.916794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.916504 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-envoy\") pod \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\" (UID: \"05d8ddc3-7f1c-4c56-a906-3722b011ab90\") " Oct 30 16:50:41.919751 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.919223 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-workload-certs" pod "05d8ddc3-7f1c-4c56-a906-3722b011ab90" (UID: "05d8ddc3-7f1c-4c56-a906-3722b011ab90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:41.919751 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.919477 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-workload-socket" pod "05d8ddc3-7f1c-4c56-a906-3722b011ab90" (UID: "05d8ddc3-7f1c-4c56-a906-3722b011ab90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:41.927151 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.927082 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-credential-socket" pod "05d8ddc3-7f1c-4c56-a906-3722b011ab90" (UID: "05d8ddc3-7f1c-4c56-a906-3722b011ab90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:41.930117 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.930087 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-data" pod "05d8ddc3-7f1c-4c56-a906-3722b011ab90" (UID: "05d8ddc3-7f1c-4c56-a906-3722b011ab90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:41.930593 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.930571 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:41.930699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.930598 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:41.930699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.930612 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:41.930699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.930627 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:41.931120 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.931090 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istiod-ca-cert" pod "05d8ddc3-7f1c-4c56-a906-3722b011ab90" (UID: "05d8ddc3-7f1c-4c56-a906-3722b011ab90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:41.941953 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.941901 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d8ddc3-7f1c-4c56-a906-3722b011ab90-kube-api-access-vr95h" pod "05d8ddc3-7f1c-4c56-a906-3722b011ab90" (UID: "05d8ddc3-7f1c-4c56-a906-3722b011ab90"). InnerVolumeSpecName "kube-api-access-vr95h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:41.952986 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.943899 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-podinfo" pod "05d8ddc3-7f1c-4c56-a906-3722b011ab90" (UID: "05d8ddc3-7f1c-4c56-a906-3722b011ab90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:41.957079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.954006 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-token" pod "05d8ddc3-7f1c-4c56-a906-3722b011ab90" (UID: "05d8ddc3-7f1c-4c56-a906-3722b011ab90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:41.957713 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:41.957676 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-envoy" pod "05d8ddc3-7f1c-4c56-a906-3722b011ab90" (UID: "05d8ddc3-7f1c-4c56-a906-3722b011ab90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:42.002788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:42.000585 6202 generic.go:356] "Generic (PLEG): container finished" podID="05d8ddc3-7f1c-4c56-a906-3722b011ab90" containerID="a9a7e14e8000b40aab2c3757c983b7dc0e74029fe2d759aca7b9f39db3448792" exitCode=0 Oct 30 16:50:42.002788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:42.002031 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" Oct 30 16:50:42.009519 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:42.007872 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" event={"ID":"05d8ddc3-7f1c-4c56-a906-3722b011ab90","Type":"ContainerDied","Data":"a9a7e14e8000b40aab2c3757c983b7dc0e74029fe2d759aca7b9f39db3448792"} Oct 30 16:50:42.009519 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:42.007950 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m" event={"ID":"05d8ddc3-7f1c-4c56-a906-3722b011ab90","Type":"ContainerDied","Data":"8fdceb121748b5937659e3209b3cac789399b05b422a1535a5ff17e28b40062c"} Oct 30 16:50:42.009519 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:42.007995 6202 scope.go:122] "RemoveContainer" containerID="a9a7e14e8000b40aab2c3757c983b7dc0e74029fe2d759aca7b9f39db3448792" Oct 30 16:50:42.034801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:42.033968 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:42.034801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:42.033999 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:42.034801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:42.034013 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:42.034801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:42.034023 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d8ddc3-7f1c-4c56-a906-3722b011ab90-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:42.034801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:42.034037 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr95h\" (UniqueName: \"kubernetes.io/projected/05d8ddc3-7f1c-4c56-a906-3722b011ab90-kube-api-access-vr95h\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:42.072924 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:42.072890 6202 scope.go:122] "RemoveContainer" containerID="a9a7e14e8000b40aab2c3757c983b7dc0e74029fe2d759aca7b9f39db3448792" Oct 30 16:50:42.074404 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:50:42.074361 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9a7e14e8000b40aab2c3757c983b7dc0e74029fe2d759aca7b9f39db3448792\": container with ID starting with a9a7e14e8000b40aab2c3757c983b7dc0e74029fe2d759aca7b9f39db3448792 not found: ID does not exist" containerID="a9a7e14e8000b40aab2c3757c983b7dc0e74029fe2d759aca7b9f39db3448792" Oct 30 16:50:42.074513 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:42.074407 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9a7e14e8000b40aab2c3757c983b7dc0e74029fe2d759aca7b9f39db3448792"} err="failed to get container status \"a9a7e14e8000b40aab2c3757c983b7dc0e74029fe2d759aca7b9f39db3448792\": rpc error: code = NotFound desc = could not find container \"a9a7e14e8000b40aab2c3757c983b7dc0e74029fe2d759aca7b9f39db3448792\": container with ID starting with a9a7e14e8000b40aab2c3757c983b7dc0e74029fe2d759aca7b9f39db3448792 not found: ID does not exist" Oct 30 16:50:42.085053 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:42.085026 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m"] Oct 30 16:50:42.090588 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:42.090028 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-h8j9m"] Oct 30 16:50:42.509788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:42.506460 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:42.559006 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:50:42.553178 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3847b39_1962_4f73_abeb_b57ea2a2f8b1.slice/crio-89967007387b48dd12c6876bd5f357bd12358179c9cf45f42282a8d5474ae3ec WatchSource:0}: Error finding container 89967007387b48dd12c6876bd5f357bd12358179c9cf45f42282a8d5474ae3ec: Status 404 returned error can't find the container with id 89967007387b48dd12c6876bd5f357bd12358179c9cf45f42282a8d5474ae3ec Oct 30 16:50:42.578635 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:42.578600 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05d8ddc3-7f1c-4c56-a906-3722b011ab90" path="/var/lib/kubelet/pods/05d8ddc3-7f1c-4c56-a906-3722b011ab90/volumes" Oct 30 16:50:42.579100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:42.579080 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j"] Oct 30 16:50:42.579214 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:42.579166 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf"] Oct 30 16:50:43.010045 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:43.010011 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" event={"ID":"d00bc805-8ee0-4851-aee7-cfc2856ee609","Type":"ContainerStarted","Data":"1896cec0ede03bac79ec5261c3b6cf3f2720755c0e36208594ac2418276c8fc1"} Oct 30 16:50:43.010522 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:43.010064 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" event={"ID":"d00bc805-8ee0-4851-aee7-cfc2856ee609","Type":"ContainerStarted","Data":"c42559e15d053d10a73d9f1b8eab6d7616d828e9c1375b6282be41050354dab1"} Oct 30 16:50:43.010522 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:43.010114 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" Oct 30 16:50:43.012953 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:43.012926 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" event={"ID":"e3847b39-1962-4f73-abeb-b57ea2a2f8b1","Type":"ContainerStarted","Data":"de0e361b09c4e4a7840bf4547dac2fc0632d3b75be73f7064010557681dfe7bd"} Oct 30 16:50:43.013096 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:43.012971 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" event={"ID":"e3847b39-1962-4f73-abeb-b57ea2a2f8b1","Type":"ContainerStarted","Data":"89967007387b48dd12c6876bd5f357bd12358179c9cf45f42282a8d5474ae3ec"} Oct 30 16:50:43.013096 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:43.012993 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" Oct 30 16:50:43.054706 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:43.054659 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" podStartSLOduration=2.054641628 podStartE2EDuration="2.054641628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:43.051317394 +0000 UTC m=+1756.761772136" watchObservedRunningTime="2026-10-30 16:50:43.054641628 +0000 UTC m=+1756.765096372" Oct 30 16:50:43.092474 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:43.092419 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" podStartSLOduration=2.092405928 podStartE2EDuration="2.092405928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:43.090754551 +0000 UTC m=+1756.801209294" watchObservedRunningTime="2026-10-30 16:50:43.092405928 +0000 UTC m=+1756.802860683" Oct 30 16:50:44.039925 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:44.039889 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7"] Oct 30 16:50:44.051334 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:44.051305 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11f8661-wh8v7"] Oct 30 16:50:44.074203 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:50:44.074154 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:50:44.251236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:44.251206 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:44.413358 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:44.413325 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48fe738b-864a-4ff6-b9f8-57d90c84788d" path="/var/lib/kubelet/pods/48fe738b-864a-4ff6-b9f8-57d90c84788d/volumes" Oct 30 16:50:45.878488 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:45.878450 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:46.269190 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:46.266815 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6"] Oct 30 16:50:46.373837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:46.373800 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:46.561849 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:46.559554 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl" podUID="4dc27f86-485d-4aed-9aa0-5b5f064bb50d" containerName="payload-processing" containerID="cri-o://b93ae686814da77860e01b1f8d8ae574045b40844140a471fa732f95a1f84dbd" gracePeriod=25 Oct 30 16:50:46.572579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:46.572198 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt" podUID="20ddef6a-15ca-42e0-960f-bff2270c35e6" containerName="payload-pre-processing" containerID="cri-o://3ba0b4156aa7913ef9d958d1abbd4e7e31a42fe2b7f385662bc4f210a8216037" gracePeriod=25 Oct 30 16:50:46.638837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:46.638631 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:46.778096 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:46.778058 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq"] Oct 30 16:50:46.779328 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:46.779305 6202 cpu_manager.go:414] "containerMap: removing container" podUID="05d8ddc3-7f1c-4c56-a906-3722b011ab90" containerName="istio-proxy" Oct 30 16:50:46.779392 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:46.779333 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d8ddc3-7f1c-4c56-a906-3722b011ab90" containerName="istio-proxy" Oct 30 16:50:46.779610 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:46.779593 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d8ddc3-7f1c-4c56-a906-3722b011ab90" containerName="istio-proxy" Oct 30 16:50:46.784376 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:46.784358 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:46.821719 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:46.821695 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq"] Oct 30 16:50:47.144020 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.135650 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.144020 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.135703 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.181590 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.166107 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9gdf\" (UniqueName: \"kubernetes.io/projected/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-kube-api-access-q9gdf\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.181590 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.166220 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.181590 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.166379 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.181590 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.166414 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.181590 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.166478 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.181590 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.166515 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.181590 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.166575 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.242075 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.242046 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt" Oct 30 16:50:47.251954 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.251911 6202 generic.go:356] "Generic (PLEG): container finished" podID="20ddef6a-15ca-42e0-960f-bff2270c35e6" containerID="3ba0b4156aa7913ef9d958d1abbd4e7e31a42fe2b7f385662bc4f210a8216037" exitCode=0 Oct 30 16:50:47.252434 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.251999 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt" event={"ID":"20ddef6a-15ca-42e0-960f-bff2270c35e6","Type":"ContainerDied","Data":"3ba0b4156aa7913ef9d958d1abbd4e7e31a42fe2b7f385662bc4f210a8216037"} Oct 30 16:50:47.253829 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.253750 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl" Oct 30 16:50:47.254533 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.254511 6202 generic.go:356] "Generic (PLEG): container finished" podID="4dc27f86-485d-4aed-9aa0-5b5f064bb50d" containerID="b93ae686814da77860e01b1f8d8ae574045b40844140a471fa732f95a1f84dbd" exitCode=0 Oct 30 16:50:47.255047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.255007 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt" event={"ID":"20ddef6a-15ca-42e0-960f-bff2270c35e6","Type":"ContainerDied","Data":"2298cc2d5c42c447470fbddd005f1ff95b743640f03f8782dfd00c9a663ca0fe"} Oct 30 16:50:47.255142 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.255060 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl" event={"ID":"4dc27f86-485d-4aed-9aa0-5b5f064bb50d","Type":"ContainerDied","Data":"b93ae686814da77860e01b1f8d8ae574045b40844140a471fa732f95a1f84dbd"} Oct 30 16:50:47.255142 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.255097 6202 scope.go:122] "RemoveContainer" containerID="3ba0b4156aa7913ef9d958d1abbd4e7e31a42fe2b7f385662bc4f210a8216037" Oct 30 16:50:47.268149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.268103 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.268292 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.268272 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.268368 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.268314 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.268431 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.268366 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.268431 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.268396 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.268561 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.268444 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.268623 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.268578 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.268623 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.268606 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.268760 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.268739 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9gdf\" (UniqueName: \"kubernetes.io/projected/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-kube-api-access-q9gdf\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.269471 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.269445 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.269706 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.269685 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.269950 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.269930 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.270731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.270697 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.271712 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.271687 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.272996 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.272980 6202 scope.go:122] "RemoveContainer" containerID="3ba0b4156aa7913ef9d958d1abbd4e7e31a42fe2b7f385662bc4f210a8216037" Oct 30 16:50:47.273408 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:50:47.273306 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba0b4156aa7913ef9d958d1abbd4e7e31a42fe2b7f385662bc4f210a8216037\": container with ID starting with 3ba0b4156aa7913ef9d958d1abbd4e7e31a42fe2b7f385662bc4f210a8216037 not found: ID does not exist" containerID="3ba0b4156aa7913ef9d958d1abbd4e7e31a42fe2b7f385662bc4f210a8216037" Oct 30 16:50:47.273408 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.273332 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba0b4156aa7913ef9d958d1abbd4e7e31a42fe2b7f385662bc4f210a8216037"} err="failed to get container status \"3ba0b4156aa7913ef9d958d1abbd4e7e31a42fe2b7f385662bc4f210a8216037\": rpc error: code = NotFound desc = could not find container \"3ba0b4156aa7913ef9d958d1abbd4e7e31a42fe2b7f385662bc4f210a8216037\": container with ID starting with 3ba0b4156aa7913ef9d958d1abbd4e7e31a42fe2b7f385662bc4f210a8216037 not found: ID does not exist" Oct 30 16:50:47.273408 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.273369 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.275244 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.275223 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.298312 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.298281 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9gdf\" (UniqueName: \"kubernetes.io/projected/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-kube-api-access-q9gdf\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.302108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.302078 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.370794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.370336 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ddef6a-15ca-42e0-960f-bff2270c35e6-kube-api-access-2qtw6\" (UniqueName: \"kubernetes.io/projected/20ddef6a-15ca-42e0-960f-bff2270c35e6-kube-api-access-2qtw6\") pod \"20ddef6a-15ca-42e0-960f-bff2270c35e6\" (UID: \"20ddef6a-15ca-42e0-960f-bff2270c35e6\") " Oct 30 16:50:47.370794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.370400 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dc27f86-485d-4aed-9aa0-5b5f064bb50d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc27f86-485d-4aed-9aa0-5b5f064bb50d-plugins-config-volume\") pod \"4dc27f86-485d-4aed-9aa0-5b5f064bb50d\" (UID: \"4dc27f86-485d-4aed-9aa0-5b5f064bb50d\") " Oct 30 16:50:47.370794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.370534 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dc27f86-485d-4aed-9aa0-5b5f064bb50d-kube-api-access-h4ffm\" (UniqueName: \"kubernetes.io/projected/4dc27f86-485d-4aed-9aa0-5b5f064bb50d-kube-api-access-h4ffm\") pod \"4dc27f86-485d-4aed-9aa0-5b5f064bb50d\" (UID: \"4dc27f86-485d-4aed-9aa0-5b5f064bb50d\") " Oct 30 16:50:47.370794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.370595 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20ddef6a-15ca-42e0-960f-bff2270c35e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ddef6a-15ca-42e0-960f-bff2270c35e6-plugins-config-volume\") pod \"20ddef6a-15ca-42e0-960f-bff2270c35e6\" (UID: \"20ddef6a-15ca-42e0-960f-bff2270c35e6\") " Oct 30 16:50:47.371123 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.370816 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc27f86-485d-4aed-9aa0-5b5f064bb50d-plugins-config-volume" pod "4dc27f86-485d-4aed-9aa0-5b5f064bb50d" (UID: "4dc27f86-485d-4aed-9aa0-5b5f064bb50d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:47.371685 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.371186 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ddef6a-15ca-42e0-960f-bff2270c35e6-plugins-config-volume" pod "20ddef6a-15ca-42e0-960f-bff2270c35e6" (UID: "20ddef6a-15ca-42e0-960f-bff2270c35e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:47.371685 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.371492 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc27f86-485d-4aed-9aa0-5b5f064bb50d-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:47.371685 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.371516 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ddef6a-15ca-42e0-960f-bff2270c35e6-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:47.375281 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.375121 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc27f86-485d-4aed-9aa0-5b5f064bb50d-kube-api-access-h4ffm" pod "4dc27f86-485d-4aed-9aa0-5b5f064bb50d" (UID: "4dc27f86-485d-4aed-9aa0-5b5f064bb50d"). InnerVolumeSpecName "kube-api-access-h4ffm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:47.377819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.376911 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ddef6a-15ca-42e0-960f-bff2270c35e6-kube-api-access-2qtw6" pod "20ddef6a-15ca-42e0-960f-bff2270c35e6" (UID: "20ddef6a-15ca-42e0-960f-bff2270c35e6"). InnerVolumeSpecName "kube-api-access-2qtw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:47.412079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.411996 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:47.475810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.474061 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4ffm\" (UniqueName: \"kubernetes.io/projected/4dc27f86-485d-4aed-9aa0-5b5f064bb50d-kube-api-access-h4ffm\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:47.475810 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.474108 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qtw6\" (UniqueName: \"kubernetes.io/projected/20ddef6a-15ca-42e0-960f-bff2270c35e6-kube-api-access-2qtw6\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:47.856790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.856580 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq"] Oct 30 16:50:47.939192 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.939150 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:50:47.939405 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.939384 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:50:47.939534 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:47.939519 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:50:48.137871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:48.137837 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:48.266177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:48.266087 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl" Oct 30 16:50:48.266177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:48.266116 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl" event={"ID":"4dc27f86-485d-4aed-9aa0-5b5f064bb50d","Type":"ContainerDied","Data":"68470a23e8fd08611fe1c6e1f49a00985d7fa8a2068f6769db98471f7e5665ca"} Oct 30 16:50:48.266734 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:48.266185 6202 scope.go:122] "RemoveContainer" containerID="b93ae686814da77860e01b1f8d8ae574045b40844140a471fa732f95a1f84dbd" Oct 30 16:50:48.270146 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:48.270113 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" event={"ID":"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1","Type":"ContainerStarted","Data":"d052b3803f05d78f678ae6f22e029fefe881aa8ef5643bc8f4b7d2ec7a9f4bf1"} Oct 30 16:50:48.270285 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:48.270172 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" event={"ID":"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1","Type":"ContainerStarted","Data":"756eecef546b34eb7260bfe44bfad862daab6e474e1b8d0ab2e1a42257e44859"} Oct 30 16:50:48.271690 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:48.271668 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt" Oct 30 16:50:48.354807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:48.354738 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" podStartSLOduration=2.354724329 podStartE2EDuration="2.354724329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:48.35091099 +0000 UTC m=+1762.061365757" watchObservedRunningTime="2026-10-30 16:50:48.354724329 +0000 UTC m=+1762.065179068" Oct 30 16:50:48.400847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:48.400816 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl"] Oct 30 16:50:48.414047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:48.414022 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ca5971-9b8d75788-jbskl"] Oct 30 16:50:48.414187 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:48.414059 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:48.436487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:48.436453 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt"] Oct 30 16:50:48.442587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:48.442543 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ca5971-8b75698f5-pjjdt"] Oct 30 16:50:48.981490 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:48.981451 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:49.414341 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:49.414307 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 16:50:49.414739 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:49.414363 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" podUID="182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 16:50:50.414699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:50.414660 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 16:50:50.415306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:50.414727 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" podUID="182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 16:50:50.418488 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:50.418457 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:50.427466 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:50.427434 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20ddef6a-15ca-42e0-960f-bff2270c35e6" path="/var/lib/kubelet/pods/20ddef6a-15ca-42e0-960f-bff2270c35e6/volumes" Oct 30 16:50:50.428373 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:50.428335 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dc27f86-485d-4aed-9aa0-5b5f064bb50d" path="/var/lib/kubelet/pods/4dc27f86-485d-4aed-9aa0-5b5f064bb50d/volumes" Oct 30 16:50:51.268794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:51.267905 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6" podUID="872ede25-b9c3-43a9-a242-88381d4c8e74" containerName="payload-pre-processing" containerID="cri-o://e8a87859cd9d22de6a25137cc3ceaf8e40d6e01e97291f29dc3b3771e94284de" gracePeriod=25 Oct 30 16:50:51.412593 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:51.412561 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 16:50:51.412725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:51.412615 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" podUID="182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 16:50:51.674400 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:51.674370 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6" Oct 30 16:50:51.724233 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:51.724201 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:51.730669 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:51.730633 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/872ede25-b9c3-43a9-a242-88381d4c8e74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872ede25-b9c3-43a9-a242-88381d4c8e74-plugins-config-volume\") pod \"872ede25-b9c3-43a9-a242-88381d4c8e74\" (UID: \"872ede25-b9c3-43a9-a242-88381d4c8e74\") " Oct 30 16:50:51.730896 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:51.730828 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872ede25-b9c3-43a9-a242-88381d4c8e74-kube-api-access-k6f4g\" (UniqueName: \"kubernetes.io/projected/872ede25-b9c3-43a9-a242-88381d4c8e74-kube-api-access-k6f4g\") pod \"872ede25-b9c3-43a9-a242-88381d4c8e74\" (UID: \"872ede25-b9c3-43a9-a242-88381d4c8e74\") " Oct 30 16:50:51.731060 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:51.731037 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872ede25-b9c3-43a9-a242-88381d4c8e74-plugins-config-volume" pod "872ede25-b9c3-43a9-a242-88381d4c8e74" (UID: "872ede25-b9c3-43a9-a242-88381d4c8e74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:51.731593 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:51.731567 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872ede25-b9c3-43a9-a242-88381d4c8e74-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:51.733620 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:51.733594 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872ede25-b9c3-43a9-a242-88381d4c8e74-kube-api-access-k6f4g" pod "872ede25-b9c3-43a9-a242-88381d4c8e74" (UID: "872ede25-b9c3-43a9-a242-88381d4c8e74"). InnerVolumeSpecName "kube-api-access-k6f4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:51.833298 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:51.833268 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6f4g\" (UniqueName: \"kubernetes.io/projected/872ede25-b9c3-43a9-a242-88381d4c8e74-kube-api-access-k6f4g\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:50:52.309324 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:52.309234 6202 generic.go:356] "Generic (PLEG): container finished" podID="872ede25-b9c3-43a9-a242-88381d4c8e74" containerID="e8a87859cd9d22de6a25137cc3ceaf8e40d6e01e97291f29dc3b3771e94284de" exitCode=0 Oct 30 16:50:52.309324 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:52.309285 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6" event={"ID":"872ede25-b9c3-43a9-a242-88381d4c8e74","Type":"ContainerDied","Data":"e8a87859cd9d22de6a25137cc3ceaf8e40d6e01e97291f29dc3b3771e94284de"} Oct 30 16:50:52.309324 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:52.309292 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6" Oct 30 16:50:52.309324 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:52.309325 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6" event={"ID":"872ede25-b9c3-43a9-a242-88381d4c8e74","Type":"ContainerDied","Data":"4529bf22832fdc77aca74ad53016107860b301dfc7385a618a40de71488ac7b0"} Oct 30 16:50:52.309844 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:52.309348 6202 scope.go:122] "RemoveContainer" containerID="e8a87859cd9d22de6a25137cc3ceaf8e40d6e01e97291f29dc3b3771e94284de" Oct 30 16:50:52.320896 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:52.320861 6202 scope.go:122] "RemoveContainer" containerID="e8a87859cd9d22de6a25137cc3ceaf8e40d6e01e97291f29dc3b3771e94284de" Oct 30 16:50:52.321230 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:50:52.321210 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a87859cd9d22de6a25137cc3ceaf8e40d6e01e97291f29dc3b3771e94284de\": container with ID starting with e8a87859cd9d22de6a25137cc3ceaf8e40d6e01e97291f29dc3b3771e94284de not found: ID does not exist" containerID="e8a87859cd9d22de6a25137cc3ceaf8e40d6e01e97291f29dc3b3771e94284de" Oct 30 16:50:52.321290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:52.321241 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a87859cd9d22de6a25137cc3ceaf8e40d6e01e97291f29dc3b3771e94284de"} err="failed to get container status \"e8a87859cd9d22de6a25137cc3ceaf8e40d6e01e97291f29dc3b3771e94284de\": rpc error: code = NotFound desc = could not find container \"e8a87859cd9d22de6a25137cc3ceaf8e40d6e01e97291f29dc3b3771e94284de\": container with ID starting with e8a87859cd9d22de6a25137cc3ceaf8e40d6e01e97291f29dc3b3771e94284de not found: ID does not exist" Oct 30 16:50:52.377935 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:52.377899 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6"] Oct 30 16:50:52.387053 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:52.387020 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8af660d-65d4cd8b55-fvff6"] Oct 30 16:50:52.412574 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:52.412543 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 16:50:52.412736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:52.412604 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" podUID="182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 16:50:52.413432 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:52.413410 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="872ede25-b9c3-43a9-a242-88381d4c8e74" path="/var/lib/kubelet/pods/872ede25-b9c3-43a9-a242-88381d4c8e74/volumes" Oct 30 16:50:52.619731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:52.619702 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:53.412573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:53.412538 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 16:50:53.413007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:53.412602 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" podUID="182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 16:50:54.024254 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:54.024224 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" Oct 30 16:50:54.024438 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:54.024329 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" Oct 30 16:50:54.351652 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:54.351615 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:54.413060 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:54.413024 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 16:50:54.413465 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:54.413075 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" podUID="182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 16:50:54.991447 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:54.991404 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:55.239600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:55.239537 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:55.324501 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:55.324470 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt"] Oct 30 16:50:55.336013 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:55.335952 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b8af660d-5dswt"] Oct 30 16:50:55.486865 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:55.486820 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:55.487557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:55.487520 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:55.488327 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:55.488293 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:50:55.615149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:55.615061 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:55.882059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:55.881961 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:56.415354 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:56.415318 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c" path="/var/lib/kubelet/pods/a3bcd3df-6929-4c0b-a70b-13abcc4c5d8c/volumes" Oct 30 16:50:57.484623 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:57.484574 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:59.172883 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:50:59.172846 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:00.781514 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:00.781481 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:00.894148 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:00.894113 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj"] Oct 30 16:51:00.981597 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:00.981555 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:01.801388 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.801350 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw"] Oct 30 16:51:01.802237 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.802219 6202 cpu_manager.go:414] "containerMap: removing container" podUID="20ddef6a-15ca-42e0-960f-bff2270c35e6" containerName="payload-pre-processing" Oct 30 16:51:01.802314 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.802239 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ddef6a-15ca-42e0-960f-bff2270c35e6" containerName="payload-pre-processing" Oct 30 16:51:01.802314 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.802270 6202 cpu_manager.go:414] "containerMap: removing container" podUID="872ede25-b9c3-43a9-a242-88381d4c8e74" containerName="payload-pre-processing" Oct 30 16:51:01.802314 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.802276 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872ede25-b9c3-43a9-a242-88381d4c8e74" containerName="payload-pre-processing" Oct 30 16:51:01.802314 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.802300 6202 cpu_manager.go:414] "containerMap: removing container" podUID="4dc27f86-485d-4aed-9aa0-5b5f064bb50d" containerName="payload-processing" Oct 30 16:51:01.802314 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.802304 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dc27f86-485d-4aed-9aa0-5b5f064bb50d" containerName="payload-processing" Oct 30 16:51:01.802476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.802436 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dc27f86-485d-4aed-9aa0-5b5f064bb50d" containerName="payload-processing" Oct 30 16:51:01.802476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.802450 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ddef6a-15ca-42e0-960f-bff2270c35e6" containerName="payload-pre-processing" Oct 30 16:51:01.802476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.802460 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="872ede25-b9c3-43a9-a242-88381d4c8e74" containerName="payload-pre-processing" Oct 30 16:51:01.807595 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.807575 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:01.842475 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.842437 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw"] Oct 30 16:51:01.960495 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.960459 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:01.960672 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.960509 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:01.960672 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.960541 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:01.960672 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.960621 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:01.960809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.960702 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc4lx\" (UniqueName: \"kubernetes.io/projected/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-kube-api-access-tc4lx\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:01.960809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.960802 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:01.960902 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.960886 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:01.960946 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.960930 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:01.960994 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:01.960984 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.062380 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.062285 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.062380 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.062367 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.062612 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.062396 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.062612 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.062467 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.062746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.062712 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.062836 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.062817 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.062893 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.062870 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.062948 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.062899 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.063356 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.063328 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc4lx\" (UniqueName: \"kubernetes.io/projected/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-kube-api-access-tc4lx\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.063936 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.063901 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.063936 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.063396 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.064129 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.063590 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.064129 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.063110 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.064129 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.063616 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.065915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.065896 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.066304 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.066288 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.086874 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.086846 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc4lx\" (UniqueName: \"kubernetes.io/projected/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-kube-api-access-tc4lx\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.087019 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.086905 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.124534 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.124506 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:02.452669 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:51:02.452636 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c4b2e9_0686_4ccd_9a97_cc6c8cbfb668.slice/crio-8e5984f943b381393ba32f4fa0c8aa1f5a8f6dce0548554b0757492f2ee8d3ce WatchSource:0}: Error finding container 8e5984f943b381393ba32f4fa0c8aa1f5a8f6dce0548554b0757492f2ee8d3ce: Status 404 returned error can't find the container with id 8e5984f943b381393ba32f4fa0c8aa1f5a8f6dce0548554b0757492f2ee8d3ce Oct 30 16:51:02.455371 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.455340 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw"] Oct 30 16:51:02.455978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.455932 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:51:02.456095 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.456045 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:51:02.456095 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.456094 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:51:02.579072 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:02.578954 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:03.422932 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:03.422898 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" event={"ID":"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668","Type":"ContainerStarted","Data":"e81e41ad3c9d14fb52a942d8e0aa082a104c9627036a03050156227379e40b66"} Oct 30 16:51:03.423347 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:03.422950 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" event={"ID":"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668","Type":"ContainerStarted","Data":"8e5984f943b381393ba32f4fa0c8aa1f5a8f6dce0548554b0757492f2ee8d3ce"} Oct 30 16:51:04.125270 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:04.125237 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:04.127034 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:04.127003 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 16:51:04.127166 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:04.127056 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" podUID="75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 16:51:04.235857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:04.235820 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:05.124922 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:05.124886 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 16:51:05.125337 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:05.124937 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" podUID="75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 16:51:06.030791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:06.029375 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:06.125532 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:06.125489 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 16:51:06.125962 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:06.125560 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" podUID="75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 16:51:07.125375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:07.125341 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 16:51:07.125549 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:07.125398 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" podUID="75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 16:51:07.770364 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:07.770325 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:08.125240 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:08.125209 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 16:51:08.125425 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:08.125258 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" podUID="75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 16:51:09.231372 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:09.231322 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:09.232367 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:09.232341 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:09.234051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:09.234028 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:09.289075 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:09.289022 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" podStartSLOduration=8.28900814 podStartE2EDuration="8.28900814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:03.51567203 +0000 UTC m=+1777.226126773" watchObservedRunningTime="2026-10-30 16:51:09.28900814 +0000 UTC m=+1782.999462907" Oct 30 16:51:09.386585 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:09.386543 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:11.077506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:11.077468 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:13.546365 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:13.546325 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:14.074054 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:51:14.074016 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:51:15.516675 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:15.516641 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:15.921644 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:15.921593 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" podUID="d83ab97a-69c3-4758-9b33-da6bb816a575" containerName="main" containerID="cri-o://8d450b225f03d13c4db98a0328f9f642524d34d53ea231a385ec57a96859d4c3" gracePeriod=45 Oct 30 16:51:16.220532 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.220505 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:51:16.312237 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.312196 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-home\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-home\") pod \"d83ab97a-69c3-4758-9b33-da6bb816a575\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " Oct 30 16:51:16.312434 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.312409 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-tmp-dir\") pod \"d83ab97a-69c3-4758-9b33-da6bb816a575\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " Oct 30 16:51:16.312512 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.312464 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d83ab97a-69c3-4758-9b33-da6bb816a575-tls-certs\" (UniqueName: \"kubernetes.io/secret/d83ab97a-69c3-4758-9b33-da6bb816a575-tls-certs\") pod \"d83ab97a-69c3-4758-9b33-da6bb816a575\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " Oct 30 16:51:16.312597 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.312579 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d83ab97a-69c3-4758-9b33-da6bb816a575-kube-api-access-5fgcn\" (UniqueName: \"kubernetes.io/projected/d83ab97a-69c3-4758-9b33-da6bb816a575-kube-api-access-5fgcn\") pod \"d83ab97a-69c3-4758-9b33-da6bb816a575\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " Oct 30 16:51:16.312664 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.312624 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-dshm\") pod \"d83ab97a-69c3-4758-9b33-da6bb816a575\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " Oct 30 16:51:16.312808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.312746 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-model-cache\") pod \"d83ab97a-69c3-4758-9b33-da6bb816a575\" (UID: \"d83ab97a-69c3-4758-9b33-da6bb816a575\") " Oct 30 16:51:16.313811 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.313781 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-model-cache" pod "d83ab97a-69c3-4758-9b33-da6bb816a575" (UID: "d83ab97a-69c3-4758-9b33-da6bb816a575"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:16.314025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.314005 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-home" pod "d83ab97a-69c3-4758-9b33-da6bb816a575" (UID: "d83ab97a-69c3-4758-9b33-da6bb816a575"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:16.314203 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.314182 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-tmp-dir" pod "d83ab97a-69c3-4758-9b33-da6bb816a575" (UID: "d83ab97a-69c3-4758-9b33-da6bb816a575"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:16.318199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.318165 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-dshm" pod "d83ab97a-69c3-4758-9b33-da6bb816a575" (UID: "d83ab97a-69c3-4758-9b33-da6bb816a575"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:16.318365 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.318342 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83ab97a-69c3-4758-9b33-da6bb816a575-kube-api-access-5fgcn" pod "d83ab97a-69c3-4758-9b33-da6bb816a575" (UID: "d83ab97a-69c3-4758-9b33-da6bb816a575"). InnerVolumeSpecName "kube-api-access-5fgcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:16.318698 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.318674 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d83ab97a-69c3-4758-9b33-da6bb816a575-tls-certs" pod "d83ab97a-69c3-4758-9b33-da6bb816a575" (UID: "d83ab97a-69c3-4758-9b33-da6bb816a575"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:16.414262 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.414230 6202 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-home\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:16.414262 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.414259 6202 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-tmp-dir\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:16.414476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.414270 6202 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83ab97a-69c3-4758-9b33-da6bb816a575-tls-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:16.414476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.414283 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fgcn\" (UniqueName: \"kubernetes.io/projected/d83ab97a-69c3-4758-9b33-da6bb816a575-kube-api-access-5fgcn\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:16.414476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.414294 6202 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-dshm\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:16.414476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.414305 6202 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83ab97a-69c3-4758-9b33-da6bb816a575-model-cache\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:16.706323 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.706292 6202 generic.go:356] "Generic (PLEG): container finished" podID="d83ab97a-69c3-4758-9b33-da6bb816a575" containerID="8d450b225f03d13c4db98a0328f9f642524d34d53ea231a385ec57a96859d4c3" exitCode=0 Oct 30 16:51:16.706727 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.706335 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" Oct 30 16:51:16.706727 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.706361 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" event={"ID":"d83ab97a-69c3-4758-9b33-da6bb816a575","Type":"ContainerDied","Data":"8d450b225f03d13c4db98a0328f9f642524d34d53ea231a385ec57a96859d4c3"} Oct 30 16:51:16.706727 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.706406 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj" event={"ID":"d83ab97a-69c3-4758-9b33-da6bb816a575","Type":"ContainerDied","Data":"e447c8fdd56774a1965f079f886ff5b846785595ade19ec704dc93f567f04db4"} Oct 30 16:51:16.706727 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.706425 6202 scope.go:122] "RemoveContainer" containerID="8d450b225f03d13c4db98a0328f9f642524d34d53ea231a385ec57a96859d4c3" Oct 30 16:51:16.718249 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.718228 6202 scope.go:122] "RemoveContainer" containerID="8d450b225f03d13c4db98a0328f9f642524d34d53ea231a385ec57a96859d4c3" Oct 30 16:51:16.718524 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:51:16.718502 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d450b225f03d13c4db98a0328f9f642524d34d53ea231a385ec57a96859d4c3\": container with ID starting with 8d450b225f03d13c4db98a0328f9f642524d34d53ea231a385ec57a96859d4c3 not found: ID does not exist" containerID="8d450b225f03d13c4db98a0328f9f642524d34d53ea231a385ec57a96859d4c3" Oct 30 16:51:16.718609 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.718531 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d450b225f03d13c4db98a0328f9f642524d34d53ea231a385ec57a96859d4c3"} err="failed to get container status \"8d450b225f03d13c4db98a0328f9f642524d34d53ea231a385ec57a96859d4c3\": rpc error: code = NotFound desc = could not find container \"8d450b225f03d13c4db98a0328f9f642524d34d53ea231a385ec57a96859d4c3\": container with ID starting with 8d450b225f03d13c4db98a0328f9f642524d34d53ea231a385ec57a96859d4c3 not found: ID does not exist" Oct 30 16:51:16.768289 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.768253 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj"] Oct 30 16:51:16.780725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:16.780691 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b8af660d/e2e-lifecycle-model-b8af660d-kserve-7787bf7f77-phhwj"] Oct 30 16:51:17.351773 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:17.351740 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:17.549414 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:17.549376 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:17.790654 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:17.790559 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:18.051109 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.051024 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:18.249002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.248964 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f"] Oct 30 16:51:18.250417 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.250387 6202 cpu_manager.go:414] "containerMap: removing container" podUID="d83ab97a-69c3-4758-9b33-da6bb816a575" containerName="main" Oct 30 16:51:18.250417 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.250417 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d83ab97a-69c3-4758-9b33-da6bb816a575" containerName="main" Oct 30 16:51:18.250673 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.250657 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="d83ab97a-69c3-4758-9b33-da6bb816a575" containerName="main" Oct 30 16:51:18.257002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.256979 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.272999 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.272956 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-480a60-openshift-default-dockercfg-q85jg\"" Oct 30 16:51:18.290552 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.290520 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f"] Oct 30 16:51:18.347473 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.347430 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3554e861-457c-4806-99ee-ad9e27786362-istio-token\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.347679 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.347581 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3554e861-457c-4806-99ee-ad9e27786362-istiod-ca-cert\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.347679 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.347626 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-istio-data\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.347828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.347737 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmt96\" (UniqueName: \"kubernetes.io/projected/3554e861-457c-4806-99ee-ad9e27786362-kube-api-access-nmt96\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.347885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.347827 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-credential-socket\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.347885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.347849 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3554e861-457c-4806-99ee-ad9e27786362-istio-podinfo\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.347885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.347870 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-workload-certs\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.348055 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.348039 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-istio-envoy\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.348113 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.348079 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-workload-socket\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.354352 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.354322 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:18.413025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.412990 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d83ab97a-69c3-4758-9b33-da6bb816a575" path="/var/lib/kubelet/pods/d83ab97a-69c3-4758-9b33-da6bb816a575/volumes" Oct 30 16:51:18.449817 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.449759 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-istio-envoy\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.450026 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.449840 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-workload-socket\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.450026 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.449889 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3554e861-457c-4806-99ee-ad9e27786362-istio-token\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.450146 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.450048 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3554e861-457c-4806-99ee-ad9e27786362-istiod-ca-cert\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.450146 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.450135 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-istio-data\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.450257 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.450237 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmt96\" (UniqueName: \"kubernetes.io/projected/3554e861-457c-4806-99ee-ad9e27786362-kube-api-access-nmt96\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.450315 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.450287 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-workload-socket\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.450368 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.450354 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-credential-socket\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.450423 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.450386 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3554e861-457c-4806-99ee-ad9e27786362-istio-podinfo\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.450476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.450426 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-workload-certs\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.450564 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.450526 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-istio-data\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.450686 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.450661 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-workload-certs\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.450952 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.450918 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-credential-socket\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.451306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.451286 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3554e861-457c-4806-99ee-ad9e27786362-istiod-ca-cert\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.453228 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.453202 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-istio-envoy\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.453432 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.453411 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3554e861-457c-4806-99ee-ad9e27786362-istio-podinfo\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.475151 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.475126 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3554e861-457c-4806-99ee-ad9e27786362-istio-token\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.476194 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.476165 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmt96\" (UniqueName: \"kubernetes.io/projected/3554e861-457c-4806-99ee-ad9e27786362-kube-api-access-nmt96\") pod \"e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.531113 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.531076 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:18.576333 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.576301 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:18.693200 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:18.686750 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:19.103083 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:19.103049 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f"] Oct 30 16:51:19.106759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:19.106723 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:51:19.106944 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:19.106824 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:51:19.106944 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:19.106876 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:51:19.390222 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:19.390112 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:19.890977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:19.890938 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" event={"ID":"3554e861-457c-4806-99ee-ad9e27786362","Type":"ContainerStarted","Data":"08461321eda010962c4ec4af030200c3f2ca572dcb0544a345b68550ea5b7a08"} Oct 30 16:51:19.891172 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:19.891002 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" event={"ID":"3554e861-457c-4806-99ee-ad9e27786362","Type":"ContainerStarted","Data":"29d7dba35aaf5d9afc5991cca87b64f74d995838170310ff35b3f4cb75406839"} Oct 30 16:51:19.961089 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:19.961030 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" podStartSLOduration=1.9610142659999998 podStartE2EDuration="1.961014266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:19.946850179 +0000 UTC m=+1793.657304924" watchObservedRunningTime="2026-10-30 16:51:19.961014266 +0000 UTC m=+1793.671469009" Oct 30 16:51:20.576523 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:20.576488 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:20.584273 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:20.584244 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:20.842809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:20.842677 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:20.898913 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:20.898883 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:20.899985 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:20.899967 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:51:21.554720 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:21.554655 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:21.884555 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:21.884526 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:21.929527 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:21.929493 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb"] Oct 30 16:51:21.937320 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:21.937284 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fd1a-2xtgb"] Oct 30 16:51:22.005809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:22.005719 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:22.412857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:22.412825 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0775c0d-b1be-472c-aa0a-7a17d8352b28" path="/var/lib/kubelet/pods/d0775c0d-b1be-472c-aa0a-7a17d8352b28/volumes" Oct 30 16:51:22.468712 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:22.468677 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:23.149183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:23.149138 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:23.279556 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:23.279515 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:23.836835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:23.836734 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:24.853942 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:24.853902 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:24.973621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:24.973588 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:25.004964 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:25.004929 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp"] Oct 30 16:51:25.011119 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:25.011098 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" Oct 30 16:51:25.032690 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:25.032663 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp"] Oct 30 16:51:25.047825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:25.040095 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-480a60-dockercfg-xcxng\"" Oct 30 16:51:25.047825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:25.040304 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-480a60\"" Oct 30 16:51:25.094462 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:25.094426 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q4f5\" (UniqueName: \"kubernetes.io/projected/bb3d3849-9fcb-414c-aece-8402ba38ab8d-kube-api-access-9q4f5\") pod \"payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp\" (UID: \"bb3d3849-9fcb-414c-aece-8402ba38ab8d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" Oct 30 16:51:25.094667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:25.094630 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3d3849-9fcb-414c-aece-8402ba38ab8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp\" (UID: \"bb3d3849-9fcb-414c-aece-8402ba38ab8d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" Oct 30 16:51:25.206842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:25.196029 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3d3849-9fcb-414c-aece-8402ba38ab8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp\" (UID: \"bb3d3849-9fcb-414c-aece-8402ba38ab8d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" Oct 30 16:51:25.206842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:25.196344 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q4f5\" (UniqueName: \"kubernetes.io/projected/bb3d3849-9fcb-414c-aece-8402ba38ab8d-kube-api-access-9q4f5\") pod \"payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp\" (UID: \"bb3d3849-9fcb-414c-aece-8402ba38ab8d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" Oct 30 16:51:25.206842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:25.197571 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3d3849-9fcb-414c-aece-8402ba38ab8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp\" (UID: \"bb3d3849-9fcb-414c-aece-8402ba38ab8d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" Oct 30 16:51:25.278795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:25.278462 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q4f5\" (UniqueName: \"kubernetes.io/projected/bb3d3849-9fcb-414c-aece-8402ba38ab8d-kube-api-access-9q4f5\") pod \"payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp\" (UID: \"bb3d3849-9fcb-414c-aece-8402ba38ab8d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" Oct 30 16:51:25.327792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:25.327290 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" Oct 30 16:51:25.749000 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:25.748970 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp"] Oct 30 16:51:25.757234 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:51:25.757169 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3d3849_9fcb_414c_aece_8402ba38ab8d.slice/crio-0830b4e23a42ddd600ae11afdcf6f011a0b1abdb298d524c070b3312e886941b WatchSource:0}: Error finding container 0830b4e23a42ddd600ae11afdcf6f011a0b1abdb298d524c070b3312e886941b: Status 404 returned error can't find the container with id 0830b4e23a42ddd600ae11afdcf6f011a0b1abdb298d524c070b3312e886941b Oct 30 16:51:25.778446 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:25.778355 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:25.950200 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:25.950152 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" event={"ID":"bb3d3849-9fcb-414c-aece-8402ba38ab8d","Type":"ContainerStarted","Data":"ec0cbfb558beaf9bbbeba16624031b8e98b14d1f4b419699b53b492113055470"} Oct 30 16:51:25.950666 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:25.950225 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" Oct 30 16:51:25.950666 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:25.950266 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" event={"ID":"bb3d3849-9fcb-414c-aece-8402ba38ab8d","Type":"ContainerStarted","Data":"0830b4e23a42ddd600ae11afdcf6f011a0b1abdb298d524c070b3312e886941b"} Oct 30 16:51:25.994946 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:25.994887 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" podStartSLOduration=1.9948720450000001 podStartE2EDuration="1.994872045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:25.990202498 +0000 UTC m=+1799.700657241" watchObservedRunningTime="2026-10-30 16:51:25.994872045 +0000 UTC m=+1799.705326789" Oct 30 16:51:27.569971 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:27.569937 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:29.550364 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:29.549579 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:29.960573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:29.960533 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:30.094066 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:30.094025 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:31.304937 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:31.304904 6202 scope.go:122] "RemoveContainer" containerID="a8fd6463c01f9eabcc795014051d6e64bea794bf171ab752e34004775cd5513b" Oct 30 16:51:31.370611 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:31.370580 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:32.754509 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:32.754455 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:33.428209 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:33.428178 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:33.599549 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:33.599514 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg"] Oct 30 16:51:33.611010 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:33.610973 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e5674c-2xmxg"] Oct 30 16:51:33.615530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:33.615498 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:33.742258 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:33.742083 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:34.138026 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:34.137990 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:34.413077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:34.412989 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e95bc524-7472-4b9a-a0a5-2f12d396170a" path="/var/lib/kubelet/pods/e95bc524-7472-4b9a-a0a5-2f12d396170a/volumes" Oct 30 16:51:35.484747 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:35.484667 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:36.970576 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:36.970547 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" Oct 30 16:51:37.205536 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:37.205505 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:37.968165 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:37.968125 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:39.034191 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:39.034131 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:39.383136 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:39.383105 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:39.713135 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:39.713039 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:40.336422 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:40.336380 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk"] Oct 30 16:51:40.336930 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:40.336673 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" podUID="8109a3b3-6567-49bc-b409-58309ac8bb62" containerName="istio-proxy" containerID="cri-o://00e14e56c4aea6b822e7592e983f035c6a4efb8415eb95cd32014af78ddde82b" gracePeriod=30 Oct 30 16:51:40.375036 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:40.374998 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:40.862658 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:40.862608 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:40.919809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:40.919757 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf"] Oct 30 16:51:40.931578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:40.931417 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf"] Oct 30 16:51:40.931578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:40.931545 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf" Oct 30 16:51:40.958530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:40.957011 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:51:40.958530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:40.957060 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:51:40.958530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:40.957125 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:51:40.958530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:40.957254 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:51:41.030200 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:41.030162 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv5w6\" (UniqueName: \"kubernetes.io/projected/71a8e320-7b5b-4235-910e-d65631f01cdd-kube-api-access-hv5w6\") pod \"maas-api-revoke-keys-e2e-api-b-407eab-6svsf\" (UID: \"71a8e320-7b5b-4235-910e-d65631f01cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf" Oct 30 16:51:41.030498 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:41.030474 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a8e320-7b5b-4235-910e-d65631f01cdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-407eab-6svsf\" (UID: \"71a8e320-7b5b-4235-910e-d65631f01cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf" Oct 30 16:51:41.030557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:41.030522 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a8e320-7b5b-4235-910e-d65631f01cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-407eab-6svsf\" (UID: \"71a8e320-7b5b-4235-910e-d65631f01cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf" Oct 30 16:51:41.131738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:41.131652 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a8e320-7b5b-4235-910e-d65631f01cdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-407eab-6svsf\" (UID: \"71a8e320-7b5b-4235-910e-d65631f01cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf" Oct 30 16:51:41.131738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:41.131700 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a8e320-7b5b-4235-910e-d65631f01cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-407eab-6svsf\" (UID: \"71a8e320-7b5b-4235-910e-d65631f01cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf" Oct 30 16:51:41.132022 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:41.131752 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv5w6\" (UniqueName: \"kubernetes.io/projected/71a8e320-7b5b-4235-910e-d65631f01cdd-kube-api-access-hv5w6\") pod \"maas-api-revoke-keys-e2e-api-b-407eab-6svsf\" (UID: \"71a8e320-7b5b-4235-910e-d65631f01cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf" Oct 30 16:51:41.132501 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:41.132483 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a8e320-7b5b-4235-910e-d65631f01cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-407eab-6svsf\" (UID: \"71a8e320-7b5b-4235-910e-d65631f01cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf" Oct 30 16:51:41.134301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:41.134279 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a8e320-7b5b-4235-910e-d65631f01cdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-407eab-6svsf\" (UID: \"71a8e320-7b5b-4235-910e-d65631f01cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf" Oct 30 16:51:41.155720 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:41.155682 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv5w6\" (UniqueName: \"kubernetes.io/projected/71a8e320-7b5b-4235-910e-d65631f01cdd-kube-api-access-hv5w6\") pod \"maas-api-revoke-keys-e2e-api-b-407eab-6svsf\" (UID: \"71a8e320-7b5b-4235-910e-d65631f01cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf" Oct 30 16:51:41.264413 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:41.264372 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf" Oct 30 16:51:41.572941 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:41.572909 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf"] Oct 30 16:51:41.574330 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:51:41.574301 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a8e320_7b5b_4235_910e_d65631f01cdd.slice/crio-6fd4ace87ec29172e54c80c05b4a333101260e8e24d2a9f9882326fc298f4c1b WatchSource:0}: Error finding container 6fd4ace87ec29172e54c80c05b4a333101260e8e24d2a9f9882326fc298f4c1b: Status 404 returned error can't find the container with id 6fd4ace87ec29172e54c80c05b4a333101260e8e24d2a9f9882326fc298f4c1b Oct 30 16:51:41.732619 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:41.732576 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:41.859427 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:41.859386 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:42.057833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:42.057799 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj"] Oct 30 16:51:42.058100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:42.058065 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" podUID="dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" containerName="istio-proxy" containerID="cri-o://63a1cf4ca3660b65bd553fb6a36a4f490b5a3b30966b4f61f869f0dde5fb7cc4" gracePeriod=30 Oct 30 16:51:42.124529 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:42.124432 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:42.144395 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:42.144260 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf" event={"ID":"71a8e320-7b5b-4235-910e-d65631f01cdd","Type":"ContainerStarted","Data":"4938d26d5e49169ffd8473950da9b7cc4d114d9c3b43c80e726f0956633a76f3"} Oct 30 16:51:42.144395 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:42.144324 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf" event={"ID":"71a8e320-7b5b-4235-910e-d65631f01cdd","Type":"ContainerStarted","Data":"6fd4ace87ec29172e54c80c05b4a333101260e8e24d2a9f9882326fc298f4c1b"} Oct 30 16:51:42.225792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:42.218150 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf" podStartSLOduration=2.21812661 podStartE2EDuration="2.21812661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:42.187194891 +0000 UTC m=+1815.897649636" watchObservedRunningTime="2026-10-30 16:51:42.21812661 +0000 UTC m=+1815.928581355" Oct 30 16:51:42.862749 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:42.862669 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:42.992742 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:42.992651 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:43.154253 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:43.154216 6202 generic.go:356] "Generic (PLEG): container finished" podID="71a8e320-7b5b-4235-910e-d65631f01cdd" containerID="4938d26d5e49169ffd8473950da9b7cc4d114d9c3b43c80e726f0956633a76f3" exitCode=0 Oct 30 16:51:43.154443 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:43.154300 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf" event={"ID":"71a8e320-7b5b-4235-910e-d65631f01cdd","Type":"ContainerDied","Data":"4938d26d5e49169ffd8473950da9b7cc4d114d9c3b43c80e726f0956633a76f3"} Oct 30 16:51:43.509188 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:43.509157 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:44.078693 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:51:44.078658 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:51:44.116717 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:44.116690 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:44.349060 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:44.348996 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf" Oct 30 16:51:44.402431 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:44.402397 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a8e320-7b5b-4235-910e-d65631f01cdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a8e320-7b5b-4235-910e-d65631f01cdd-maas-api-openshift-ca\") pod \"71a8e320-7b5b-4235-910e-d65631f01cdd\" (UID: \"71a8e320-7b5b-4235-910e-d65631f01cdd\") " Oct 30 16:51:44.402628 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:44.402520 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a8e320-7b5b-4235-910e-d65631f01cdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a8e320-7b5b-4235-910e-d65631f01cdd-maas-api-xks-ca\") pod \"71a8e320-7b5b-4235-910e-d65631f01cdd\" (UID: \"71a8e320-7b5b-4235-910e-d65631f01cdd\") " Oct 30 16:51:44.402628 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:44.402621 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a8e320-7b5b-4235-910e-d65631f01cdd-kube-api-access-hv5w6\" (UniqueName: \"kubernetes.io/projected/71a8e320-7b5b-4235-910e-d65631f01cdd-kube-api-access-hv5w6\") pod \"71a8e320-7b5b-4235-910e-d65631f01cdd\" (UID: \"71a8e320-7b5b-4235-910e-d65631f01cdd\") " Oct 30 16:51:44.403150 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:44.403121 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a8e320-7b5b-4235-910e-d65631f01cdd-maas-api-openshift-ca" pod "71a8e320-7b5b-4235-910e-d65631f01cdd" (UID: "71a8e320-7b5b-4235-910e-d65631f01cdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:44.403557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:44.403532 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a8e320-7b5b-4235-910e-d65631f01cdd-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:44.406284 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:44.406248 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a8e320-7b5b-4235-910e-d65631f01cdd-kube-api-access-hv5w6" pod "71a8e320-7b5b-4235-910e-d65631f01cdd" (UID: "71a8e320-7b5b-4235-910e-d65631f01cdd"). InnerVolumeSpecName "kube-api-access-hv5w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:44.413384 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:44.413353 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a8e320-7b5b-4235-910e-d65631f01cdd-maas-api-xks-ca" pod "71a8e320-7b5b-4235-910e-d65631f01cdd" (UID: "71a8e320-7b5b-4235-910e-d65631f01cdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:44.505461 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:44.505426 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a8e320-7b5b-4235-910e-d65631f01cdd-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:44.505461 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:44.505455 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv5w6\" (UniqueName: \"kubernetes.io/projected/71a8e320-7b5b-4235-910e-d65631f01cdd-kube-api-access-hv5w6\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:44.738238 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:44.738147 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:44.902805 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:44.898854 6202 patch_prober.go:29] interesting pod/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:51:44.902805 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:44.898912 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" podUID="dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:51:45.179703 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.179669 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf" Oct 30 16:51:45.179703 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.179689 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf" event={"ID":"71a8e320-7b5b-4235-910e-d65631f01cdd","Type":"ContainerDied","Data":"6fd4ace87ec29172e54c80c05b4a333101260e8e24d2a9f9882326fc298f4c1b"} Oct 30 16:51:45.180189 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.179792 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fd4ace87ec29172e54c80c05b4a333101260e8e24d2a9f9882326fc298f4c1b" Oct 30 16:51:45.621955 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.621917 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:45.727289 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.727210 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:51:45.850451 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.850025 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:45.856937 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.856898 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb"] Oct 30 16:51:45.861482 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.861450 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca5971-5vdgb"] Oct 30 16:51:45.886025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.885986 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-podinfo\") pod \"8109a3b3-6567-49bc-b409-58309ac8bb62\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " Oct 30 16:51:45.886293 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.886049 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-token\" (UniqueName: \"kubernetes.io/projected/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-token\") pod \"8109a3b3-6567-49bc-b409-58309ac8bb62\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " Oct 30 16:51:45.886293 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.886175 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-workload-socket\") pod \"8109a3b3-6567-49bc-b409-58309ac8bb62\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " Oct 30 16:51:45.886293 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.886270 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-credential-socket\") pod \"8109a3b3-6567-49bc-b409-58309ac8bb62\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " Oct 30 16:51:45.886562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.886469 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-workload-certs\") pod \"8109a3b3-6567-49bc-b409-58309ac8bb62\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " Oct 30 16:51:45.886562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.886538 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8109a3b3-6567-49bc-b409-58309ac8bb62-kube-api-access-hcr7z\" (UniqueName: \"kubernetes.io/projected/8109a3b3-6567-49bc-b409-58309ac8bb62-kube-api-access-hcr7z\") pod \"8109a3b3-6567-49bc-b409-58309ac8bb62\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " Oct 30 16:51:45.886670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.886636 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-envoy\") pod \"8109a3b3-6567-49bc-b409-58309ac8bb62\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " Oct 30 16:51:45.886733 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.886708 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-data\") pod \"8109a3b3-6567-49bc-b409-58309ac8bb62\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " Oct 30 16:51:45.886839 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.886812 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8109a3b3-6567-49bc-b409-58309ac8bb62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8109a3b3-6567-49bc-b409-58309ac8bb62-istiod-ca-cert\") pod \"8109a3b3-6567-49bc-b409-58309ac8bb62\" (UID: \"8109a3b3-6567-49bc-b409-58309ac8bb62\") " Oct 30 16:51:45.888102 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.887830 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8109a3b3-6567-49bc-b409-58309ac8bb62-istiod-ca-cert" pod "8109a3b3-6567-49bc-b409-58309ac8bb62" (UID: "8109a3b3-6567-49bc-b409-58309ac8bb62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:45.888102 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.888079 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-workload-certs" pod "8109a3b3-6567-49bc-b409-58309ac8bb62" (UID: "8109a3b3-6567-49bc-b409-58309ac8bb62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:45.888283 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.888197 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-data" pod "8109a3b3-6567-49bc-b409-58309ac8bb62" (UID: "8109a3b3-6567-49bc-b409-58309ac8bb62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:45.888637 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.888428 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-workload-socket" pod "8109a3b3-6567-49bc-b409-58309ac8bb62" (UID: "8109a3b3-6567-49bc-b409-58309ac8bb62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:45.888835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.888799 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-credential-socket" pod "8109a3b3-6567-49bc-b409-58309ac8bb62" (UID: "8109a3b3-6567-49bc-b409-58309ac8bb62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:45.891313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.891258 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-podinfo" pod "8109a3b3-6567-49bc-b409-58309ac8bb62" (UID: "8109a3b3-6567-49bc-b409-58309ac8bb62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:51:45.891454 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.891426 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-token" pod "8109a3b3-6567-49bc-b409-58309ac8bb62" (UID: "8109a3b3-6567-49bc-b409-58309ac8bb62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:45.911020 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.910536 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8109a3b3-6567-49bc-b409-58309ac8bb62-kube-api-access-hcr7z" pod "8109a3b3-6567-49bc-b409-58309ac8bb62" (UID: "8109a3b3-6567-49bc-b409-58309ac8bb62"). InnerVolumeSpecName "kube-api-access-hcr7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:45.912253 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.912206 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-envoy" pod "8109a3b3-6567-49bc-b409-58309ac8bb62" (UID: "8109a3b3-6567-49bc-b409-58309ac8bb62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:45.963488 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.963412 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:45.989670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.989560 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:45.989670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.989606 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8109a3b3-6567-49bc-b409-58309ac8bb62-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:45.989670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.989621 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:45.989670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.989633 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:45.989670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.989646 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:45.989670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.989658 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:45.989670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.989671 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:45.990154 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.989685 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcr7z\" (UniqueName: \"kubernetes.io/projected/8109a3b3-6567-49bc-b409-58309ac8bb62-kube-api-access-hcr7z\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:45.990154 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:45.989699 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8109a3b3-6567-49bc-b409-58309ac8bb62-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:46.075436 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.075402 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:46.188192 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.188143 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:46.194446 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.194406 6202 generic.go:356] "Generic (PLEG): container finished" podID="8109a3b3-6567-49bc-b409-58309ac8bb62" containerID="00e14e56c4aea6b822e7592e983f035c6a4efb8415eb95cd32014af78ddde82b" exitCode=0 Oct 30 16:51:46.194619 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.194462 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" event={"ID":"8109a3b3-6567-49bc-b409-58309ac8bb62","Type":"ContainerDied","Data":"00e14e56c4aea6b822e7592e983f035c6a4efb8415eb95cd32014af78ddde82b"} Oct 30 16:51:46.194619 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.194476 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" Oct 30 16:51:46.194619 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.194510 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk" event={"ID":"8109a3b3-6567-49bc-b409-58309ac8bb62","Type":"ContainerDied","Data":"9bf029aed60a9d7c509ce261d109877771cc2c217a1862893dcf3d5b88e6db1a"} Oct 30 16:51:46.194619 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.194533 6202 scope.go:122] "RemoveContainer" containerID="00e14e56c4aea6b822e7592e983f035c6a4efb8415eb95cd32014af78ddde82b" Oct 30 16:51:46.207687 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.207664 6202 scope.go:122] "RemoveContainer" containerID="5029ec767242166c92c33ab92d6d4013c2b092d235cdc5307e775c193f1a8b6f" Oct 30 16:51:46.218528 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.218504 6202 scope.go:122] "RemoveContainer" containerID="00e14e56c4aea6b822e7592e983f035c6a4efb8415eb95cd32014af78ddde82b" Oct 30 16:51:46.218847 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:51:46.218818 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e14e56c4aea6b822e7592e983f035c6a4efb8415eb95cd32014af78ddde82b\": container with ID starting with 00e14e56c4aea6b822e7592e983f035c6a4efb8415eb95cd32014af78ddde82b not found: ID does not exist" containerID="00e14e56c4aea6b822e7592e983f035c6a4efb8415eb95cd32014af78ddde82b" Oct 30 16:51:46.218917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.218861 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e14e56c4aea6b822e7592e983f035c6a4efb8415eb95cd32014af78ddde82b"} err="failed to get container status \"00e14e56c4aea6b822e7592e983f035c6a4efb8415eb95cd32014af78ddde82b\": rpc error: code = NotFound desc = could not find container \"00e14e56c4aea6b822e7592e983f035c6a4efb8415eb95cd32014af78ddde82b\": container with ID starting with 00e14e56c4aea6b822e7592e983f035c6a4efb8415eb95cd32014af78ddde82b not found: ID does not exist" Oct 30 16:51:46.218917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.218887 6202 scope.go:122] "RemoveContainer" containerID="5029ec767242166c92c33ab92d6d4013c2b092d235cdc5307e775c193f1a8b6f" Oct 30 16:51:46.219179 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:51:46.219153 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5029ec767242166c92c33ab92d6d4013c2b092d235cdc5307e775c193f1a8b6f\": container with ID starting with 5029ec767242166c92c33ab92d6d4013c2b092d235cdc5307e775c193f1a8b6f not found: ID does not exist" containerID="5029ec767242166c92c33ab92d6d4013c2b092d235cdc5307e775c193f1a8b6f" Oct 30 16:51:46.219290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.219183 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5029ec767242166c92c33ab92d6d4013c2b092d235cdc5307e775c193f1a8b6f"} err="failed to get container status \"5029ec767242166c92c33ab92d6d4013c2b092d235cdc5307e775c193f1a8b6f\": rpc error: code = NotFound desc = could not find container \"5029ec767242166c92c33ab92d6d4013c2b092d235cdc5307e775c193f1a8b6f\": container with ID starting with 5029ec767242166c92c33ab92d6d4013c2b092d235cdc5307e775c193f1a8b6f not found: ID does not exist" Oct 30 16:51:46.261156 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.261058 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk"] Oct 30 16:51:46.272155 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.272121 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-00fd1a-openshift-default-7b67b5bdbf-twhtk"] Oct 30 16:51:46.419904 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.419872 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:46.428840 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.428797 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="065c113c-8979-4fde-871d-3f88dea368b3" path="/var/lib/kubelet/pods/065c113c-8979-4fde-871d-3f88dea368b3/volumes" Oct 30 16:51:46.429578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.429548 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8109a3b3-6567-49bc-b409-58309ac8bb62" path="/var/lib/kubelet/pods/8109a3b3-6567-49bc-b409-58309ac8bb62/volumes" Oct 30 16:51:46.436966 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.436932 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2"] Oct 30 16:51:46.437207 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.437179 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" podUID="cb03379b-4139-4f06-a67b-73249017ccb8" containerName="istio-proxy" containerID="cri-o://9df63ede14499c496f3683a2c91c76a54a3a618cbdd878a58ae2b1c585ba29c8" gracePeriod=30 Oct 30 16:51:46.466171 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.459830 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw"] Oct 30 16:51:46.466171 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.460233 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" podUID="75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" containerName="istio-proxy" containerID="cri-o://e81e41ad3c9d14fb52a942d8e0aa082a104c9627036a03050156227379e40b66" gracePeriod=30 Oct 30 16:51:46.468833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.468302 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq"] Oct 30 16:51:46.468833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.468579 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" podUID="182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" containerName="istio-proxy" containerID="cri-o://d052b3803f05d78f678ae6f22e029fefe881aa8ef5643bc8f4b7d2ec7a9f4bf1" gracePeriod=30 Oct 30 16:51:46.474809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.474737 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr"] Oct 30 16:51:46.474995 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.474971 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" podUID="53877150-e074-4622-afb6-eb0699548ecb" containerName="istio-proxy" containerID="cri-o://e372756145a9b4243d25c1b2a64eb92d630e8f9bf3706ffa27e7d9a9560e9ef5" gracePeriod=30 Oct 30 16:51:46.559612 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:46.559474 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:47.206790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.206018 6202 generic.go:356] "Generic (PLEG): container finished" podID="dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" containerID="63a1cf4ca3660b65bd553fb6a36a4f490b5a3b30966b4f61f869f0dde5fb7cc4" exitCode=0 Oct 30 16:51:47.206790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.206089 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" event={"ID":"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b","Type":"ContainerDied","Data":"63a1cf4ca3660b65bd553fb6a36a4f490b5a3b30966b4f61f869f0dde5fb7cc4"} Oct 30 16:51:47.206790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.206153 6202 scope.go:122] "RemoveContainer" containerID="99c07b28fa71a9da8c59c9aa59e2b90745a025152b2d1d22e4ebce6fbcc070f7" Oct 30 16:51:47.369680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.369621 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:51:47.442206 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.442167 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-envoy\") pod \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " Oct 30 16:51:47.442385 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.442254 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-data\") pod \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " Oct 30 16:51:47.442385 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.442312 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-podinfo\") pod \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " Oct 30 16:51:47.442597 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.442430 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-workload-certs\") pod \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " Oct 30 16:51:47.442597 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.442535 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-workload-socket\") pod \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " Oct 30 16:51:47.442597 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.442563 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-token\" (UniqueName: \"kubernetes.io/projected/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-token\") pod \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " Oct 30 16:51:47.442732 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.442666 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-credential-socket\") pod \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " Oct 30 16:51:47.442982 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.442827 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istiod-ca-cert\") pod \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " Oct 30 16:51:47.442982 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.442862 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-kube-api-access-tj2pd\" (UniqueName: \"kubernetes.io/projected/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-kube-api-access-tj2pd\") pod \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\" (UID: \"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b\") " Oct 30 16:51:47.443119 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.443040 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-data" pod "dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" (UID: "dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:47.443290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.443268 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-workload-certs" pod "dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" (UID: "dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:47.443637 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.443613 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:47.443759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.443645 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:47.444255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.444227 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istiod-ca-cert" pod "dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" (UID: "dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:47.444573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.444549 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-workload-socket" pod "dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" (UID: "dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:47.444889 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.444861 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-credential-socket" pod "dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" (UID: "dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:47.446204 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.446183 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-token" pod "dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" (UID: "dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:47.447199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.447160 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-kube-api-access-tj2pd" pod "dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" (UID: "dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b"). InnerVolumeSpecName "kube-api-access-tj2pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:47.450201 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.450170 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-podinfo" pod "dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" (UID: "dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:51:47.450936 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.450913 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-envoy" pod "dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" (UID: "dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:47.545149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.545102 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:47.545149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.545148 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj2pd\" (UniqueName: \"kubernetes.io/projected/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-kube-api-access-tj2pd\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:47.545149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.545159 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:47.545149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.545168 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:47.545448 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.545177 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:47.545448 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.545185 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:47.545448 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.545193 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:47.882470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.882424 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:47.888866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.888834 6202 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:51:47.888998 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:47.888895 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" podUID="cb03379b-4139-4f06-a67b-73249017ccb8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:51:48.058570 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.058535 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg"] Oct 30 16:51:48.059757 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.059739 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8109a3b3-6567-49bc-b409-58309ac8bb62" containerName="istio-proxy" Oct 30 16:51:48.059837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.059760 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8109a3b3-6567-49bc-b409-58309ac8bb62" containerName="istio-proxy" Oct 30 16:51:48.059837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.059797 6202 cpu_manager.go:414] "containerMap: removing container" podUID="dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" containerName="istio-proxy" Oct 30 16:51:48.059837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.059805 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" containerName="istio-proxy" Oct 30 16:51:48.059837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.059834 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8109a3b3-6567-49bc-b409-58309ac8bb62" containerName="istio-proxy" Oct 30 16:51:48.059978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.059842 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8109a3b3-6567-49bc-b409-58309ac8bb62" containerName="istio-proxy" Oct 30 16:51:48.059978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.059856 6202 cpu_manager.go:414] "containerMap: removing container" podUID="71a8e320-7b5b-4235-910e-d65631f01cdd" containerName="revoke-keys" Oct 30 16:51:48.059978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.059863 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a8e320-7b5b-4235-910e-d65631f01cdd" containerName="revoke-keys" Oct 30 16:51:48.059978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.059893 6202 cpu_manager.go:414] "containerMap: removing container" podUID="dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" containerName="istio-proxy" Oct 30 16:51:48.059978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.059901 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" containerName="istio-proxy" Oct 30 16:51:48.060149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.060137 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8109a3b3-6567-49bc-b409-58309ac8bb62" containerName="istio-proxy" Oct 30 16:51:48.060185 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.060151 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" containerName="istio-proxy" Oct 30 16:51:48.060185 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.060162 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8109a3b3-6567-49bc-b409-58309ac8bb62" containerName="istio-proxy" Oct 30 16:51:48.060185 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.060174 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a8e320-7b5b-4235-910e-d65631f01cdd" containerName="revoke-keys" Oct 30 16:51:48.060294 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.060184 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" containerName="istio-proxy" Oct 30 16:51:48.065666 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.065643 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg" Oct 30 16:51:48.082090 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.082061 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:51:48.082090 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.082087 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:51:48.082285 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.082067 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg"] Oct 30 16:51:48.082285 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.082119 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:51:48.082481 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.082457 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:51:48.155119 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.155035 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78l6g\" (UniqueName: \"kubernetes.io/projected/d90175c6-171e-41b4-9fe7-3370494ef31d-kube-api-access-78l6g\") pod \"maas-api-revoke-sub-e71b55cf-26qkg\" (UID: \"d90175c6-171e-41b4-9fe7-3370494ef31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg" Oct 30 16:51:48.155271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.155138 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90175c6-171e-41b4-9fe7-3370494ef31d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e71b55cf-26qkg\" (UID: \"d90175c6-171e-41b4-9fe7-3370494ef31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg" Oct 30 16:51:48.155271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.155207 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90175c6-171e-41b4-9fe7-3370494ef31d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e71b55cf-26qkg\" (UID: \"d90175c6-171e-41b4-9fe7-3370494ef31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg" Oct 30 16:51:48.217003 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.216970 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" Oct 30 16:51:48.217425 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.216969 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj" event={"ID":"dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b","Type":"ContainerDied","Data":"7b36c227cd7d250d4ef967ea1496cbdf3762e0a1c19271aa94af63d83869ebe0"} Oct 30 16:51:48.217425 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.217109 6202 scope.go:122] "RemoveContainer" containerID="63a1cf4ca3660b65bd553fb6a36a4f490b5a3b30966b4f61f869f0dde5fb7cc4" Oct 30 16:51:48.256976 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.256947 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78l6g\" (UniqueName: \"kubernetes.io/projected/d90175c6-171e-41b4-9fe7-3370494ef31d-kube-api-access-78l6g\") pod \"maas-api-revoke-sub-e71b55cf-26qkg\" (UID: \"d90175c6-171e-41b4-9fe7-3370494ef31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg" Oct 30 16:51:48.257124 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.257019 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90175c6-171e-41b4-9fe7-3370494ef31d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e71b55cf-26qkg\" (UID: \"d90175c6-171e-41b4-9fe7-3370494ef31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg" Oct 30 16:51:48.257124 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.257083 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90175c6-171e-41b4-9fe7-3370494ef31d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e71b55cf-26qkg\" (UID: \"d90175c6-171e-41b4-9fe7-3370494ef31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg" Oct 30 16:51:48.257936 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.257902 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90175c6-171e-41b4-9fe7-3370494ef31d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e71b55cf-26qkg\" (UID: \"d90175c6-171e-41b4-9fe7-3370494ef31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg" Oct 30 16:51:48.259700 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.259676 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90175c6-171e-41b4-9fe7-3370494ef31d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e71b55cf-26qkg\" (UID: \"d90175c6-171e-41b4-9fe7-3370494ef31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg" Oct 30 16:51:48.272456 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.272428 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj"] Oct 30 16:51:48.276813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.276757 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e5674c-openshift-default-fbb9899-mgjmj"] Oct 30 16:51:48.280579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.280548 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78l6g\" (UniqueName: \"kubernetes.io/projected/d90175c6-171e-41b4-9fe7-3370494ef31d-kube-api-access-78l6g\") pod \"maas-api-revoke-sub-e71b55cf-26qkg\" (UID: \"d90175c6-171e-41b4-9fe7-3370494ef31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg" Oct 30 16:51:48.384598 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.384567 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg" Oct 30 16:51:48.414535 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.414462 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b" path="/var/lib/kubelet/pods/dbbb77be-53e1-4ede-bfe4-59f7f2cf5d0b/volumes" Oct 30 16:51:48.692433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:48.692377 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg"] Oct 30 16:51:49.228391 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:49.228292 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg" event={"ID":"d90175c6-171e-41b4-9fe7-3370494ef31d","Type":"ContainerStarted","Data":"1f0f5ff7273daaef38573e8dba2c0dfd2381a7b2c203e0082418ae4a5dcf60be"} Oct 30 16:51:49.228391 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:49.228355 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg" event={"ID":"d90175c6-171e-41b4-9fe7-3370494ef31d","Type":"ContainerStarted","Data":"c1a27018dd6f5e63512d80d0e35cc9df943d7a2eb389592da56760a0f5dd0227"} Oct 30 16:51:49.270126 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:49.270069 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg" podStartSLOduration=1.27005529 podStartE2EDuration="1.27005529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:49.268495945 +0000 UTC m=+1822.978950691" watchObservedRunningTime="2026-10-30 16:51:49.27005529 +0000 UTC m=+1822.980510035" Oct 30 16:51:49.708420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:49.708384 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:50.237941 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:50.237905 6202 generic.go:356] "Generic (PLEG): container finished" podID="d90175c6-171e-41b4-9fe7-3370494ef31d" containerID="1f0f5ff7273daaef38573e8dba2c0dfd2381a7b2c203e0082418ae4a5dcf60be" exitCode=0 Oct 30 16:51:50.238311 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:50.237997 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg" event={"ID":"d90175c6-171e-41b4-9fe7-3370494ef31d","Type":"ContainerDied","Data":"1f0f5ff7273daaef38573e8dba2c0dfd2381a7b2c203e0082418ae4a5dcf60be"} Oct 30 16:51:51.306561 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.306507 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:51.312912 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.312885 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md"] Oct 30 16:51:51.347040 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.346555 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz"] Oct 30 16:51:51.495547 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.495516 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:51.695659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.695029 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:51.711779 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.711679 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg" Oct 30 16:51:51.813224 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.813183 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d90175c6-171e-41b4-9fe7-3370494ef31d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90175c6-171e-41b4-9fe7-3370494ef31d-maas-api-openshift-ca\") pod \"d90175c6-171e-41b4-9fe7-3370494ef31d\" (UID: \"d90175c6-171e-41b4-9fe7-3370494ef31d\") " Oct 30 16:51:51.813391 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.813374 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d90175c6-171e-41b4-9fe7-3370494ef31d-kube-api-access-78l6g\" (UniqueName: \"kubernetes.io/projected/d90175c6-171e-41b4-9fe7-3370494ef31d-kube-api-access-78l6g\") pod \"d90175c6-171e-41b4-9fe7-3370494ef31d\" (UID: \"d90175c6-171e-41b4-9fe7-3370494ef31d\") " Oct 30 16:51:51.813459 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.813433 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d90175c6-171e-41b4-9fe7-3370494ef31d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90175c6-171e-41b4-9fe7-3370494ef31d-maas-api-xks-ca\") pod \"d90175c6-171e-41b4-9fe7-3370494ef31d\" (UID: \"d90175c6-171e-41b4-9fe7-3370494ef31d\") " Oct 30 16:51:51.814542 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.814515 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90175c6-171e-41b4-9fe7-3370494ef31d-maas-api-openshift-ca" pod "d90175c6-171e-41b4-9fe7-3370494ef31d" (UID: "d90175c6-171e-41b4-9fe7-3370494ef31d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:51.817571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.817511 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90175c6-171e-41b4-9fe7-3370494ef31d-kube-api-access-78l6g" pod "d90175c6-171e-41b4-9fe7-3370494ef31d" (UID: "d90175c6-171e-41b4-9fe7-3370494ef31d"). InnerVolumeSpecName "kube-api-access-78l6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:51.818069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.818035 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d90175c6-171e-41b4-9fe7-3370494ef31d-maas-api-xks-ca" pod "d90175c6-171e-41b4-9fe7-3370494ef31d" (UID: "d90175c6-171e-41b4-9fe7-3370494ef31d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:51.854028 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.854001 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:51:51.878857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.878732 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:51:51.915157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.915129 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-kube-api-access-q9gdf\" (UniqueName: \"kubernetes.io/projected/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-kube-api-access-q9gdf\") pod \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " Oct 30 16:51:51.915781 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.915737 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb03379b-4139-4f06-a67b-73249017ccb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb03379b-4139-4f06-a67b-73249017ccb8-istiod-ca-cert\") pod \"cb03379b-4139-4f06-a67b-73249017ccb8\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " Oct 30 16:51:51.916366 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.916261 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb03379b-4139-4f06-a67b-73249017ccb8-istiod-ca-cert" pod "cb03379b-4139-4f06-a67b-73249017ccb8" (UID: "cb03379b-4139-4f06-a67b-73249017ccb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:51.916738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.916356 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-istio-data\") pod \"cb03379b-4139-4f06-a67b-73249017ccb8\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " Oct 30 16:51:51.916738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.916554 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb03379b-4139-4f06-a67b-73249017ccb8-kube-api-access-g4fws\" (UniqueName: \"kubernetes.io/projected/cb03379b-4139-4f06-a67b-73249017ccb8-kube-api-access-g4fws\") pod \"cb03379b-4139-4f06-a67b-73249017ccb8\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " Oct 30 16:51:51.916738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.916603 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb03379b-4139-4f06-a67b-73249017ccb8-istio-token\" (UniqueName: \"kubernetes.io/projected/cb03379b-4139-4f06-a67b-73249017ccb8-istio-token\") pod \"cb03379b-4139-4f06-a67b-73249017ccb8\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " Oct 30 16:51:51.916738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.916726 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-istio-data" pod "cb03379b-4139-4f06-a67b-73249017ccb8" (UID: "cb03379b-4139-4f06-a67b-73249017ccb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:51.917003 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.916985 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-token\" (UniqueName: \"kubernetes.io/projected/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-token\") pod \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " Oct 30 16:51:51.917094 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.917039 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-workload-certs\") pod \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " Oct 30 16:51:51.917094 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.917075 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-workload-socket\") pod \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " Oct 30 16:51:51.917197 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.917151 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-workload-socket\") pod \"cb03379b-4139-4f06-a67b-73249017ccb8\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " Oct 30 16:51:51.917197 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.917189 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-podinfo\") pod \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " Oct 30 16:51:51.917313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.917237 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-istio-envoy\") pod \"cb03379b-4139-4f06-a67b-73249017ccb8\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " Oct 30 16:51:51.917313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.917295 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-credential-socket\") pod \"cb03379b-4139-4f06-a67b-73249017ccb8\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " Oct 30 16:51:51.917413 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.917331 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-credential-socket\") pod \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " Oct 30 16:51:51.917413 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.917408 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-envoy\") pod \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " Oct 30 16:51:51.917570 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.917467 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istiod-ca-cert\") pod \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " Oct 30 16:51:51.917570 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.917554 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-workload-certs\") pod \"cb03379b-4139-4f06-a67b-73249017ccb8\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " Oct 30 16:51:51.917687 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.917611 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-data\") pod \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\" (UID: \"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1\") " Oct 30 16:51:51.917743 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.917717 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb03379b-4139-4f06-a67b-73249017ccb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb03379b-4139-4f06-a67b-73249017ccb8-istio-podinfo\") pod \"cb03379b-4139-4f06-a67b-73249017ccb8\" (UID: \"cb03379b-4139-4f06-a67b-73249017ccb8\") " Oct 30 16:51:51.917879 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.917853 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-workload-socket" pod "cb03379b-4139-4f06-a67b-73249017ccb8" (UID: "cb03379b-4139-4f06-a67b-73249017ccb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:51.918083 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.918064 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-workload-certs" pod "182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" (UID: "182b9b7a-fdf7-4150-8ebe-20701bc4a4d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:51.918285 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.918267 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-workload-socket" pod "182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" (UID: "182b9b7a-fdf7-4150-8ebe-20701bc4a4d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:51.919293 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.918872 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-credential-socket" pod "cb03379b-4139-4f06-a67b-73249017ccb8" (UID: "cb03379b-4139-4f06-a67b-73249017ccb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:51.920095 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.920068 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-kube-api-access-q9gdf" pod "182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" (UID: "182b9b7a-fdf7-4150-8ebe-20701bc4a4d1"). InnerVolumeSpecName "kube-api-access-q9gdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:51.920734 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.920264 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-workload-certs" pod "cb03379b-4139-4f06-a67b-73249017ccb8" (UID: "cb03379b-4139-4f06-a67b-73249017ccb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:51.920734 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.920573 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istiod-ca-cert" pod "182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" (UID: "182b9b7a-fdf7-4150-8ebe-20701bc4a4d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:51.920899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.920810 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-credential-socket" pod "182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" (UID: "182b9b7a-fdf7-4150-8ebe-20701bc4a4d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:51.921049 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.921027 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-data" pod "182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" (UID: "182b9b7a-fdf7-4150-8ebe-20701bc4a4d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:51.921604 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.921579 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb03379b-4139-4f06-a67b-73249017ccb8-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:51.922051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.921621 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:51.922051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.921637 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:51.922051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.921653 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:51.922051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.921666 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:51.922051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.921682 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90175c6-171e-41b4-9fe7-3370494ef31d-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:51.922051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.921698 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78l6g\" (UniqueName: \"kubernetes.io/projected/d90175c6-171e-41b4-9fe7-3370494ef31d-kube-api-access-78l6g\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:51.922051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.921713 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90175c6-171e-41b4-9fe7-3370494ef31d-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:51.924675 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.924648 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-envoy" pod "182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" (UID: "182b9b7a-fdf7-4150-8ebe-20701bc4a4d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:51.924946 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.924922 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb03379b-4139-4f06-a67b-73249017ccb8-kube-api-access-g4fws" pod "cb03379b-4139-4f06-a67b-73249017ccb8" (UID: "cb03379b-4139-4f06-a67b-73249017ccb8"). InnerVolumeSpecName "kube-api-access-g4fws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:51.926063 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.926027 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-token" pod "182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" (UID: "182b9b7a-fdf7-4150-8ebe-20701bc4a4d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:51.926147 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.926032 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb03379b-4139-4f06-a67b-73249017ccb8-istio-podinfo" pod "cb03379b-4139-4f06-a67b-73249017ccb8" (UID: "cb03379b-4139-4f06-a67b-73249017ccb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:51:51.926147 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.926128 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb03379b-4139-4f06-a67b-73249017ccb8-istio-token" pod "cb03379b-4139-4f06-a67b-73249017ccb8" (UID: "cb03379b-4139-4f06-a67b-73249017ccb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:51.927019 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.926983 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-podinfo" pod "182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" (UID: "182b9b7a-fdf7-4150-8ebe-20701bc4a4d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:51:51.927272 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:51.927246 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-istio-envoy" pod "cb03379b-4139-4f06-a67b-73249017ccb8" (UID: "cb03379b-4139-4f06-a67b-73249017ccb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.010475 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.010445 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:52.014210 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.014134 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:51:52.024114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.023904 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.024114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.023933 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.024114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.023949 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.024114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.023971 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.024114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.023984 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.024114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.023999 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.024114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.024014 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.024114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.024026 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb03379b-4139-4f06-a67b-73249017ccb8-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.024114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.024038 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.024114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.024051 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb03379b-4139-4f06-a67b-73249017ccb8-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.024114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.024064 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9gdf\" (UniqueName: \"kubernetes.io/projected/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1-kube-api-access-q9gdf\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.024114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.024076 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4fws\" (UniqueName: \"kubernetes.io/projected/cb03379b-4139-4f06-a67b-73249017ccb8-kube-api-access-g4fws\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.024114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.024089 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb03379b-4139-4f06-a67b-73249017ccb8-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.131791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.131322 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istiod-ca-cert\") pod \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " Oct 30 16:51:52.131791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.131428 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-envoy\") pod \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " Oct 30 16:51:52.131791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.131460 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-data\") pod \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " Oct 30 16:51:52.131791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.131604 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53877150-e074-4622-afb6-eb0699548ecb-istio-token\" (UniqueName: \"kubernetes.io/projected/53877150-e074-4622-afb6-eb0699548ecb-istio-token\") pod \"53877150-e074-4622-afb6-eb0699548ecb\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " Oct 30 16:51:52.131791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.131732 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-istio-data\") pod \"53877150-e074-4622-afb6-eb0699548ecb\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " Oct 30 16:51:52.131791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.131794 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-workload-certs\") pod \"53877150-e074-4622-afb6-eb0699548ecb\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " Oct 30 16:51:52.132307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.131825 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-kube-api-access-tc4lx\" (UniqueName: \"kubernetes.io/projected/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-kube-api-access-tc4lx\") pod \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " Oct 30 16:51:52.132307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.131870 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-podinfo\") pod \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " Oct 30 16:51:52.132307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.131911 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-credential-socket\") pod \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " Oct 30 16:51:52.132307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.132045 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53877150-e074-4622-afb6-eb0699548ecb-kube-api-access-dn9td\" (UniqueName: \"kubernetes.io/projected/53877150-e074-4622-afb6-eb0699548ecb-kube-api-access-dn9td\") pod \"53877150-e074-4622-afb6-eb0699548ecb\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " Oct 30 16:51:52.132307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.132083 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53877150-e074-4622-afb6-eb0699548ecb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53877150-e074-4622-afb6-eb0699548ecb-istio-podinfo\") pod \"53877150-e074-4622-afb6-eb0699548ecb\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " Oct 30 16:51:52.132307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.132115 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-istio-envoy\") pod \"53877150-e074-4622-afb6-eb0699548ecb\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " Oct 30 16:51:52.132307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.132144 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-token\" (UniqueName: \"kubernetes.io/projected/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-token\") pod \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " Oct 30 16:51:52.132307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.132178 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-workload-certs\") pod \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " Oct 30 16:51:52.132307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.132222 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-credential-socket\") pod \"53877150-e074-4622-afb6-eb0699548ecb\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " Oct 30 16:51:52.132307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.132263 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53877150-e074-4622-afb6-eb0699548ecb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53877150-e074-4622-afb6-eb0699548ecb-istiod-ca-cert\") pod \"53877150-e074-4622-afb6-eb0699548ecb\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " Oct 30 16:51:52.132307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.132294 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-workload-socket\") pod \"53877150-e074-4622-afb6-eb0699548ecb\" (UID: \"53877150-e074-4622-afb6-eb0699548ecb\") " Oct 30 16:51:52.132940 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.132338 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-workload-socket\") pod \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\" (UID: \"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668\") " Oct 30 16:51:52.283745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.191918 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-workload-socket" pod "75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" (UID: "75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.283745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.193096 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-credential-socket" pod "75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" (UID: "75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.283745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.193940 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istiod-ca-cert" pod "75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" (UID: "75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:52.283745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.224873 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-istio-data" pod "53877150-e074-4622-afb6-eb0699548ecb" (UID: "53877150-e074-4622-afb6-eb0699548ecb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.283745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.225111 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-data" pod "75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" (UID: "75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.283745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.230118 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-workload-certs" pod "53877150-e074-4622-afb6-eb0699548ecb" (UID: "53877150-e074-4622-afb6-eb0699548ecb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.283745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.249400 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.283745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.249435 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.283745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.249449 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.283745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.249462 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.283745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.249474 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.283745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.249487 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.283745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.274226 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-token" pod "75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" (UID: "75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:52.283745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.274671 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53877150-e074-4622-afb6-eb0699548ecb-istiod-ca-cert" pod "53877150-e074-4622-afb6-eb0699548ecb" (UID: "53877150-e074-4622-afb6-eb0699548ecb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:52.283745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.274990 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-workload-socket" pod "53877150-e074-4622-afb6-eb0699548ecb" (UID: "53877150-e074-4622-afb6-eb0699548ecb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.283745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.275189 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-credential-socket" pod "53877150-e074-4622-afb6-eb0699548ecb" (UID: "53877150-e074-4622-afb6-eb0699548ecb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.288741 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.284854 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-workload-certs" pod "75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" (UID: "75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.332110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.325201 6202 generic.go:356] "Generic (PLEG): container finished" podID="75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" containerID="e81e41ad3c9d14fb52a942d8e0aa082a104c9627036a03050156227379e40b66" exitCode=0 Oct 30 16:51:52.332110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.325318 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" event={"ID":"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668","Type":"ContainerDied","Data":"e81e41ad3c9d14fb52a942d8e0aa082a104c9627036a03050156227379e40b66"} Oct 30 16:51:52.332110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.325372 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" event={"ID":"75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668","Type":"ContainerDied","Data":"8e5984f943b381393ba32f4fa0c8aa1f5a8f6dce0548554b0757492f2ee8d3ce"} Oct 30 16:51:52.332110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.325404 6202 scope.go:122] "RemoveContainer" containerID="e81e41ad3c9d14fb52a942d8e0aa082a104c9627036a03050156227379e40b66" Oct 30 16:51:52.332110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.325547 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw" Oct 30 16:51:52.332110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.331644 6202 generic.go:356] "Generic (PLEG): container finished" podID="cb03379b-4139-4f06-a67b-73249017ccb8" containerID="9df63ede14499c496f3683a2c91c76a54a3a618cbdd878a58ae2b1c585ba29c8" exitCode=0 Oct 30 16:51:52.332110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.331827 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" event={"ID":"cb03379b-4139-4f06-a67b-73249017ccb8","Type":"ContainerDied","Data":"9df63ede14499c496f3683a2c91c76a54a3a618cbdd878a58ae2b1c585ba29c8"} Oct 30 16:51:52.332110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.331905 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" event={"ID":"cb03379b-4139-4f06-a67b-73249017ccb8","Type":"ContainerDied","Data":"923122bd868eb831cd21d68c804da1eac1b4d3f33307d342283ff954c8c65b0d"} Oct 30 16:51:52.332110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.332049 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2" Oct 30 16:51:52.342415 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.342383 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-envoy" pod "75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" (UID: "75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.344553 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.344523 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg" event={"ID":"d90175c6-171e-41b4-9fe7-3370494ef31d","Type":"ContainerDied","Data":"c1a27018dd6f5e63512d80d0e35cc9df943d7a2eb389592da56760a0f5dd0227"} Oct 30 16:51:52.344943 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.344915 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1a27018dd6f5e63512d80d0e35cc9df943d7a2eb389592da56760a0f5dd0227" Oct 30 16:51:52.345100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.344835 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg" Oct 30 16:51:52.345309 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.344706 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53877150-e074-4622-afb6-eb0699548ecb-kube-api-access-dn9td" pod "53877150-e074-4622-afb6-eb0699548ecb" (UID: "53877150-e074-4622-afb6-eb0699548ecb"). InnerVolumeSpecName "kube-api-access-dn9td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:52.346590 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.346289 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-istio-envoy" pod "53877150-e074-4622-afb6-eb0699548ecb" (UID: "53877150-e074-4622-afb6-eb0699548ecb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.347980 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.347301 6202 generic.go:356] "Generic (PLEG): container finished" podID="53877150-e074-4622-afb6-eb0699548ecb" containerID="e372756145a9b4243d25c1b2a64eb92d630e8f9bf3706ffa27e7d9a9560e9ef5" exitCode=0 Oct 30 16:51:52.347980 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.347388 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" event={"ID":"53877150-e074-4622-afb6-eb0699548ecb","Type":"ContainerDied","Data":"e372756145a9b4243d25c1b2a64eb92d630e8f9bf3706ffa27e7d9a9560e9ef5"} Oct 30 16:51:52.347980 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.347429 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" event={"ID":"53877150-e074-4622-afb6-eb0699548ecb","Type":"ContainerDied","Data":"604831d265cfc2fd712e5e4d3d4d1aa18df5d7af2598284cb00f1cbc90d0ade2"} Oct 30 16:51:52.348445 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.347508 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr" Oct 30 16:51:52.350180 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.350158 6202 generic.go:356] "Generic (PLEG): container finished" podID="182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" containerID="d052b3803f05d78f678ae6f22e029fefe881aa8ef5643bc8f4b7d2ec7a9f4bf1" exitCode=0 Oct 30 16:51:52.350816 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.350797 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" Oct 30 16:51:52.353556 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.352809 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" event={"ID":"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1","Type":"ContainerDied","Data":"d052b3803f05d78f678ae6f22e029fefe881aa8ef5643bc8f4b7d2ec7a9f4bf1"} Oct 30 16:51:52.353556 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.352864 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq" event={"ID":"182b9b7a-fdf7-4150-8ebe-20701bc4a4d1","Type":"ContainerDied","Data":"756eecef546b34eb7260bfe44bfad862daab6e474e1b8d0ab2e1a42257e44859"} Oct 30 16:51:52.358241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.355151 6202 scope.go:122] "RemoveContainer" containerID="e81e41ad3c9d14fb52a942d8e0aa082a104c9627036a03050156227379e40b66" Oct 30 16:51:52.358241 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:51:52.356617 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e81e41ad3c9d14fb52a942d8e0aa082a104c9627036a03050156227379e40b66\": container with ID starting with e81e41ad3c9d14fb52a942d8e0aa082a104c9627036a03050156227379e40b66 not found: ID does not exist" containerID="e81e41ad3c9d14fb52a942d8e0aa082a104c9627036a03050156227379e40b66" Oct 30 16:51:52.358241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.356649 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e81e41ad3c9d14fb52a942d8e0aa082a104c9627036a03050156227379e40b66"} err="failed to get container status \"e81e41ad3c9d14fb52a942d8e0aa082a104c9627036a03050156227379e40b66\": rpc error: code = NotFound desc = could not find container \"e81e41ad3c9d14fb52a942d8e0aa082a104c9627036a03050156227379e40b66\": container with ID starting with e81e41ad3c9d14fb52a942d8e0aa082a104c9627036a03050156227379e40b66 not found: ID does not exist" Oct 30 16:51:52.358241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.356672 6202 scope.go:122] "RemoveContainer" containerID="9df63ede14499c496f3683a2c91c76a54a3a618cbdd878a58ae2b1c585ba29c8" Oct 30 16:51:52.358241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.357037 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn9td\" (UniqueName: \"kubernetes.io/projected/53877150-e074-4622-afb6-eb0699548ecb-kube-api-access-dn9td\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.358241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.357055 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.358241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.357067 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.358241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.357075 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.358241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.357083 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.358241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.357093 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53877150-e074-4622-afb6-eb0699548ecb-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.358241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.357101 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53877150-e074-4622-afb6-eb0699548ecb-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.358241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.357108 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.360516 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.360306 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53877150-e074-4622-afb6-eb0699548ecb-istio-podinfo" pod "53877150-e074-4622-afb6-eb0699548ecb" (UID: "53877150-e074-4622-afb6-eb0699548ecb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:51:52.361162 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.361138 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-kube-api-access-tc4lx" pod "75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" (UID: "75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668"). InnerVolumeSpecName "kube-api-access-tc4lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:52.361478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.361453 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53877150-e074-4622-afb6-eb0699548ecb-istio-token" pod "53877150-e074-4622-afb6-eb0699548ecb" (UID: "53877150-e074-4622-afb6-eb0699548ecb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:52.364714 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.364680 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-podinfo" pod "75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" (UID: "75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:51:52.398390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.398351 6202 scope.go:122] "RemoveContainer" containerID="8e9e9bdaa8de63e518d25dcfcf0cb2cfe66445e6a3ea6b22fc0c517903e46b4e" Oct 30 16:51:52.411899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.411878 6202 scope.go:122] "RemoveContainer" containerID="9df63ede14499c496f3683a2c91c76a54a3a618cbdd878a58ae2b1c585ba29c8" Oct 30 16:51:52.412217 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:51:52.412193 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df63ede14499c496f3683a2c91c76a54a3a618cbdd878a58ae2b1c585ba29c8\": container with ID starting with 9df63ede14499c496f3683a2c91c76a54a3a618cbdd878a58ae2b1c585ba29c8 not found: ID does not exist" containerID="9df63ede14499c496f3683a2c91c76a54a3a618cbdd878a58ae2b1c585ba29c8" Oct 30 16:51:52.412321 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.412224 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df63ede14499c496f3683a2c91c76a54a3a618cbdd878a58ae2b1c585ba29c8"} err="failed to get container status \"9df63ede14499c496f3683a2c91c76a54a3a618cbdd878a58ae2b1c585ba29c8\": rpc error: code = NotFound desc = could not find container \"9df63ede14499c496f3683a2c91c76a54a3a618cbdd878a58ae2b1c585ba29c8\": container with ID starting with 9df63ede14499c496f3683a2c91c76a54a3a618cbdd878a58ae2b1c585ba29c8 not found: ID does not exist" Oct 30 16:51:52.412321 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.412245 6202 scope.go:122] "RemoveContainer" containerID="8e9e9bdaa8de63e518d25dcfcf0cb2cfe66445e6a3ea6b22fc0c517903e46b4e" Oct 30 16:51:52.412514 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:51:52.412497 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e9e9bdaa8de63e518d25dcfcf0cb2cfe66445e6a3ea6b22fc0c517903e46b4e\": container with ID starting with 8e9e9bdaa8de63e518d25dcfcf0cb2cfe66445e6a3ea6b22fc0c517903e46b4e not found: ID does not exist" containerID="8e9e9bdaa8de63e518d25dcfcf0cb2cfe66445e6a3ea6b22fc0c517903e46b4e" Oct 30 16:51:52.412584 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.412521 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e9e9bdaa8de63e518d25dcfcf0cb2cfe66445e6a3ea6b22fc0c517903e46b4e"} err="failed to get container status \"8e9e9bdaa8de63e518d25dcfcf0cb2cfe66445e6a3ea6b22fc0c517903e46b4e\": rpc error: code = NotFound desc = could not find container \"8e9e9bdaa8de63e518d25dcfcf0cb2cfe66445e6a3ea6b22fc0c517903e46b4e\": container with ID starting with 8e9e9bdaa8de63e518d25dcfcf0cb2cfe66445e6a3ea6b22fc0c517903e46b4e not found: ID does not exist" Oct 30 16:51:52.412584 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.412537 6202 scope.go:122] "RemoveContainer" containerID="e372756145a9b4243d25c1b2a64eb92d630e8f9bf3706ffa27e7d9a9560e9ef5" Oct 30 16:51:52.425377 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.425356 6202 scope.go:122] "RemoveContainer" containerID="e372756145a9b4243d25c1b2a64eb92d630e8f9bf3706ffa27e7d9a9560e9ef5" Oct 30 16:51:52.425707 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:51:52.425685 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e372756145a9b4243d25c1b2a64eb92d630e8f9bf3706ffa27e7d9a9560e9ef5\": container with ID starting with e372756145a9b4243d25c1b2a64eb92d630e8f9bf3706ffa27e7d9a9560e9ef5 not found: ID does not exist" containerID="e372756145a9b4243d25c1b2a64eb92d630e8f9bf3706ffa27e7d9a9560e9ef5" Oct 30 16:51:52.425806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.425718 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e372756145a9b4243d25c1b2a64eb92d630e8f9bf3706ffa27e7d9a9560e9ef5"} err="failed to get container status \"e372756145a9b4243d25c1b2a64eb92d630e8f9bf3706ffa27e7d9a9560e9ef5\": rpc error: code = NotFound desc = could not find container \"e372756145a9b4243d25c1b2a64eb92d630e8f9bf3706ffa27e7d9a9560e9ef5\": container with ID starting with e372756145a9b4243d25c1b2a64eb92d630e8f9bf3706ffa27e7d9a9560e9ef5 not found: ID does not exist" Oct 30 16:51:52.425806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.425744 6202 scope.go:122] "RemoveContainer" containerID="d052b3803f05d78f678ae6f22e029fefe881aa8ef5643bc8f4b7d2ec7a9f4bf1" Oct 30 16:51:52.433798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.433721 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2"] Oct 30 16:51:52.438541 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.438515 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-tnzn2"] Oct 30 16:51:52.459390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.459361 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53877150-e074-4622-afb6-eb0699548ecb-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.459556 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.459394 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53877150-e074-4622-afb6-eb0699548ecb-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.459556 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.459407 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc4lx\" (UniqueName: \"kubernetes.io/projected/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-kube-api-access-tc4lx\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.459556 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.459423 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.475548 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.475382 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq"] Oct 30 16:51:52.479619 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.479594 6202 scope.go:122] "RemoveContainer" containerID="d052b3803f05d78f678ae6f22e029fefe881aa8ef5643bc8f4b7d2ec7a9f4bf1" Oct 30 16:51:52.480018 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:51:52.479996 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d052b3803f05d78f678ae6f22e029fefe881aa8ef5643bc8f4b7d2ec7a9f4bf1\": container with ID starting with d052b3803f05d78f678ae6f22e029fefe881aa8ef5643bc8f4b7d2ec7a9f4bf1 not found: ID does not exist" containerID="d052b3803f05d78f678ae6f22e029fefe881aa8ef5643bc8f4b7d2ec7a9f4bf1" Oct 30 16:51:52.480104 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.480028 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d052b3803f05d78f678ae6f22e029fefe881aa8ef5643bc8f4b7d2ec7a9f4bf1"} err="failed to get container status \"d052b3803f05d78f678ae6f22e029fefe881aa8ef5643bc8f4b7d2ec7a9f4bf1\": rpc error: code = NotFound desc = could not find container \"d052b3803f05d78f678ae6f22e029fefe881aa8ef5643bc8f4b7d2ec7a9f4bf1\": container with ID starting with d052b3803f05d78f678ae6f22e029fefe881aa8ef5643bc8f4b7d2ec7a9f4bf1 not found: ID does not exist" Oct 30 16:51:52.480609 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.480589 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-f8glq"] Oct 30 16:51:52.667560 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.667468 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw"] Oct 30 16:51:52.671353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.671325 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-8hvsw"] Oct 30 16:51:52.705030 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.705001 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr"] Oct 30 16:51:52.709889 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:52.709864 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-4v6xr"] Oct 30 16:51:53.002446 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:53.002370 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:53.808920 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:53.808886 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:53.922927 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:53.922891 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:54.174635 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:54.174596 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:54.297592 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:54.297549 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:54.412391 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:54.412356 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" path="/var/lib/kubelet/pods/182b9b7a-fdf7-4150-8ebe-20701bc4a4d1/volumes" Oct 30 16:51:54.412895 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:54.412881 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53877150-e074-4622-afb6-eb0699548ecb" path="/var/lib/kubelet/pods/53877150-e074-4622-afb6-eb0699548ecb/volumes" Oct 30 16:51:54.413319 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:54.413307 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" path="/var/lib/kubelet/pods/75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668/volumes" Oct 30 16:51:54.414085 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:54.414069 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb03379b-4139-4f06-a67b-73249017ccb8" path="/var/lib/kubelet/pods/cb03379b-4139-4f06-a67b-73249017ccb8/volumes" Oct 30 16:51:54.543941 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:54.543846 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:54.660677 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:54.660640 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:56.193066 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:56.192918 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:56.316542 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:56.314022 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md" podUID="6215914d-d1a2-45e0-a811-469d418061df" containerName="payload-processing" containerID="cri-o://6fed0e73f163a4ffd9a0a69432e797e7e0de53724fed5f533e91d1d993e914c6" gracePeriod=25 Oct 30 16:51:56.348470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:56.348418 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz" podUID="efc9c129-4290-4f6b-b02f-e59979426e5a" containerName="payload-pre-processing" containerID="cri-o://519916685c6cb622a42d1901cc2805ecc39ec51f878ba6242bbb35e739cc973e" gracePeriod=25 Oct 30 16:51:56.700944 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:56.700894 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md" Oct 30 16:51:56.776326 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:56.776296 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz" Oct 30 16:51:56.838670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:56.838636 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6215914d-d1a2-45e0-a811-469d418061df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6215914d-d1a2-45e0-a811-469d418061df-plugins-config-volume\") pod \"6215914d-d1a2-45e0-a811-469d418061df\" (UID: \"6215914d-d1a2-45e0-a811-469d418061df\") " Oct 30 16:51:56.839072 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:56.838715 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6215914d-d1a2-45e0-a811-469d418061df-kube-api-access-vqz8q\" (UniqueName: \"kubernetes.io/projected/6215914d-d1a2-45e0-a811-469d418061df-kube-api-access-vqz8q\") pod \"6215914d-d1a2-45e0-a811-469d418061df\" (UID: \"6215914d-d1a2-45e0-a811-469d418061df\") " Oct 30 16:51:56.839992 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:56.839968 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6215914d-d1a2-45e0-a811-469d418061df-plugins-config-volume" pod "6215914d-d1a2-45e0-a811-469d418061df" (UID: "6215914d-d1a2-45e0-a811-469d418061df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:56.842663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:56.842639 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6215914d-d1a2-45e0-a811-469d418061df-kube-api-access-vqz8q" pod "6215914d-d1a2-45e0-a811-469d418061df" (UID: "6215914d-d1a2-45e0-a811-469d418061df"). InnerVolumeSpecName "kube-api-access-vqz8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:56.940472 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:56.940433 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efc9c129-4290-4f6b-b02f-e59979426e5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc9c129-4290-4f6b-b02f-e59979426e5a-plugins-config-volume\") pod \"efc9c129-4290-4f6b-b02f-e59979426e5a\" (UID: \"efc9c129-4290-4f6b-b02f-e59979426e5a\") " Oct 30 16:51:56.940671 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:56.940549 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc9c129-4290-4f6b-b02f-e59979426e5a-kube-api-access-qfm94\" (UniqueName: \"kubernetes.io/projected/efc9c129-4290-4f6b-b02f-e59979426e5a-kube-api-access-qfm94\") pod \"efc9c129-4290-4f6b-b02f-e59979426e5a\" (UID: \"efc9c129-4290-4f6b-b02f-e59979426e5a\") " Oct 30 16:51:56.940866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:56.940838 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc9c129-4290-4f6b-b02f-e59979426e5a-plugins-config-volume" pod "efc9c129-4290-4f6b-b02f-e59979426e5a" (UID: "efc9c129-4290-4f6b-b02f-e59979426e5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:56.941310 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:56.941288 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc9c129-4290-4f6b-b02f-e59979426e5a-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:56.941388 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:56.941314 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6215914d-d1a2-45e0-a811-469d418061df-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:56.941388 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:56.941330 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqz8q\" (UniqueName: \"kubernetes.io/projected/6215914d-d1a2-45e0-a811-469d418061df-kube-api-access-vqz8q\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:56.943196 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:56.943168 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc9c129-4290-4f6b-b02f-e59979426e5a-kube-api-access-qfm94" pod "efc9c129-4290-4f6b-b02f-e59979426e5a" (UID: "efc9c129-4290-4f6b-b02f-e59979426e5a"). InnerVolumeSpecName "kube-api-access-qfm94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:57.043122 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.043029 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfm94\" (UniqueName: \"kubernetes.io/projected/efc9c129-4290-4f6b-b02f-e59979426e5a-kube-api-access-qfm94\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:51:57.409729 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.409691 6202 generic.go:356] "Generic (PLEG): container finished" podID="efc9c129-4290-4f6b-b02f-e59979426e5a" containerID="519916685c6cb622a42d1901cc2805ecc39ec51f878ba6242bbb35e739cc973e" exitCode=0 Oct 30 16:51:57.410279 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.409749 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz" Oct 30 16:51:57.410279 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.409808 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz" event={"ID":"efc9c129-4290-4f6b-b02f-e59979426e5a","Type":"ContainerDied","Data":"519916685c6cb622a42d1901cc2805ecc39ec51f878ba6242bbb35e739cc973e"} Oct 30 16:51:57.410279 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.409861 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz" event={"ID":"efc9c129-4290-4f6b-b02f-e59979426e5a","Type":"ContainerDied","Data":"c3f38daa6ebf91e07ff9da72968eb99611d964bb01f63af9c26dce52ab7cdb9d"} Oct 30 16:51:57.410279 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.409892 6202 scope.go:122] "RemoveContainer" containerID="519916685c6cb622a42d1901cc2805ecc39ec51f878ba6242bbb35e739cc973e" Oct 30 16:51:57.411743 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.411722 6202 generic.go:356] "Generic (PLEG): container finished" podID="6215914d-d1a2-45e0-a811-469d418061df" containerID="6fed0e73f163a4ffd9a0a69432e797e7e0de53724fed5f533e91d1d993e914c6" exitCode=0 Oct 30 16:51:57.411887 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.411820 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md" event={"ID":"6215914d-d1a2-45e0-a811-469d418061df","Type":"ContainerDied","Data":"6fed0e73f163a4ffd9a0a69432e797e7e0de53724fed5f533e91d1d993e914c6"} Oct 30 16:51:57.411887 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.411861 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md" event={"ID":"6215914d-d1a2-45e0-a811-469d418061df","Type":"ContainerDied","Data":"c7c75dba8d28a062ddb5ac9ee6752f8c962d39030c3fb136004539a4071f41ad"} Oct 30 16:51:57.412014 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.411917 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md" Oct 30 16:51:57.422193 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.422175 6202 scope.go:122] "RemoveContainer" containerID="519916685c6cb622a42d1901cc2805ecc39ec51f878ba6242bbb35e739cc973e" Oct 30 16:51:57.422504 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:51:57.422481 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519916685c6cb622a42d1901cc2805ecc39ec51f878ba6242bbb35e739cc973e\": container with ID starting with 519916685c6cb622a42d1901cc2805ecc39ec51f878ba6242bbb35e739cc973e not found: ID does not exist" containerID="519916685c6cb622a42d1901cc2805ecc39ec51f878ba6242bbb35e739cc973e" Oct 30 16:51:57.422590 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.422518 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519916685c6cb622a42d1901cc2805ecc39ec51f878ba6242bbb35e739cc973e"} err="failed to get container status \"519916685c6cb622a42d1901cc2805ecc39ec51f878ba6242bbb35e739cc973e\": rpc error: code = NotFound desc = could not find container \"519916685c6cb622a42d1901cc2805ecc39ec51f878ba6242bbb35e739cc973e\": container with ID starting with 519916685c6cb622a42d1901cc2805ecc39ec51f878ba6242bbb35e739cc973e not found: ID does not exist" Oct 30 16:51:57.422590 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.422542 6202 scope.go:122] "RemoveContainer" containerID="6fed0e73f163a4ffd9a0a69432e797e7e0de53724fed5f533e91d1d993e914c6" Oct 30 16:51:57.432848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.432830 6202 scope.go:122] "RemoveContainer" containerID="6fed0e73f163a4ffd9a0a69432e797e7e0de53724fed5f533e91d1d993e914c6" Oct 30 16:51:57.433148 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:51:57.433119 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fed0e73f163a4ffd9a0a69432e797e7e0de53724fed5f533e91d1d993e914c6\": container with ID starting with 6fed0e73f163a4ffd9a0a69432e797e7e0de53724fed5f533e91d1d993e914c6 not found: ID does not exist" containerID="6fed0e73f163a4ffd9a0a69432e797e7e0de53724fed5f533e91d1d993e914c6" Oct 30 16:51:57.433236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.433148 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fed0e73f163a4ffd9a0a69432e797e7e0de53724fed5f533e91d1d993e914c6"} err="failed to get container status \"6fed0e73f163a4ffd9a0a69432e797e7e0de53724fed5f533e91d1d993e914c6\": rpc error: code = NotFound desc = could not find container \"6fed0e73f163a4ffd9a0a69432e797e7e0de53724fed5f533e91d1d993e914c6\": container with ID starting with 6fed0e73f163a4ffd9a0a69432e797e7e0de53724fed5f533e91d1d993e914c6 not found: ID does not exist" Oct 30 16:51:57.469949 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.469915 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz"] Oct 30 16:51:57.481801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.481753 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-407eab-7d4cbcb787-5fzsz"] Oct 30 16:51:57.515659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.515628 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md"] Oct 30 16:51:57.536331 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:57.536298 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-407eab-66f8cb4786-g92md"] Oct 30 16:51:58.082407 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:58.082377 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:58.248121 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:58.248087 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:58.403967 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:58.403843 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:58.423787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:58.423722 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6215914d-d1a2-45e0-a811-469d418061df" path="/var/lib/kubelet/pods/6215914d-d1a2-45e0-a811-469d418061df/volumes" Oct 30 16:51:58.425446 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:58.425420 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efc9c129-4290-4f6b-b02f-e59979426e5a" path="/var/lib/kubelet/pods/efc9c129-4290-4f6b-b02f-e59979426e5a/volumes" Oct 30 16:51:58.820122 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:58.820033 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:59.290285 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.290163 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq"] Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291325 6202 cpu_manager.go:414] "containerMap: removing container" podUID="6215914d-d1a2-45e0-a811-469d418061df" containerName="payload-processing" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291351 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6215914d-d1a2-45e0-a811-469d418061df" containerName="payload-processing" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291370 6202 cpu_manager.go:414] "containerMap: removing container" podUID="cb03379b-4139-4f06-a67b-73249017ccb8" containerName="istio-proxy" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291378 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb03379b-4139-4f06-a67b-73249017ccb8" containerName="istio-proxy" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291395 6202 cpu_manager.go:414] "containerMap: removing container" podUID="efc9c129-4290-4f6b-b02f-e59979426e5a" containerName="payload-pre-processing" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291402 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc9c129-4290-4f6b-b02f-e59979426e5a" containerName="payload-pre-processing" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291424 6202 cpu_manager.go:414] "containerMap: removing container" podUID="d90175c6-171e-41b4-9fe7-3370494ef31d" containerName="revoke-keys" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291430 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90175c6-171e-41b4-9fe7-3370494ef31d" containerName="revoke-keys" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291456 6202 cpu_manager.go:414] "containerMap: removing container" podUID="cb03379b-4139-4f06-a67b-73249017ccb8" containerName="istio-proxy" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291463 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb03379b-4139-4f06-a67b-73249017ccb8" containerName="istio-proxy" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291478 6202 cpu_manager.go:414] "containerMap: removing container" podUID="75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" containerName="istio-proxy" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291486 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" containerName="istio-proxy" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291517 6202 cpu_manager.go:414] "containerMap: removing container" podUID="182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" containerName="istio-proxy" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291526 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" containerName="istio-proxy" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291538 6202 cpu_manager.go:414] "containerMap: removing container" podUID="53877150-e074-4622-afb6-eb0699548ecb" containerName="istio-proxy" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291545 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53877150-e074-4622-afb6-eb0699548ecb" containerName="istio-proxy" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291702 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="d90175c6-171e-41b4-9fe7-3370494ef31d" containerName="revoke-keys" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291717 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb03379b-4139-4f06-a67b-73249017ccb8" containerName="istio-proxy" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291728 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="75c4b2e9-0686-4ccd-9a97-cc6c8cbfb668" containerName="istio-proxy" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291741 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="182b9b7a-fdf7-4150-8ebe-20701bc4a4d1" containerName="istio-proxy" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291751 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="53877150-e074-4622-afb6-eb0699548ecb" containerName="istio-proxy" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291785 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="6215914d-d1a2-45e0-a811-469d418061df" containerName="payload-processing" Oct 30 16:51:59.291931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.291799 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="efc9c129-4290-4f6b-b02f-e59979426e5a" containerName="payload-pre-processing" Oct 30 16:51:59.298958 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.298176 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.302009 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.301985 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq"] Oct 30 16:51:59.401799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.400733 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-3ed10a-openshift-default-dockercfg-gt5hj\"" Oct 30 16:51:59.614497 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.614449 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:59.637796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.634549 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-workload-socket\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.638130 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.638099 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4hwp\" (UniqueName: \"kubernetes.io/projected/2b6965bb-5f5c-40ca-b098-9f67c7433381-kube-api-access-f4hwp\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.651433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.651396 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-data\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.651839 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.651803 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-token\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.652039 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.652023 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-envoy\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.652219 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.652193 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-workload-certs\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.652366 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.652350 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-podinfo\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.652495 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.652478 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-credential-socket\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.652691 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.652671 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6965bb-5f5c-40ca-b098-9f67c7433381-istiod-ca-cert\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.753792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.749306 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:59.755087 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.755054 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-data\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.755297 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.755277 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-token\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.755450 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.755434 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-envoy\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.755627 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.755611 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-workload-certs\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.755756 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.755742 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-podinfo\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.755921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.755905 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-credential-socket\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.756127 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.756111 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6965bb-5f5c-40ca-b098-9f67c7433381-istiod-ca-cert\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.756292 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.756277 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-workload-socket\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.756445 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.756426 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4hwp\" (UniqueName: \"kubernetes.io/projected/2b6965bb-5f5c-40ca-b098-9f67c7433381-kube-api-access-f4hwp\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.767445 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.756539 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-data\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.767654 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.757978 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-workload-socket\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.767801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.758189 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-workload-certs\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.769531 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.758392 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-credential-socket\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.772272 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.772245 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-envoy\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.772562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.772526 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6965bb-5f5c-40ca-b098-9f67c7433381-istiod-ca-cert\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.774200 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.774177 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-podinfo\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.781013 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.780968 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-token\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.796262 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.796217 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4hwp\" (UniqueName: \"kubernetes.io/projected/2b6965bb-5f5c-40ca-b098-9f67c7433381-kube-api-access-f4hwp\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:51:59.917044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:51:59.916969 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:52:00.289231 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.289201 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq"] Oct 30 16:52:00.290681 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:52:00.290653 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6965bb_5f5c_40ca_b098_9f67c7433381.slice/crio-2c2438e43479bc5a8507577f8c3be898f50868106fcc413ed905bd5f149d1d74 WatchSource:0}: Error finding container 2c2438e43479bc5a8507577f8c3be898f50868106fcc413ed905bd5f149d1d74: Status 404 returned error can't find the container with id 2c2438e43479bc5a8507577f8c3be898f50868106fcc413ed905bd5f149d1d74 Oct 30 16:52:00.293611 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.293567 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:52:00.293732 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.293660 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:52:00.293732 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.293705 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:52:00.421387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.420601 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f"] Oct 30 16:52:00.422246 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.422135 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb03379b-4139-4f06-a67b-73249017ccb8" containerName="istio-proxy" Oct 30 16:52:00.444804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.437481 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" Oct 30 16:52:00.468798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.466714 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a59502\"" Oct 30 16:52:00.483004 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.481823 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:00.492044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.489930 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0da4a2d-617f-4b93-8e0a-19db319bee2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f\" (UID: \"b0da4a2d-617f-4b93-8e0a-19db319bee2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" Oct 30 16:52:00.492044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.490114 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fbdh\" (UniqueName: \"kubernetes.io/projected/b0da4a2d-617f-4b93-8e0a-19db319bee2f-kube-api-access-4fbdh\") pod \"payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f\" (UID: \"b0da4a2d-617f-4b93-8e0a-19db319bee2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" Oct 30 16:52:00.492044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.490357 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm"] Oct 30 16:52:00.510989 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.505567 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f"] Oct 30 16:52:00.510989 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.505624 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm"] Oct 30 16:52:00.510989 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.505719 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" Oct 30 16:52:00.641795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.592085 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fbdh\" (UniqueName: \"kubernetes.io/projected/b0da4a2d-617f-4b93-8e0a-19db319bee2f-kube-api-access-4fbdh\") pod \"payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f\" (UID: \"b0da4a2d-617f-4b93-8e0a-19db319bee2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" Oct 30 16:52:00.641795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.592398 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0da4a2d-617f-4b93-8e0a-19db319bee2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f\" (UID: \"b0da4a2d-617f-4b93-8e0a-19db319bee2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" Oct 30 16:52:00.641795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.624876 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a59502-dockercfg-j7htw\"" Oct 30 16:52:00.658097 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.643272 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0da4a2d-617f-4b93-8e0a-19db319bee2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f\" (UID: \"b0da4a2d-617f-4b93-8e0a-19db319bee2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" Oct 30 16:52:00.658932 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.658900 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:00.720799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.717886 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s66q5\" (UniqueName: \"kubernetes.io/projected/940b403f-6a30-4c02-bae8-4215fc9cd643-kube-api-access-s66q5\") pod \"payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm\" (UID: \"940b403f-6a30-4c02-bae8-4215fc9cd643\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" Oct 30 16:52:00.720799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.718214 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940b403f-6a30-4c02-bae8-4215fc9cd643-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm\" (UID: \"940b403f-6a30-4c02-bae8-4215fc9cd643\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" Oct 30 16:52:00.827820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.823820 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940b403f-6a30-4c02-bae8-4215fc9cd643-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm\" (UID: \"940b403f-6a30-4c02-bae8-4215fc9cd643\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" Oct 30 16:52:00.827820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.824021 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s66q5\" (UniqueName: \"kubernetes.io/projected/940b403f-6a30-4c02-bae8-4215fc9cd643-kube-api-access-s66q5\") pod \"payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm\" (UID: \"940b403f-6a30-4c02-bae8-4215fc9cd643\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" Oct 30 16:52:00.849815 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.829825 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940b403f-6a30-4c02-bae8-4215fc9cd643-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm\" (UID: \"940b403f-6a30-4c02-bae8-4215fc9cd643\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" Oct 30 16:52:00.879795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.875528 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:00.880079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.878025 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fbdh\" (UniqueName: \"kubernetes.io/projected/b0da4a2d-617f-4b93-8e0a-19db319bee2f-kube-api-access-4fbdh\") pod \"payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f\" (UID: \"b0da4a2d-617f-4b93-8e0a-19db319bee2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" Oct 30 16:52:00.901207 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.888442 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s66q5\" (UniqueName: \"kubernetes.io/projected/940b403f-6a30-4c02-bae8-4215fc9cd643-kube-api-access-s66q5\") pod \"payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm\" (UID: \"940b403f-6a30-4c02-bae8-4215fc9cd643\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" Oct 30 16:52:00.901538 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.901357 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" event={"ID":"2b6965bb-5f5c-40ca-b098-9f67c7433381","Type":"ContainerStarted","Data":"b59da60ecc764d40930bc401e680c533f6c1a12b9b5f935c27159b26ad047367"} Oct 30 16:52:00.901538 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.901435 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" event={"ID":"2b6965bb-5f5c-40ca-b098-9f67c7433381","Type":"ContainerStarted","Data":"2c2438e43479bc5a8507577f8c3be898f50868106fcc413ed905bd5f149d1d74"} Oct 30 16:52:00.920977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.920911 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:52:00.974566 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:00.974501 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" podStartSLOduration=1.9744796949999999 podStartE2EDuration="1.974479695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:00.957556927 +0000 UTC m=+1834.668011673" watchObservedRunningTime="2026-10-30 16:52:00.974479695 +0000 UTC m=+1834.684934441" Oct 30 16:52:01.078334 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:01.078283 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:01.123578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:01.123544 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" Oct 30 16:52:01.162648 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:01.162558 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" Oct 30 16:52:01.578151 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:01.577700 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f"] Oct 30 16:52:01.599308 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:01.599275 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:01.812332 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:01.812309 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm"] Oct 30 16:52:01.815562 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:52:01.815534 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940b403f_6a30_4c02_bae8_4215fc9cd643.slice/crio-10886bea11482ad2e0d62a1aac240bccd3e7006031f78b5417d011d086a15da7 WatchSource:0}: Error finding container 10886bea11482ad2e0d62a1aac240bccd3e7006031f78b5417d011d086a15da7: Status 404 returned error can't find the container with id 10886bea11482ad2e0d62a1aac240bccd3e7006031f78b5417d011d086a15da7 Oct 30 16:52:02.005853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:01.963093 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:52:02.075712 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.075667 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" event={"ID":"b0da4a2d-617f-4b93-8e0a-19db319bee2f","Type":"ContainerStarted","Data":"fc26a141889022af3f0bac3cad9e62fe26986fd8e1e9693da1830949809c2eee"} Oct 30 16:52:02.078888 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.078851 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" event={"ID":"b0da4a2d-617f-4b93-8e0a-19db319bee2f","Type":"ContainerStarted","Data":"e22e873425ca29c32447610faaba84edd83ca098f20e917a3f2d480c11f81468"} Oct 30 16:52:02.087793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.080218 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" Oct 30 16:52:02.117077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.117040 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" event={"ID":"940b403f-6a30-4c02-bae8-4215fc9cd643","Type":"ContainerStarted","Data":"10886bea11482ad2e0d62a1aac240bccd3e7006031f78b5417d011d086a15da7"} Oct 30 16:52:02.117892 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.117867 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:52:02.125284 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.125242 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8"] Oct 30 16:52:02.167043 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.167013 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:52:02.178800 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.167595 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.196594 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.196570 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9bfcac-openshift-default-dockercfg-7dd52\"" Oct 30 16:52:02.199704 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.199548 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8"] Oct 30 16:52:02.200028 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.199980 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" podStartSLOduration=2.199962357 podStartE2EDuration="2.199962357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:02.135329025 +0000 UTC m=+1835.845783773" watchObservedRunningTime="2026-10-30 16:52:02.199962357 +0000 UTC m=+1835.910417083" Oct 30 16:52:02.205619 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.205470 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:02.304848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.294500 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d729b26-7129-4f64-95d9-860074961e2d-istio-podinfo\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.304848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.294584 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d729b26-7129-4f64-95d9-860074961e2d-istio-token\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.304848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.294679 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-istio-envoy\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.304848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.294728 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-workload-certs\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.304848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.294799 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d729b26-7129-4f64-95d9-860074961e2d-istiod-ca-cert\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.304848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.294909 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqdbn\" (UniqueName: \"kubernetes.io/projected/5d729b26-7129-4f64-95d9-860074961e2d-kube-api-access-lqdbn\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.304848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.294986 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-credential-socket\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.304848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.295018 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-istio-data\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.304848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.295146 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-workload-socket\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.400791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.397880 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-credential-socket\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.400791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.398018 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-istio-data\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.400791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.398157 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-workload-socket\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.400791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.398225 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d729b26-7129-4f64-95d9-860074961e2d-istio-podinfo\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.400791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.398276 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d729b26-7129-4f64-95d9-860074961e2d-istio-token\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.400791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.398365 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-istio-envoy\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.400791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.398415 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-workload-certs\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.400791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.398477 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d729b26-7129-4f64-95d9-860074961e2d-istiod-ca-cert\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.400791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.398569 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqdbn\" (UniqueName: \"kubernetes.io/projected/5d729b26-7129-4f64-95d9-860074961e2d-kube-api-access-lqdbn\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.400791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.399746 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-credential-socket\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.400791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.400031 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-istio-data\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.400791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.400266 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-workload-socket\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.402222 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.402198 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-workload-certs\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.432791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.403251 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d729b26-7129-4f64-95d9-860074961e2d-istiod-ca-cert\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.579846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.539536 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:02.691554 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.691513 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqdbn\" (UniqueName: \"kubernetes.io/projected/5d729b26-7129-4f64-95d9-860074961e2d-kube-api-access-lqdbn\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.692691 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.692664 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d729b26-7129-4f64-95d9-860074961e2d-istio-podinfo\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.694092 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.694052 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d729b26-7129-4f64-95d9-860074961e2d-istio-token\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.694655 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.694630 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-istio-envoy\") pod \"e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:02.774308 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.774206 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:02.816249 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:02.815570 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:03.130542 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:03.130428 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" event={"ID":"940b403f-6a30-4c02-bae8-4215fc9cd643","Type":"ContainerStarted","Data":"191efb322cff1203681d730ec13bb2e295da5c7e66ffffaf16b15cc9ac2efdba"} Oct 30 16:52:03.131320 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:03.131290 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" Oct 30 16:52:03.270867 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:03.262316 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" podStartSLOduration=3.262292635 podStartE2EDuration="3.262292635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:03.178752605 +0000 UTC m=+1836.889207348" watchObservedRunningTime="2026-10-30 16:52:03.262292635 +0000 UTC m=+1836.972747380" Oct 30 16:52:03.276089 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:03.276056 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8"] Oct 30 16:52:03.284720 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:03.284683 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:52:03.284956 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:03.284930 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:52:03.285110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:03.285091 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:52:04.172788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:04.172722 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" event={"ID":"5d729b26-7129-4f64-95d9-860074961e2d","Type":"ContainerStarted","Data":"bfba618ec63d7227ae99120abb61ae940f89d53e880cceb04b3b442fab5705f2"} Oct 30 16:52:04.173738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:04.172828 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" event={"ID":"5d729b26-7129-4f64-95d9-860074961e2d","Type":"ContainerStarted","Data":"976fe0b4665a0752493c68e429bbda9a0321b068d18282d859b5591366e7b34d"} Oct 30 16:52:04.271103 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:04.271037 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" podStartSLOduration=2.271020197 podStartE2EDuration="2.271020197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:04.235922353 +0000 UTC m=+1837.946377097" watchObservedRunningTime="2026-10-30 16:52:04.271020197 +0000 UTC m=+1837.981474941" Oct 30 16:52:04.611609 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:04.611577 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:04.816403 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:04.816365 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:04.823673 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:04.823643 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:05.184281 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:05.184180 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:05.185282 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:05.185262 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:52:05.419340 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:05.419301 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:05.906340 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:05.906303 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:06.133793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:06.127286 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:06.564705 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:06.561044 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z"] Oct 30 16:52:06.564705 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:06.561281 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" podUID="51725959-f0c8-4b0f-bd22-547a9995c437" containerName="istio-proxy" containerID="cri-o://53e4f54a138ade97e8bccc433bc15259dbd591316a44d5d6e906e787fd598a15" gracePeriod=30 Oct 30 16:52:06.833629 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:06.833593 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:06.974286 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:06.974249 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:08.450335 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:08.450299 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:08.571475 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:08.571440 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq"] Oct 30 16:52:08.577847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:08.577821 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" Oct 30 16:52:08.610340 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:08.610308 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq"] Oct 30 16:52:08.612635 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:08.612600 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:08.614807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:08.614784 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9bfcac-dockercfg-n6pwx\"" Oct 30 16:52:08.614807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:08.614783 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9bfcac\"" Oct 30 16:52:08.726677 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:08.716495 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/242e1339-d85b-4360-99c9-6aab93ac561d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq\" (UID: \"242e1339-d85b-4360-99c9-6aab93ac561d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" Oct 30 16:52:08.731793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:08.727292 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp7k2\" (UniqueName: \"kubernetes.io/projected/242e1339-d85b-4360-99c9-6aab93ac561d-kube-api-access-gp7k2\") pod \"payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq\" (UID: \"242e1339-d85b-4360-99c9-6aab93ac561d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" Oct 30 16:52:08.831321 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:08.831285 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:08.836267 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:08.836226 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/242e1339-d85b-4360-99c9-6aab93ac561d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq\" (UID: \"242e1339-d85b-4360-99c9-6aab93ac561d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" Oct 30 16:52:08.837848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:08.837817 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp7k2\" (UniqueName: \"kubernetes.io/projected/242e1339-d85b-4360-99c9-6aab93ac561d-kube-api-access-gp7k2\") pod \"payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq\" (UID: \"242e1339-d85b-4360-99c9-6aab93ac561d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" Oct 30 16:52:08.838464 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:08.837417 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/242e1339-d85b-4360-99c9-6aab93ac561d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq\" (UID: \"242e1339-d85b-4360-99c9-6aab93ac561d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" Oct 30 16:52:08.880966 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:08.880921 6202 patch_prober.go:29] interesting pod/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:52:08.881289 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:08.881242 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" podUID="927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:08.953269 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:08.953233 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp7k2\" (UniqueName: \"kubernetes.io/projected/242e1339-d85b-4360-99c9-6aab93ac561d-kube-api-access-gp7k2\") pod \"payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq\" (UID: \"242e1339-d85b-4360-99c9-6aab93ac561d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" Oct 30 16:52:09.208376 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:09.208342 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" Oct 30 16:52:09.520903 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:09.520871 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq"] Oct 30 16:52:09.521678 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:52:09.521653 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242e1339_d85b_4360_99c9_6aab93ac561d.slice/crio-b85eaced89969bc7815f39c3410ac1dc1b35eaf6b53ebde1e2e232ef91f9b09d WatchSource:0}: Error finding container b85eaced89969bc7815f39c3410ac1dc1b35eaf6b53ebde1e2e232ef91f9b09d: Status 404 returned error can't find the container with id b85eaced89969bc7815f39c3410ac1dc1b35eaf6b53ebde1e2e232ef91f9b09d Oct 30 16:52:09.857945 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:09.857912 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:09.988217 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:09.988119 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:10.292600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:10.292570 6202 generic.go:356] "Generic (PLEG): container finished" podID="927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" containerID="c8c0a858d4907d0a55f065d3af3b8eeef108dd178fe3f6e47f43304fbfbdef15" exitCode=0 Oct 30 16:52:10.292813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:10.292650 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" event={"ID":"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5","Type":"ContainerDied","Data":"c8c0a858d4907d0a55f065d3af3b8eeef108dd178fe3f6e47f43304fbfbdef15"} Oct 30 16:52:10.293205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:10.293173 6202 scope.go:122] "RemoveContainer" containerID="c8c0a858d4907d0a55f065d3af3b8eeef108dd178fe3f6e47f43304fbfbdef15" Oct 30 16:52:10.294432 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:10.294407 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" event={"ID":"242e1339-d85b-4360-99c9-6aab93ac561d","Type":"ContainerStarted","Data":"ea0e811746fe7c4e37a8ccd1628b13deafdcbfbfb97277f978ae6ef29f699b2b"} Oct 30 16:52:10.294536 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:10.294456 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" event={"ID":"242e1339-d85b-4360-99c9-6aab93ac561d","Type":"ContainerStarted","Data":"b85eaced89969bc7815f39c3410ac1dc1b35eaf6b53ebde1e2e232ef91f9b09d"} Oct 30 16:52:10.294536 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:10.294493 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" Oct 30 16:52:10.295161 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:10.295131 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:52:10.295257 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:10.295216 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:52:10.295322 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:10.295265 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:52:10.400930 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:10.400870 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" podStartSLOduration=2.400850146 podStartE2EDuration="2.400850146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:10.394126707 +0000 UTC m=+1844.104581457" watchObservedRunningTime="2026-10-30 16:52:10.400850146 +0000 UTC m=+1844.111304891" Oct 30 16:52:10.930483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:10.930445 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:11.305699 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:11.305615 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" event={"ID":"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5","Type":"ContainerStarted","Data":"9b88b039b2e82009b766fe282e75aa79c4426a72e970bf35e3edf5bea53d2646"} Oct 30 16:52:11.567874 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:11.559191 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:11.617322 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:11.617231 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:52:11.617625 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:11.617601 6202 patch_prober.go:29] interesting pod/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 30 16:52:11.617711 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:11.617658 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" podUID="927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 30 16:52:11.686451 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:11.686368 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:11.689725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:11.689668 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw"] Oct 30 16:52:11.696691 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:11.696652 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw"] Oct 30 16:52:11.697034 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:11.697004 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" Oct 30 16:52:11.715195 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:11.714644 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-17f60f\"" Oct 30 16:52:11.809819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:11.809760 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17152dfe-8124-40b1-b6b9-163d4901d269-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw\" (UID: \"17152dfe-8124-40b1-b6b9-163d4901d269\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" Oct 30 16:52:11.810136 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:11.810117 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rl9m\" (UniqueName: \"kubernetes.io/projected/17152dfe-8124-40b1-b6b9-163d4901d269-kube-api-access-5rl9m\") pod \"payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw\" (UID: \"17152dfe-8124-40b1-b6b9-163d4901d269\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" Oct 30 16:52:11.912954 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:11.912790 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rl9m\" (UniqueName: \"kubernetes.io/projected/17152dfe-8124-40b1-b6b9-163d4901d269-kube-api-access-5rl9m\") pod \"payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw\" (UID: \"17152dfe-8124-40b1-b6b9-163d4901d269\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" Oct 30 16:52:11.912954 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:11.912860 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17152dfe-8124-40b1-b6b9-163d4901d269-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw\" (UID: \"17152dfe-8124-40b1-b6b9-163d4901d269\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" Oct 30 16:52:11.913716 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:11.913690 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17152dfe-8124-40b1-b6b9-163d4901d269-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw\" (UID: \"17152dfe-8124-40b1-b6b9-163d4901d269\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" Oct 30 16:52:12.023845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:11.983897 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rl9m\" (UniqueName: \"kubernetes.io/projected/17152dfe-8124-40b1-b6b9-163d4901d269-kube-api-access-5rl9m\") pod \"payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw\" (UID: \"17152dfe-8124-40b1-b6b9-163d4901d269\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" Oct 30 16:52:12.041812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.033064 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" Oct 30 16:52:12.223923 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.223809 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:52:12.326369 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.326334 6202 generic.go:356] "Generic (PLEG): container finished" podID="51725959-f0c8-4b0f-bd22-547a9995c437" containerID="53e4f54a138ade97e8bccc433bc15259dbd591316a44d5d6e906e787fd598a15" exitCode=0 Oct 30 16:52:12.327458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.327432 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" Oct 30 16:52:12.327828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.327803 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" event={"ID":"51725959-f0c8-4b0f-bd22-547a9995c437","Type":"ContainerDied","Data":"53e4f54a138ade97e8bccc433bc15259dbd591316a44d5d6e906e787fd598a15"} Oct 30 16:52:12.327947 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.327859 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z" event={"ID":"51725959-f0c8-4b0f-bd22-547a9995c437","Type":"ContainerDied","Data":"2043764ce2092a62e0c5d9159af0dc9e90ebb9a0e7151c2cd3c38acd888f7bc9"} Oct 30 16:52:12.327947 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.327889 6202 scope.go:122] "RemoveContainer" containerID="53e4f54a138ade97e8bccc433bc15259dbd591316a44d5d6e906e787fd598a15" Oct 30 16:52:12.396496 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.353946 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-istio-data\") pod \"51725959-f0c8-4b0f-bd22-547a9995c437\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " Oct 30 16:52:12.398145 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.398112 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51725959-f0c8-4b0f-bd22-547a9995c437-kube-api-access-jlvf9\" (UniqueName: \"kubernetes.io/projected/51725959-f0c8-4b0f-bd22-547a9995c437-kube-api-access-jlvf9\") pod \"51725959-f0c8-4b0f-bd22-547a9995c437\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " Oct 30 16:52:12.398298 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.398280 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-credential-socket\") pod \"51725959-f0c8-4b0f-bd22-547a9995c437\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " Oct 30 16:52:12.398367 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.398351 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51725959-f0c8-4b0f-bd22-547a9995c437-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51725959-f0c8-4b0f-bd22-547a9995c437-istiod-ca-cert\") pod \"51725959-f0c8-4b0f-bd22-547a9995c437\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " Oct 30 16:52:12.398512 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.398496 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51725959-f0c8-4b0f-bd22-547a9995c437-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51725959-f0c8-4b0f-bd22-547a9995c437-istio-podinfo\") pod \"51725959-f0c8-4b0f-bd22-547a9995c437\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " Oct 30 16:52:12.398571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.398550 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51725959-f0c8-4b0f-bd22-547a9995c437-istio-token\" (UniqueName: \"kubernetes.io/projected/51725959-f0c8-4b0f-bd22-547a9995c437-istio-token\") pod \"51725959-f0c8-4b0f-bd22-547a9995c437\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " Oct 30 16:52:12.398653 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.398638 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-workload-certs\") pod \"51725959-f0c8-4b0f-bd22-547a9995c437\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " Oct 30 16:52:12.398737 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.398722 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-istio-envoy\") pod \"51725959-f0c8-4b0f-bd22-547a9995c437\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " Oct 30 16:52:12.398889 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.398873 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-workload-socket\") pod \"51725959-f0c8-4b0f-bd22-547a9995c437\" (UID: \"51725959-f0c8-4b0f-bd22-547a9995c437\") " Oct 30 16:52:12.401075 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.401042 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-workload-socket" pod "51725959-f0c8-4b0f-bd22-547a9995c437" (UID: "51725959-f0c8-4b0f-bd22-547a9995c437"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:12.402678 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.401841 6202 scope.go:122] "RemoveContainer" containerID="53e4f54a138ade97e8bccc433bc15259dbd591316a44d5d6e906e787fd598a15" Oct 30 16:52:12.403547 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:52:12.403275 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e4f54a138ade97e8bccc433bc15259dbd591316a44d5d6e906e787fd598a15\": container with ID starting with 53e4f54a138ade97e8bccc433bc15259dbd591316a44d5d6e906e787fd598a15 not found: ID does not exist" containerID="53e4f54a138ade97e8bccc433bc15259dbd591316a44d5d6e906e787fd598a15" Oct 30 16:52:12.403660 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.403565 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-workload-certs" pod "51725959-f0c8-4b0f-bd22-547a9995c437" (UID: "51725959-f0c8-4b0f-bd22-547a9995c437"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:12.403660 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.403562 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e4f54a138ade97e8bccc433bc15259dbd591316a44d5d6e906e787fd598a15"} err="failed to get container status \"53e4f54a138ade97e8bccc433bc15259dbd591316a44d5d6e906e787fd598a15\": rpc error: code = NotFound desc = could not find container \"53e4f54a138ade97e8bccc433bc15259dbd591316a44d5d6e906e787fd598a15\": container with ID starting with 53e4f54a138ade97e8bccc433bc15259dbd591316a44d5d6e906e787fd598a15 not found: ID does not exist" Oct 30 16:52:12.404550 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.404212 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-credential-socket" pod "51725959-f0c8-4b0f-bd22-547a9995c437" (UID: "51725959-f0c8-4b0f-bd22-547a9995c437"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:12.408811 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.404948 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-istio-data" pod "51725959-f0c8-4b0f-bd22-547a9995c437" (UID: "51725959-f0c8-4b0f-bd22-547a9995c437"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:12.408811 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.404855 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51725959-f0c8-4b0f-bd22-547a9995c437-istiod-ca-cert" pod "51725959-f0c8-4b0f-bd22-547a9995c437" (UID: "51725959-f0c8-4b0f-bd22-547a9995c437"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:52:12.408811 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.405153 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51725959-f0c8-4b0f-bd22-547a9995c437-kube-api-access-jlvf9" pod "51725959-f0c8-4b0f-bd22-547a9995c437" (UID: "51725959-f0c8-4b0f-bd22-547a9995c437"). InnerVolumeSpecName "kube-api-access-jlvf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:12.418069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.418033 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51725959-f0c8-4b0f-bd22-547a9995c437-istio-token" pod "51725959-f0c8-4b0f-bd22-547a9995c437" (UID: "51725959-f0c8-4b0f-bd22-547a9995c437"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:12.420789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.420742 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-istio-envoy" pod "51725959-f0c8-4b0f-bd22-547a9995c437" (UID: "51725959-f0c8-4b0f-bd22-547a9995c437"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:12.465826 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.464529 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51725959-f0c8-4b0f-bd22-547a9995c437-istio-podinfo" pod "51725959-f0c8-4b0f-bd22-547a9995c437" (UID: "51725959-f0c8-4b0f-bd22-547a9995c437"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:52:12.502157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.501883 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:52:12.502325 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.502314 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51725959-f0c8-4b0f-bd22-547a9995c437-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:52:12.502397 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.502390 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51725959-f0c8-4b0f-bd22-547a9995c437-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:52:12.502495 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.502488 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51725959-f0c8-4b0f-bd22-547a9995c437-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:52:12.502548 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.502542 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:52:12.502602 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.502596 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:52:12.502656 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.502650 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:52:12.502707 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.502700 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51725959-f0c8-4b0f-bd22-547a9995c437-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:52:12.502760 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.502752 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlvf9\" (UniqueName: \"kubernetes.io/projected/51725959-f0c8-4b0f-bd22-547a9995c437-kube-api-access-jlvf9\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:52:12.617056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.616976 6202 patch_prober.go:29] interesting pod/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 30 16:52:12.617056 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.617031 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" podUID="927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 30 16:52:12.682383 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:52:12.682357 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17152dfe_8124_40b1_b6b9_163d4901d269.slice/crio-8fb681c7a1bbcffd6e5bdfa26086b36b3863cf52b75cec574102a1b3c4cba161 WatchSource:0}: Error finding container 8fb681c7a1bbcffd6e5bdfa26086b36b3863cf52b75cec574102a1b3c4cba161: Status 404 returned error can't find the container with id 8fb681c7a1bbcffd6e5bdfa26086b36b3863cf52b75cec574102a1b3c4cba161 Oct 30 16:52:12.686228 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.685859 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw"] Oct 30 16:52:12.691483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.691454 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z"] Oct 30 16:52:12.695241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.695215 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-d6z8z"] Oct 30 16:52:12.760110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:12.745262 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:13.336711 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:13.336672 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" event={"ID":"17152dfe-8124-40b1-b6b9-163d4901d269","Type":"ContainerStarted","Data":"1a203d503635ffffbf156efa8af11d1d1e95d3af1836a98ac304f9881a932a75"} Oct 30 16:52:13.337280 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:13.336742 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" event={"ID":"17152dfe-8124-40b1-b6b9-163d4901d269","Type":"ContainerStarted","Data":"8fb681c7a1bbcffd6e5bdfa26086b36b3863cf52b75cec574102a1b3c4cba161"} Oct 30 16:52:13.337280 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:13.336937 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" Oct 30 16:52:13.388596 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:13.388528 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" podStartSLOduration=2.388509941 podStartE2EDuration="2.388509941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:13.385264512 +0000 UTC m=+1847.095719275" watchObservedRunningTime="2026-10-30 16:52:13.388509941 +0000 UTC m=+1847.098964685" Oct 30 16:52:13.617455 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:13.617346 6202 patch_prober.go:29] interesting pod/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 30 16:52:13.617455 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:13.617431 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" podUID="927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 30 16:52:14.105470 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:52:14.105422 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:52:14.178042 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:14.178009 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" Oct 30 16:52:14.306730 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:14.306695 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:14.432662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:14.432578 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51725959-f0c8-4b0f-bd22-547a9995c437" path="/var/lib/kubelet/pods/51725959-f0c8-4b0f-bd22-547a9995c437/volumes" Oct 30 16:52:14.617794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:14.617734 6202 patch_prober.go:29] interesting pod/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 30 16:52:14.617968 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:14.617811 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" podUID="927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 30 16:52:15.187491 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:15.187463 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" Oct 30 16:52:15.617695 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:15.617662 6202 patch_prober.go:29] interesting pod/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 30 16:52:15.618092 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:15.617714 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" podUID="927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 30 16:52:15.980868 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:15.980385 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:16.617351 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:16.617314 6202 patch_prober.go:29] interesting pod/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 30 16:52:16.617554 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:16.617370 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" podUID="927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 30 16:52:17.031177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:17.031094 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz"] Oct 30 16:52:17.042733 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:17.042703 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-743d2769-2nwbz"] Oct 30 16:52:17.455683 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:17.455636 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:17.622323 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:17.622285 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:52:17.622678 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:17.622647 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:52:17.623400 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:17.623376 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 16:52:18.413440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:18.413408 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0bf836f-6716-4a39-8422-0c9639aa5f8c" path="/var/lib/kubelet/pods/d0bf836f-6716-4a39-8422-0c9639aa5f8c/volumes" Oct 30 16:52:19.085177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:19.085142 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:20.580045 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:20.580009 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:21.309628 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:21.309596 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" Oct 30 16:52:22.356387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:22.356346 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:23.428119 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:23.428081 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:23.564378 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:23.564342 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:23.687950 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:23.687862 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:24.125150 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:24.125117 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:24.427732 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:24.427657 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" Oct 30 16:52:25.306829 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:25.306786 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:25.426809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:25.426780 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:25.439929 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:25.439902 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7"] Oct 30 16:52:25.440809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:25.440792 6202 cpu_manager.go:414] "containerMap: removing container" podUID="51725959-f0c8-4b0f-bd22-547a9995c437" containerName="istio-proxy" Oct 30 16:52:25.440873 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:25.440812 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51725959-f0c8-4b0f-bd22-547a9995c437" containerName="istio-proxy" Oct 30 16:52:25.440976 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:25.440965 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="51725959-f0c8-4b0f-bd22-547a9995c437" containerName="istio-proxy" Oct 30 16:52:25.445037 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:25.445016 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" Oct 30 16:52:25.465100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:25.465076 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7"] Oct 30 16:52:25.472339 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:25.472310 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-3ed10a\"" Oct 30 16:52:25.472490 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:25.472324 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-3ed10a-dockercfg-m5r2m\"" Oct 30 16:52:25.671905 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:25.668032 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw4gv\" (UniqueName: \"kubernetes.io/projected/2d98a198-b4de-422f-a2f4-df5e059d105d-kube-api-access-lw4gv\") pod \"payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7\" (UID: \"2d98a198-b4de-422f-a2f4-df5e059d105d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" Oct 30 16:52:25.671905 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:25.668201 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d98a198-b4de-422f-a2f4-df5e059d105d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7\" (UID: \"2d98a198-b4de-422f-a2f4-df5e059d105d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" Oct 30 16:52:25.771018 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:25.770978 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d98a198-b4de-422f-a2f4-df5e059d105d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7\" (UID: \"2d98a198-b4de-422f-a2f4-df5e059d105d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" Oct 30 16:52:25.771504 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:25.771476 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw4gv\" (UniqueName: \"kubernetes.io/projected/2d98a198-b4de-422f-a2f4-df5e059d105d-kube-api-access-lw4gv\") pod \"payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7\" (UID: \"2d98a198-b4de-422f-a2f4-df5e059d105d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" Oct 30 16:52:25.772131 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:25.772108 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d98a198-b4de-422f-a2f4-df5e059d105d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7\" (UID: \"2d98a198-b4de-422f-a2f4-df5e059d105d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" Oct 30 16:52:25.795507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:25.795470 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw4gv\" (UniqueName: \"kubernetes.io/projected/2d98a198-b4de-422f-a2f4-df5e059d105d-kube-api-access-lw4gv\") pod \"payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7\" (UID: \"2d98a198-b4de-422f-a2f4-df5e059d105d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" Oct 30 16:52:26.060365 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:26.060277 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" Oct 30 16:52:26.101406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:26.101369 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:26.368459 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:52:26.368419 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d98a198_b4de_422f_a2f4_df5e059d105d.slice/crio-294643be0c731aff857adc4731a8b609013d71547933207efff32a1259d57af5 WatchSource:0}: Error finding container 294643be0c731aff857adc4731a8b609013d71547933207efff32a1259d57af5: Status 404 returned error can't find the container with id 294643be0c731aff857adc4731a8b609013d71547933207efff32a1259d57af5 Oct 30 16:52:26.369024 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:26.368996 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7"] Oct 30 16:52:26.685292 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:26.685206 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" event={"ID":"2d98a198-b4de-422f-a2f4-df5e059d105d","Type":"ContainerStarted","Data":"e1c17a9be77260afe59bef057caab8c8060c9f1e911ee1b0533718c9f7559cfd"} Oct 30 16:52:26.685292 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:26.685259 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" event={"ID":"2d98a198-b4de-422f-a2f4-df5e059d105d","Type":"ContainerStarted","Data":"294643be0c731aff857adc4731a8b609013d71547933207efff32a1259d57af5"} Oct 30 16:52:26.685292 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:26.685292 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" Oct 30 16:52:26.728734 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:26.728684 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" podStartSLOduration=1.728670685 podStartE2EDuration="1.728670685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:26.724444966 +0000 UTC m=+1860.434899710" watchObservedRunningTime="2026-10-30 16:52:26.728670685 +0000 UTC m=+1860.439125429" Oct 30 16:52:27.739079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:27.739043 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:29.556078 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:29.556041 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:31.138967 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:31.138921 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:31.372613 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:31.372576 6202 scope.go:122] "RemoveContainer" containerID="9c3f2cc44e023f1d325ab0396d0a4cb7df04b8e335e590272d965df44fd2deef" Oct 30 16:52:32.718349 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:32.718287 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:34.342134 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:34.342101 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:35.798565 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:35.798523 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:35.918601 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:35.918567 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:36.037361 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:36.037316 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:37.573861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:37.573826 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:37.698037 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:37.698000 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" Oct 30 16:52:39.056162 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:39.056125 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp"] Oct 30 16:52:39.059850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:39.059815 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae109b2d-46tbp"] Oct 30 16:52:39.077392 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:39.077364 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:39.674248 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:39.674213 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:39.815952 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:39.815924 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:40.120750 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:40.107000 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:40.131000 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:40.130965 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf"] Oct 30 16:52:40.136968 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:40.136932 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-407eab-6svsf"] Oct 30 16:52:40.251317 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:40.251277 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:40.413022 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:40.412945 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d8a6c96-1fff-4292-88b5-4d939fd0c99a" path="/var/lib/kubelet/pods/4d8a6c96-1fff-4292-88b5-4d939fd0c99a/volumes" Oct 30 16:52:40.413606 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:40.413580 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a8e320-7b5b-4235-910e-d65631f01cdd" path="/var/lib/kubelet/pods/71a8e320-7b5b-4235-910e-d65631f01cdd/volumes" Oct 30 16:52:40.681275 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:40.681191 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:42.202420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:42.202386 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:43.886534 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:43.886502 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:44.124080 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:52:44.124043 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:52:45.575577 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:45.575536 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:45.731251 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:45.731216 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:45.884648 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:45.884515 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:47.688217 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:47.688183 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:49.143520 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:49.143469 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:49.301298 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:49.301004 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:49.531492 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:49.531372 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:49.709308 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:49.709268 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:49.939484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:49.939448 6202 generic.go:356] "Generic (PLEG): container finished" podID="3554e861-457c-4806-99ee-ad9e27786362" containerID="08461321eda010962c4ec4af030200c3f2ca572dcb0544a345b68550ea5b7a08" exitCode=0 Oct 30 16:52:49.939484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:49.939476 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" event={"ID":"3554e861-457c-4806-99ee-ad9e27786362","Type":"ContainerDied","Data":"08461321eda010962c4ec4af030200c3f2ca572dcb0544a345b68550ea5b7a08"} Oct 30 16:52:49.940014 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:49.939995 6202 scope.go:122] "RemoveContainer" containerID="08461321eda010962c4ec4af030200c3f2ca572dcb0544a345b68550ea5b7a08" Oct 30 16:52:49.942282 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:49.942248 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:52:49.942356 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:49.942341 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:52:49.942409 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:49.942393 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:52:50.506553 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:50.506520 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:50.900115 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:50.899614 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:52:50.906506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:50.906471 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg"] Oct 30 16:52:50.912935 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:50.912906 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:50.934646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:50.934618 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ea77d8-openshift-default-dockercfg-hkvs8\"" Oct 30 16:52:50.980980 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:50.969412 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg"] Oct 30 16:52:50.999429 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:50.999372 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" event={"ID":"3554e861-457c-4806-99ee-ad9e27786362","Type":"ContainerStarted","Data":"cad1712e838221e01ef37e03a0caf99d554bbaa897a1c914f260d016ddcb1a14"} Oct 30 16:52:51.001037 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.001007 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-data\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.001281 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.001261 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcq79\" (UniqueName: \"kubernetes.io/projected/323c4d64-a380-4e62-9779-1a1a87ae3308-kube-api-access-tcq79\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.001434 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.001417 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-workload-socket\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.001583 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.001569 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323c4d64-a380-4e62-9779-1a1a87ae3308-istiod-ca-cert\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.001746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.001731 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-token\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.001936 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.001920 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-workload-certs\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.002084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.002046 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-credential-socket\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.002264 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.002250 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-podinfo\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.002381 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.002368 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-envoy\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.260843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.259219 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-envoy\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.260843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.259325 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-data\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.260843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.259398 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcq79\" (UniqueName: \"kubernetes.io/projected/323c4d64-a380-4e62-9779-1a1a87ae3308-kube-api-access-tcq79\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.260843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.259435 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-workload-socket\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.260843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.259506 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323c4d64-a380-4e62-9779-1a1a87ae3308-istiod-ca-cert\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.260843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.259582 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-token\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.260843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.259669 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-workload-certs\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.260843 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.259702 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-credential-socket\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.297304 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.281397 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-podinfo\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.326568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.303399 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-data\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.326568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.304627 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323c4d64-a380-4e62-9779-1a1a87ae3308-istiod-ca-cert\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.326568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.304848 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-workload-certs\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.326568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.304917 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-credential-socket\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.326568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.325345 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-workload-socket\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.329324 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.329246 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-envoy\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.366851 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.343412 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-podinfo\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.366851 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.343984 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:51.366851 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.344968 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcq79\" (UniqueName: \"kubernetes.io/projected/323c4d64-a380-4e62-9779-1a1a87ae3308-kube-api-access-tcq79\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.407881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.398434 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-token\") pod \"e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.477582 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.477539 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:51.577158 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.577121 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:52:51.578646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.578617 6202 patch_prober.go:29] interesting pod/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 16:52:51.578833 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.578688 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" podUID="3554e861-457c-4806-99ee-ad9e27786362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 16:52:51.599204 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.599143 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:51.599204 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:51.599165 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:52.058792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:52.038656 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg"] Oct 30 16:52:52.058792 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:52:52.040898 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323c4d64_a380_4e62_9779_1a1a87ae3308.slice/crio-a65441b9a4547e143efe66b23b151f8d20cdcfbb942bba27bf1d6fc376d48f17 WatchSource:0}: Error finding container a65441b9a4547e143efe66b23b151f8d20cdcfbb942bba27bf1d6fc376d48f17: Status 404 returned error can't find the container with id a65441b9a4547e143efe66b23b151f8d20cdcfbb942bba27bf1d6fc376d48f17 Oct 30 16:52:52.077444 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:52.077393 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:52:52.084784 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:52.077821 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:52:52.084784 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:52.078136 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:52:52.577218 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:52.577180 6202 patch_prober.go:29] interesting pod/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 16:52:52.577782 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:52.577251 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" podUID="3554e861-457c-4806-99ee-ad9e27786362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 16:52:53.016909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:53.016820 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" event={"ID":"323c4d64-a380-4e62-9779-1a1a87ae3308","Type":"ContainerStarted","Data":"381811ac503f75277f36a76ce038e7b9d3cc91bcf5273179879969a09538c8c7"} Oct 30 16:52:53.016909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:53.016873 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" event={"ID":"323c4d64-a380-4e62-9779-1a1a87ae3308","Type":"ContainerStarted","Data":"a65441b9a4547e143efe66b23b151f8d20cdcfbb942bba27bf1d6fc376d48f17"} Oct 30 16:52:53.075701 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:53.075640 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" podStartSLOduration=3.07561915 podStartE2EDuration="3.07561915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:53.074183837 +0000 UTC m=+1886.784638585" watchObservedRunningTime="2026-10-30 16:52:53.07561915 +0000 UTC m=+1886.786073891" Oct 30 16:52:53.256360 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:53.256263 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:53.577358 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:53.577319 6202 patch_prober.go:29] interesting pod/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 16:52:53.577932 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:53.577389 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" podUID="3554e861-457c-4806-99ee-ad9e27786362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 16:52:53.600388 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:53.600357 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:53.607060 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:53.607035 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:54.025535 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:54.025459 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:54.026452 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:54.026435 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:52:54.576760 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:54.576733 6202 patch_prober.go:29] interesting pod/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 16:52:54.577046 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:54.577018 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" podUID="3554e861-457c-4806-99ee-ad9e27786362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 16:52:54.620679 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:54.620647 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:55.087603 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:55.087564 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:55.549847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:55.542890 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:55.577980 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:55.577949 6202 patch_prober.go:29] interesting pod/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 16:52:55.578239 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:55.578214 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" podUID="3554e861-457c-4806-99ee-ad9e27786362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 16:52:55.762986 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:55.762946 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:56.108428 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:56.108382 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:56.635845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:56.635810 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:52:56.636619 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:56.636538 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:52:56.638838 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:56.638812 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:52:57.195724 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:57.195683 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:58.023323 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:58.023281 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:58.152439 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:58.152408 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:59.033374 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:52:59.033337 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:00.650185 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:00.650127 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:02.135363 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:02.135330 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:02.934853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:02.934814 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:03.058538 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:03.058501 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:03.754450 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:03.754415 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:05.247912 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:05.247880 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:05.369423 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:05.369339 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:06.283211 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:06.283172 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7"] Oct 30 16:53:06.283719 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:06.283459 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" podUID="78eb7d90-810f-48fb-aaaf-e54f2806d02a" containerName="istio-proxy" containerID="cri-o://c5fe7f2711a9a958a8ac8eea0ee3fa528100114e6a79c5c74d1bcbc8ff663503" gracePeriod=30 Oct 30 16:53:07.654786 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:07.654728 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:09.600016 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:09.599898 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:11.279086 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.279049 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:11.629369 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.629348 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:53:11.777474 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.777441 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-credential-socket\") pod \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " Oct 30 16:53:11.777687 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.777675 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-data\") pod \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " Oct 30 16:53:11.777829 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.777816 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istiod-ca-cert\") pod \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " Oct 30 16:53:11.778062 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.778050 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-token\" (UniqueName: \"kubernetes.io/projected/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-token\") pod \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " Oct 30 16:53:11.778178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.778169 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-envoy\") pod \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " Oct 30 16:53:11.778263 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.778255 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78eb7d90-810f-48fb-aaaf-e54f2806d02a-kube-api-access-b4wxw\" (UniqueName: \"kubernetes.io/projected/78eb7d90-810f-48fb-aaaf-e54f2806d02a-kube-api-access-b4wxw\") pod \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " Oct 30 16:53:11.778427 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.778418 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-podinfo\") pod \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " Oct 30 16:53:11.778500 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.778491 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-workload-socket\") pod \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " Oct 30 16:53:11.778567 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.778559 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-workload-certs\") pod \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\" (UID: \"78eb7d90-810f-48fb-aaaf-e54f2806d02a\") " Oct 30 16:53:11.779505 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.779473 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-data" pod "78eb7d90-810f-48fb-aaaf-e54f2806d02a" (UID: "78eb7d90-810f-48fb-aaaf-e54f2806d02a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:11.779695 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.779486 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-workload-certs" pod "78eb7d90-810f-48fb-aaaf-e54f2806d02a" (UID: "78eb7d90-810f-48fb-aaaf-e54f2806d02a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:11.779796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.779700 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-credential-socket" pod "78eb7d90-810f-48fb-aaaf-e54f2806d02a" (UID: "78eb7d90-810f-48fb-aaaf-e54f2806d02a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:11.783043 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.782872 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-podinfo" pod "78eb7d90-810f-48fb-aaaf-e54f2806d02a" (UID: "78eb7d90-810f-48fb-aaaf-e54f2806d02a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:53:11.783313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.783149 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-workload-socket" pod "78eb7d90-810f-48fb-aaaf-e54f2806d02a" (UID: "78eb7d90-810f-48fb-aaaf-e54f2806d02a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:11.783313 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.783210 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-envoy" pod "78eb7d90-810f-48fb-aaaf-e54f2806d02a" (UID: "78eb7d90-810f-48fb-aaaf-e54f2806d02a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:11.783465 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.783443 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istiod-ca-cert" pod "78eb7d90-810f-48fb-aaaf-e54f2806d02a" (UID: "78eb7d90-810f-48fb-aaaf-e54f2806d02a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:53:11.796157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.796121 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-token" pod "78eb7d90-810f-48fb-aaaf-e54f2806d02a" (UID: "78eb7d90-810f-48fb-aaaf-e54f2806d02a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:11.796291 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.796275 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78eb7d90-810f-48fb-aaaf-e54f2806d02a-kube-api-access-b4wxw" pod "78eb7d90-810f-48fb-aaaf-e54f2806d02a" (UID: "78eb7d90-810f-48fb-aaaf-e54f2806d02a"). InnerVolumeSpecName "kube-api-access-b4wxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:11.881351 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.881264 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4wxw\" (UniqueName: \"kubernetes.io/projected/78eb7d90-810f-48fb-aaaf-e54f2806d02a-kube-api-access-b4wxw\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:53:11.881563 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.881546 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:53:11.881657 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.881646 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:53:11.881743 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.881732 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:53:11.881855 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.881844 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:53:11.881986 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.881974 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:53:11.882075 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.882065 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:53:11.882153 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.882144 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:53:11.882230 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:11.882221 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78eb7d90-810f-48fb-aaaf-e54f2806d02a-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:53:12.273855 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:12.273737 6202 generic.go:356] "Generic (PLEG): container finished" podID="78eb7d90-810f-48fb-aaaf-e54f2806d02a" containerID="c5fe7f2711a9a958a8ac8eea0ee3fa528100114e6a79c5c74d1bcbc8ff663503" exitCode=0 Oct 30 16:53:12.273855 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:12.273821 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" event={"ID":"78eb7d90-810f-48fb-aaaf-e54f2806d02a","Type":"ContainerDied","Data":"c5fe7f2711a9a958a8ac8eea0ee3fa528100114e6a79c5c74d1bcbc8ff663503"} Oct 30 16:53:12.274099 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:12.273867 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" event={"ID":"78eb7d90-810f-48fb-aaaf-e54f2806d02a","Type":"ContainerDied","Data":"dd46a1d1a93592e460db7ff78b3f77e86fc78bf9ae46e89733a9cffaca42e41d"} Oct 30 16:53:12.274099 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:12.273898 6202 scope.go:122] "RemoveContainer" containerID="c5fe7f2711a9a958a8ac8eea0ee3fa528100114e6a79c5c74d1bcbc8ff663503" Oct 30 16:53:12.274099 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:12.274041 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7" Oct 30 16:53:12.289117 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:12.289085 6202 scope.go:122] "RemoveContainer" containerID="c5fe7f2711a9a958a8ac8eea0ee3fa528100114e6a79c5c74d1bcbc8ff663503" Oct 30 16:53:12.289525 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:53:12.289448 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5fe7f2711a9a958a8ac8eea0ee3fa528100114e6a79c5c74d1bcbc8ff663503\": container with ID starting with c5fe7f2711a9a958a8ac8eea0ee3fa528100114e6a79c5c74d1bcbc8ff663503 not found: ID does not exist" containerID="c5fe7f2711a9a958a8ac8eea0ee3fa528100114e6a79c5c74d1bcbc8ff663503" Oct 30 16:53:12.289525 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:12.289483 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5fe7f2711a9a958a8ac8eea0ee3fa528100114e6a79c5c74d1bcbc8ff663503"} err="failed to get container status \"c5fe7f2711a9a958a8ac8eea0ee3fa528100114e6a79c5c74d1bcbc8ff663503\": rpc error: code = NotFound desc = could not find container \"c5fe7f2711a9a958a8ac8eea0ee3fa528100114e6a79c5c74d1bcbc8ff663503\": container with ID starting with c5fe7f2711a9a958a8ac8eea0ee3fa528100114e6a79c5c74d1bcbc8ff663503 not found: ID does not exist" Oct 30 16:53:12.328978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:12.328947 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7"] Oct 30 16:53:12.362789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:12.362705 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-l2cn7"] Oct 30 16:53:12.413615 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:12.413586 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78eb7d90-810f-48fb-aaaf-e54f2806d02a" path="/var/lib/kubelet/pods/78eb7d90-810f-48fb-aaaf-e54f2806d02a/volumes" Oct 30 16:53:13.017158 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:13.017036 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:14.077511 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:53:14.077470 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:53:14.818871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:14.818827 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:16.286037 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:16.285990 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:17.346130 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:17.346096 6202 generic.go:356] "Generic (PLEG): container finished" podID="5d729b26-7129-4f64-95d9-860074961e2d" containerID="bfba618ec63d7227ae99120abb61ae940f89d53e880cceb04b3b442fab5705f2" exitCode=0 Oct 30 16:53:17.346130 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:17.346133 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" event={"ID":"5d729b26-7129-4f64-95d9-860074961e2d","Type":"ContainerDied","Data":"bfba618ec63d7227ae99120abb61ae940f89d53e880cceb04b3b442fab5705f2"} Oct 30 16:53:17.346676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:17.346532 6202 scope.go:122] "RemoveContainer" containerID="bfba618ec63d7227ae99120abb61ae940f89d53e880cceb04b3b442fab5705f2" Oct 30 16:53:17.348496 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:17.348458 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:53:17.348620 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:17.348549 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:53:17.348620 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:17.348602 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:53:17.968815 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:17.968748 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:18.388943 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:18.388905 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" event={"ID":"5d729b26-7129-4f64-95d9-860074961e2d","Type":"ContainerStarted","Data":"841b4ffd95bd9502a5e301f4809ff02ba68eb8604a08f7938eca92e0f7bc6f1b"} Oct 30 16:53:18.816563 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:18.816481 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:53:18.817940 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:18.817909 6202 patch_prober.go:29] interesting pod/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 16:53:18.818064 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:18.817971 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" podUID="5d729b26-7129-4f64-95d9-860074961e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 16:53:19.577331 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:19.577300 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:19.816882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:19.816848 6202 patch_prober.go:29] interesting pod/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 16:53:19.817077 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:19.816907 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" podUID="5d729b26-7129-4f64-95d9-860074961e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 16:53:20.184845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:20.184813 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:53:20.817357 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:20.817318 6202 patch_prober.go:29] interesting pod/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 16:53:20.817785 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:20.817384 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" podUID="5d729b26-7129-4f64-95d9-860074961e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 16:53:21.095226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:21.095186 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:21.817064 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:21.817032 6202 patch_prober.go:29] interesting pod/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 16:53:21.817257 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:21.817097 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" podUID="5d729b26-7129-4f64-95d9-860074961e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 16:53:22.727205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:22.726812 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:22.816568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:22.816525 6202 patch_prober.go:29] interesting pod/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 16:53:22.816882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:22.816589 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" podUID="5d729b26-7129-4f64-95d9-860074961e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 16:53:23.890111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:23.890076 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:53:23.894306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:23.894281 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 16:53:24.890099 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:24.890069 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:25.010167 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:25.010138 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:25.139140 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:25.139094 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:26.510482 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:26.510430 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:28.163404 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:28.163364 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:29.716101 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:29.716062 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:31.241819 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:31.241059 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:31.452178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:31.452145 6202 scope.go:122] "RemoveContainer" containerID="b0bd806e805e56199ee57f91b736210203e3f3858efff40cab280be7d3e78cab" Oct 30 16:53:31.464070 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:31.464047 6202 scope.go:122] "RemoveContainer" containerID="997978b7af66b4c7370150be4ce6ee0cc724876715d72de8cdb913480c735bcd" Oct 30 16:53:33.160218 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:33.160181 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:33.986207 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:33.986169 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:34.878580 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:34.878547 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:35.385868 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:35.385827 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:35.850175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:35.850132 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv"] Oct 30 16:53:35.851757 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:35.851727 6202 cpu_manager.go:414] "containerMap: removing container" podUID="78eb7d90-810f-48fb-aaaf-e54f2806d02a" containerName="istio-proxy" Oct 30 16:53:35.851960 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:35.851945 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78eb7d90-810f-48fb-aaaf-e54f2806d02a" containerName="istio-proxy" Oct 30 16:53:35.852338 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:35.852315 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="78eb7d90-810f-48fb-aaaf-e54f2806d02a" containerName="istio-proxy" Oct 30 16:53:35.858812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:35.858745 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:35.879044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:35.879013 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv"] Oct 30 16:53:35.884262 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:35.884213 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9c1e26-openshift-default-dockercfg-r522j\"" Oct 30 16:53:35.904247 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:35.904215 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:35.952991 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:35.946084 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istiod-ca-cert\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:35.967776 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:35.967712 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-token\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:35.968174 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:35.968145 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-credential-socket\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:35.968299 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:35.968275 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-envoy\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:35.970001 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:35.968421 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-workload-socket\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:35.970001 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:35.968635 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-podinfo\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:35.970001 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:35.968708 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-data\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:35.970001 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:35.968744 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-workload-certs\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:35.970001 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:35.969031 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvk5x\" (UniqueName: \"kubernetes.io/projected/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-kube-api-access-kvk5x\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.110858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.078529 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvk5x\" (UniqueName: \"kubernetes.io/projected/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-kube-api-access-kvk5x\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.110858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.078608 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istiod-ca-cert\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.110858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.078643 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-token\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.110858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.078705 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-credential-socket\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.110858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.078824 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-envoy\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.110858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.078914 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-workload-socket\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.110858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.078947 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-podinfo\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.110858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.079005 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-data\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.110858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.079039 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-workload-certs\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.110858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.079694 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-workload-certs\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.110858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.080101 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istiod-ca-cert\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.110858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.081268 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-credential-socket\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.110858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.081512 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-workload-socket\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.110858 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.081816 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-data\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.122998 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.122800 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:36.155652 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.151733 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-envoy\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.155652 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.154500 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvk5x\" (UniqueName: \"kubernetes.io/projected/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-kube-api-access-kvk5x\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.155652 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.155582 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-podinfo\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.163796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.161955 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-token\") pod \"e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.192885 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.192099 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:36.568751 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.568711 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv"] Oct 30 16:53:36.569655 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:53:36.569626 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7320e48_7c58_40b0_bd5b_03b9f4bb551f.slice/crio-09eba606f8738347fd1c504bad5379cd3100a85f406aa3b6fa1dff565da17fbb WatchSource:0}: Error finding container 09eba606f8738347fd1c504bad5379cd3100a85f406aa3b6fa1dff565da17fbb: Status 404 returned error can't find the container with id 09eba606f8738347fd1c504bad5379cd3100a85f406aa3b6fa1dff565da17fbb Oct 30 16:53:36.572742 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.572705 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:53:36.572861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.572799 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:53:36.572861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.572830 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:53:36.784957 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:36.784920 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:37.008615 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:37.008584 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:37.556465 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:37.556429 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:37.560850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:37.560818 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" event={"ID":"c7320e48-7c58-40b0-bd5b-03b9f4bb551f","Type":"ContainerStarted","Data":"d579cd1a477fd869c51061a833073b281c5b2e206e1911192dfc00a4835ac8b1"} Oct 30 16:53:37.561005 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:37.560879 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" event={"ID":"c7320e48-7c58-40b0-bd5b-03b9f4bb551f","Type":"ContainerStarted","Data":"09eba606f8738347fd1c504bad5379cd3100a85f406aa3b6fa1dff565da17fbb"} Oct 30 16:53:37.635209 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:37.635155 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" podStartSLOduration=2.63514177 podStartE2EDuration="2.63514177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:37.622189908 +0000 UTC m=+1931.332644650" watchObservedRunningTime="2026-10-30 16:53:37.63514177 +0000 UTC m=+1931.345596511" Oct 30 16:53:37.676820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:37.676749 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:37.850471 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:37.850435 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:38.200232 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:38.200145 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:38.209002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:38.208974 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:38.540640 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:38.540540 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:38.570735 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:38.570700 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:38.572049 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:38.572023 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:53:38.660256 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:38.660204 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:39.784717 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:39.784679 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:40.096440 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:40.096392 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:40.213457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:40.213427 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:40.529682 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:40.529593 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:40.664564 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:40.664525 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:42.170728 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:42.170696 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:42.284080 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:42.284038 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:42.339193 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:42.339159 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5"] Oct 30 16:53:42.344743 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:42.344722 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" Oct 30 16:53:42.360191 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:42.360141 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9c1e26\"" Oct 30 16:53:42.372092 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:42.372054 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9c1e26-dockercfg-b7b4m\"" Oct 30 16:53:42.519348 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:42.519256 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv9zn\" (UniqueName: \"kubernetes.io/projected/3ddb2df8-3531-4d65-a0fe-7fe1facc43ee-kube-api-access-sv9zn\") pod \"payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5\" (UID: \"3ddb2df8-3531-4d65-a0fe-7fe1facc43ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" Oct 30 16:53:42.618428 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:42.618359 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ddb2df8-3531-4d65-a0fe-7fe1facc43ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5\" (UID: \"3ddb2df8-3531-4d65-a0fe-7fe1facc43ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" Oct 30 16:53:42.622567 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:42.622537 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5"] Oct 30 16:53:42.666874 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:42.666823 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:42.720808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:42.720725 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv9zn\" (UniqueName: \"kubernetes.io/projected/3ddb2df8-3531-4d65-a0fe-7fe1facc43ee-kube-api-access-sv9zn\") pod \"payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5\" (UID: \"3ddb2df8-3531-4d65-a0fe-7fe1facc43ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" Oct 30 16:53:42.721015 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:42.720937 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ddb2df8-3531-4d65-a0fe-7fe1facc43ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5\" (UID: \"3ddb2df8-3531-4d65-a0fe-7fe1facc43ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" Oct 30 16:53:42.721667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:42.721641 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ddb2df8-3531-4d65-a0fe-7fe1facc43ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5\" (UID: \"3ddb2df8-3531-4d65-a0fe-7fe1facc43ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" Oct 30 16:53:42.747873 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:42.747843 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv9zn\" (UniqueName: \"kubernetes.io/projected/3ddb2df8-3531-4d65-a0fe-7fe1facc43ee-kube-api-access-sv9zn\") pod \"payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5\" (UID: \"3ddb2df8-3531-4d65-a0fe-7fe1facc43ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" Oct 30 16:53:42.961044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:42.961010 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" Oct 30 16:53:43.265553 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:43.265521 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5"] Oct 30 16:53:43.266256 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:53:43.266208 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ddb2df8_3531_4d65_a0fe_7fe1facc43ee.slice/crio-e1b147057570e022a1c9d0e0b5719ae0bb165118479c9a416e00f8b8f01ed60e WatchSource:0}: Error finding container e1b147057570e022a1c9d0e0b5719ae0bb165118479c9a416e00f8b8f01ed60e: Status 404 returned error can't find the container with id e1b147057570e022a1c9d0e0b5719ae0bb165118479c9a416e00f8b8f01ed60e Oct 30 16:53:43.475950 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:43.475910 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:43.610400 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:43.610359 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:43.683123 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:43.683088 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" event={"ID":"3ddb2df8-3531-4d65-a0fe-7fe1facc43ee","Type":"ContainerStarted","Data":"a28c6990e01344e61b4c79d43cb62bf0ef4f692cb32791d1844fb9ed834b8744"} Oct 30 16:53:43.683305 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:43.683157 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" Oct 30 16:53:43.683305 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:43.683179 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" event={"ID":"3ddb2df8-3531-4d65-a0fe-7fe1facc43ee","Type":"ContainerStarted","Data":"e1b147057570e022a1c9d0e0b5719ae0bb165118479c9a416e00f8b8f01ed60e"} Oct 30 16:53:43.731289 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:43.731245 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" podStartSLOduration=1.731231889 podStartE2EDuration="1.731231889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:43.722721138 +0000 UTC m=+1937.433175880" watchObservedRunningTime="2026-10-30 16:53:43.731231889 +0000 UTC m=+1937.441686680" Oct 30 16:53:44.075080 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:53:44.075042 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:53:44.394059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:44.393958 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:46.113808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:46.113742 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:47.571400 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:47.571359 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:47.840172 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:47.840094 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:47.989399 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:47.989364 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r"] Oct 30 16:53:47.994153 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:47.994129 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.018917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.018897 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-689899-openshift-default-dockercfg-x7vsc\"" Oct 30 16:53:48.032938 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.032909 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r"] Oct 30 16:53:48.067788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.067688 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:48.117258 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.117178 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.117258 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.117211 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84mjj\" (UniqueName: \"kubernetes.io/projected/318e8454-0cf7-485e-b774-9ce1b368cac6-kube-api-access-84mjj\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.117449 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.117371 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.117449 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.117439 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.117551 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.117533 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.117607 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.117559 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.117689 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.117676 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.117738 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.117727 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.117828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.117816 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318e8454-0cf7-485e-b774-9ce1b368cac6-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.199533 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.199503 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:48.219234 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.219204 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.219375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.219248 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318e8454-0cf7-485e-b774-9ce1b368cac6-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.219433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.219384 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.219433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.219411 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84mjj\" (UniqueName: \"kubernetes.io/projected/318e8454-0cf7-485e-b774-9ce1b368cac6-kube-api-access-84mjj\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.219528 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.219470 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.219568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.219540 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.219635 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.219606 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.219684 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.219640 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.219749 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.219732 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.219838 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.219804 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.220091 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.220073 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.220147 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.220133 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318e8454-0cf7-485e-b774-9ce1b368cac6-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.220186 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.220145 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.220371 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.220340 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.221702 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.221683 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.222431 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.222410 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.245157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.245113 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84mjj\" (UniqueName: \"kubernetes.io/projected/318e8454-0cf7-485e-b774-9ce1b368cac6-kube-api-access-84mjj\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.245715 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.245691 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.312723 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.312387 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:48.818001 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.817973 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r"] Oct 30 16:53:48.818389 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:53:48.818263 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318e8454_0cf7_485e_b774_9ce1b368cac6.slice/crio-4e9f533b4aaab210475f76ad937b5f1f26bb10cb2bcf27dcc053c444dfc460e9 WatchSource:0}: Error finding container 4e9f533b4aaab210475f76ad937b5f1f26bb10cb2bcf27dcc053c444dfc460e9: Status 404 returned error can't find the container with id 4e9f533b4aaab210475f76ad937b5f1f26bb10cb2bcf27dcc053c444dfc460e9 Oct 30 16:53:48.821199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.821166 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:53:48.821306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.821230 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:53:48.821306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:48.821261 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:53:49.564163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:49.564131 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:49.753025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:49.752898 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" event={"ID":"318e8454-0cf7-485e-b774-9ce1b368cac6","Type":"ContainerStarted","Data":"fcaf838b4520c9a9d6431c3843ebbe8f76a404c6e31aa532650a80b032f843c7"} Oct 30 16:53:49.753025 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:49.752986 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" event={"ID":"318e8454-0cf7-485e-b774-9ce1b368cac6","Type":"ContainerStarted","Data":"4e9f533b4aaab210475f76ad937b5f1f26bb10cb2bcf27dcc053c444dfc460e9"} Oct 30 16:53:49.856587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:49.856513 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" podStartSLOduration=2.856490105 podStartE2EDuration="2.856490105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:49.818735633 +0000 UTC m=+1943.529190400" watchObservedRunningTime="2026-10-30 16:53:49.856490105 +0000 UTC m=+1943.566944854" Oct 30 16:53:50.313646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:50.313555 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:50.321214 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:50.321187 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:50.763150 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:50.763121 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:50.764314 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:50.764291 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:53:51.534973 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:51.534937 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:51.884258 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:51.884226 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r"] Oct 30 16:53:51.884469 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:51.884442 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" podUID="970bb829-3799-40ba-a22d-f1e1ae061631" containerName="istio-proxy" containerID="cri-o://8a39b3ba925f37d64a88a7b2a4e699344740cdb0deb9aa169baeec6a394c820e" gracePeriod=30 Oct 30 16:53:52.626677 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:52.626642 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:52.773300 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:52.773250 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:53.745266 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:53.745231 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:54.695241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:54.695212 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" Oct 30 16:53:55.331683 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:55.331651 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:55.477488 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:55.477450 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:55.719033 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:55.718945 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:56.126316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:56.126276 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:56.245315 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:56.245283 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:56.860955 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:56.860909 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g"] Oct 30 16:53:56.894155 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:56.894122 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:56.910821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:56.897949 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:56.930220 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:56.930184 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g"] Oct 30 16:53:56.955478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:56.955387 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-7c4b84-openshift-default-dockercfg-5sg2r\"" Oct 30 16:53:57.029160 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.025230 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-istio-data\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.029160 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.025583 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-workload-certs\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.029160 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.025637 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9720d030-7806-479f-840c-38401180d116-istio-podinfo\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.029160 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.025680 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h66db\" (UniqueName: \"kubernetes.io/projected/9720d030-7806-479f-840c-38401180d116-kube-api-access-h66db\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.029160 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.025710 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-workload-socket\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.029160 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.025806 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9720d030-7806-479f-840c-38401180d116-istiod-ca-cert\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.029160 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.025843 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-credential-socket\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.029160 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.025894 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9720d030-7806-479f-840c-38401180d116-istio-token\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.029160 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.026019 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-istio-envoy\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.129168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.127420 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-workload-certs\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.129168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.127478 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9720d030-7806-479f-840c-38401180d116-istio-podinfo\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.129168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.127517 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h66db\" (UniqueName: \"kubernetes.io/projected/9720d030-7806-479f-840c-38401180d116-kube-api-access-h66db\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.129168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.127546 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-workload-socket\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.129168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.127610 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9720d030-7806-479f-840c-38401180d116-istiod-ca-cert\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.129168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.127644 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-credential-socket\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.129168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.127689 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9720d030-7806-479f-840c-38401180d116-istio-token\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.129168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.127807 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-istio-envoy\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.129168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.127930 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-istio-data\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.129168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.128463 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-istio-data\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.129168 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.128710 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-workload-certs\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.133106 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.133074 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9720d030-7806-479f-840c-38401180d116-istiod-ca-cert\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.134537 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.133837 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-workload-socket\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.134657 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.134261 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-credential-socket\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.139177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.139104 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-istio-envoy\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.196573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.196290 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9720d030-7806-479f-840c-38401180d116-istio-token\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.198671 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.198641 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9720d030-7806-479f-840c-38401180d116-istio-podinfo\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.201485 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.201460 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6"] Oct 30 16:53:57.202087 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.202058 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h66db\" (UniqueName: \"kubernetes.io/projected/9720d030-7806-479f-840c-38401180d116-kube-api-access-h66db\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.205812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.205522 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a130e2-zcpv6"] Oct 30 16:53:57.320085 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.317470 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:57.428854 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.425026 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:57.496428 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.496142 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:53:57.683142 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.682437 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/970bb829-3799-40ba-a22d-f1e1ae061631-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/970bb829-3799-40ba-a22d-f1e1ae061631-istiod-ca-cert\") pod \"970bb829-3799-40ba-a22d-f1e1ae061631\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " Oct 30 16:53:57.683142 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.682516 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-workload-certs\") pod \"970bb829-3799-40ba-a22d-f1e1ae061631\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " Oct 30 16:53:57.683142 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.682647 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/970bb829-3799-40ba-a22d-f1e1ae061631-kube-api-access-r8tk5\" (UniqueName: \"kubernetes.io/projected/970bb829-3799-40ba-a22d-f1e1ae061631-kube-api-access-r8tk5\") pod \"970bb829-3799-40ba-a22d-f1e1ae061631\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " Oct 30 16:53:57.683142 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.682736 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/970bb829-3799-40ba-a22d-f1e1ae061631-istio-token\" (UniqueName: \"kubernetes.io/projected/970bb829-3799-40ba-a22d-f1e1ae061631-istio-token\") pod \"970bb829-3799-40ba-a22d-f1e1ae061631\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " Oct 30 16:53:57.683142 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.683082 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-workload-certs" pod "970bb829-3799-40ba-a22d-f1e1ae061631" (UID: "970bb829-3799-40ba-a22d-f1e1ae061631"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:57.683743 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.683435 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/970bb829-3799-40ba-a22d-f1e1ae061631-istiod-ca-cert" pod "970bb829-3799-40ba-a22d-f1e1ae061631" (UID: "970bb829-3799-40ba-a22d-f1e1ae061631"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:53:57.683945 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.683922 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-istio-envoy\") pod \"970bb829-3799-40ba-a22d-f1e1ae061631\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " Oct 30 16:53:57.684087 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.684072 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-workload-socket\") pod \"970bb829-3799-40ba-a22d-f1e1ae061631\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " Oct 30 16:53:57.684233 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.684217 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-credential-socket\") pod \"970bb829-3799-40ba-a22d-f1e1ae061631\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " Oct 30 16:53:57.684439 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.684406 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-istio-data\") pod \"970bb829-3799-40ba-a22d-f1e1ae061631\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " Oct 30 16:53:57.684710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.684686 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-workload-socket" pod "970bb829-3799-40ba-a22d-f1e1ae061631" (UID: "970bb829-3799-40ba-a22d-f1e1ae061631"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:57.685423 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.685399 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/970bb829-3799-40ba-a22d-f1e1ae061631-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/970bb829-3799-40ba-a22d-f1e1ae061631-istio-podinfo\") pod \"970bb829-3799-40ba-a22d-f1e1ae061631\" (UID: \"970bb829-3799-40ba-a22d-f1e1ae061631\") " Oct 30 16:53:57.686562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.685441 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-istio-data" pod "970bb829-3799-40ba-a22d-f1e1ae061631" (UID: "970bb829-3799-40ba-a22d-f1e1ae061631"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:57.686562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.685589 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-credential-socket" pod "970bb829-3799-40ba-a22d-f1e1ae061631" (UID: "970bb829-3799-40ba-a22d-f1e1ae061631"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:57.686562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.686450 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970bb829-3799-40ba-a22d-f1e1ae061631-kube-api-access-r8tk5" pod "970bb829-3799-40ba-a22d-f1e1ae061631" (UID: "970bb829-3799-40ba-a22d-f1e1ae061631"). InnerVolumeSpecName "kube-api-access-r8tk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:57.686794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.686756 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970bb829-3799-40ba-a22d-f1e1ae061631-istio-token" pod "970bb829-3799-40ba-a22d-f1e1ae061631" (UID: "970bb829-3799-40ba-a22d-f1e1ae061631"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:57.686958 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.686939 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:53:57.687133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.687116 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/970bb829-3799-40ba-a22d-f1e1ae061631-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:53:57.687288 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.687273 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:53:57.688055 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.688029 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/970bb829-3799-40ba-a22d-f1e1ae061631-istio-podinfo" pod "970bb829-3799-40ba-a22d-f1e1ae061631" (UID: "970bb829-3799-40ba-a22d-f1e1ae061631"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:53:57.688731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.688643 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-istio-envoy" pod "970bb829-3799-40ba-a22d-f1e1ae061631" (UID: "970bb829-3799-40ba-a22d-f1e1ae061631"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:57.788834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.788797 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/970bb829-3799-40ba-a22d-f1e1ae061631-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:53:57.788834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.788830 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8tk5\" (UniqueName: \"kubernetes.io/projected/970bb829-3799-40ba-a22d-f1e1ae061631-kube-api-access-r8tk5\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:53:57.788834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.788843 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/970bb829-3799-40ba-a22d-f1e1ae061631-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:53:57.789085 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.788852 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:53:57.789085 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.788860 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:53:57.789085 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.788874 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/970bb829-3799-40ba-a22d-f1e1ae061631-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:53:57.804732 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:53:57.804703 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9720d030_7806_479f_840c_38401180d116.slice/crio-8dbe6c25b2a4a30141489e50375bcb4f6a768f5b92ef5f7cd1b336a377c71178 WatchSource:0}: Error finding container 8dbe6c25b2a4a30141489e50375bcb4f6a768f5b92ef5f7cd1b336a377c71178: Status 404 returned error can't find the container with id 8dbe6c25b2a4a30141489e50375bcb4f6a768f5b92ef5f7cd1b336a377c71178 Oct 30 16:53:57.805564 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.805540 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g"] Oct 30 16:53:57.807775 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.807731 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:53:57.807851 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.807810 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:53:57.807851 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.807841 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:53:57.871606 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:57.871221 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:58.132615 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:58.132577 6202 generic.go:356] "Generic (PLEG): container finished" podID="970bb829-3799-40ba-a22d-f1e1ae061631" containerID="8a39b3ba925f37d64a88a7b2a4e699344740cdb0deb9aa169baeec6a394c820e" exitCode=0 Oct 30 16:53:58.132884 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:58.132860 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" Oct 30 16:53:58.137161 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:58.132780 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" event={"ID":"970bb829-3799-40ba-a22d-f1e1ae061631","Type":"ContainerDied","Data":"8a39b3ba925f37d64a88a7b2a4e699344740cdb0deb9aa169baeec6a394c820e"} Oct 30 16:53:58.137292 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:58.137210 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r" event={"ID":"970bb829-3799-40ba-a22d-f1e1ae061631","Type":"ContainerDied","Data":"a8ae791ae5e4596f7ca85497917f11211546d9ea851f843c4cad8d01e433d871"} Oct 30 16:53:58.137292 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:58.137248 6202 scope.go:122] "RemoveContainer" containerID="8a39b3ba925f37d64a88a7b2a4e699344740cdb0deb9aa169baeec6a394c820e" Oct 30 16:53:58.139505 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:58.139478 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" event={"ID":"9720d030-7806-479f-840c-38401180d116","Type":"ContainerStarted","Data":"cf0bdbadc7890ddb7d870142156a3ced9ac0fbd159a375d3a80d5ff0a2e8529e"} Oct 30 16:53:58.139599 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:58.139530 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" event={"ID":"9720d030-7806-479f-840c-38401180d116","Type":"ContainerStarted","Data":"8dbe6c25b2a4a30141489e50375bcb4f6a768f5b92ef5f7cd1b336a377c71178"} Oct 30 16:53:58.158791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:58.158749 6202 scope.go:122] "RemoveContainer" containerID="8a39b3ba925f37d64a88a7b2a4e699344740cdb0deb9aa169baeec6a394c820e" Oct 30 16:53:58.159122 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:53:58.159101 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a39b3ba925f37d64a88a7b2a4e699344740cdb0deb9aa169baeec6a394c820e\": container with ID starting with 8a39b3ba925f37d64a88a7b2a4e699344740cdb0deb9aa169baeec6a394c820e not found: ID does not exist" containerID="8a39b3ba925f37d64a88a7b2a4e699344740cdb0deb9aa169baeec6a394c820e" Oct 30 16:53:58.159205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:58.159132 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a39b3ba925f37d64a88a7b2a4e699344740cdb0deb9aa169baeec6a394c820e"} err="failed to get container status \"8a39b3ba925f37d64a88a7b2a4e699344740cdb0deb9aa169baeec6a394c820e\": rpc error: code = NotFound desc = could not find container \"8a39b3ba925f37d64a88a7b2a4e699344740cdb0deb9aa169baeec6a394c820e\": container with ID starting with 8a39b3ba925f37d64a88a7b2a4e699344740cdb0deb9aa169baeec6a394c820e not found: ID does not exist" Oct 30 16:53:58.200495 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:58.200436 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" podStartSLOduration=2.2004177289999998 podStartE2EDuration="2.200417729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:58.195638994 +0000 UTC m=+1951.906093739" watchObservedRunningTime="2026-10-30 16:53:58.200417729 +0000 UTC m=+1951.910872474" Oct 30 16:53:58.237823 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:58.237265 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r"] Oct 30 16:53:58.243209 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:58.243175 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-2wr2r"] Oct 30 16:53:58.434524 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:58.434486 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 16:53:58.434805 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:58.434550 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" podUID="9720d030-7806-479f-840c-38401180d116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 16:53:58.440260 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:58.440225 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c849ce-219a-48c0-8646-071636dc498a" path="/var/lib/kubelet/pods/19c849ce-219a-48c0-8646-071636dc498a/volumes" Oct 30 16:53:58.440904 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:58.440881 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="970bb829-3799-40ba-a22d-f1e1ae061631" path="/var/lib/kubelet/pods/970bb829-3799-40ba-a22d-f1e1ae061631/volumes" Oct 30 16:53:58.441401 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:58.441379 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:59.438731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:59.438700 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:53:59.554400 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:53:59.554364 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:00.160536 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:00.160507 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:54:00.161538 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:00.161516 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:54:00.848751 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:00.848701 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:00.990314 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:00.990265 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:01.692582 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:01.692446 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:02.119750 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.119678 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 16:54:02.119750 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.119733 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" podUID="2b6965bb-5f5c-40ca-b098-9f67c7433381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 16:54:02.180936 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.180904 6202 generic.go:356] "Generic (PLEG): container finished" podID="2b6965bb-5f5c-40ca-b098-9f67c7433381" containerID="b59da60ecc764d40930bc401e680c533f6c1a12b9b5f935c27159b26ad047367" exitCode=0 Oct 30 16:54:02.181195 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.180981 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" event={"ID":"2b6965bb-5f5c-40ca-b098-9f67c7433381","Type":"ContainerDied","Data":"b59da60ecc764d40930bc401e680c533f6c1a12b9b5f935c27159b26ad047367"} Oct 30 16:54:02.181471 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.181454 6202 scope.go:122] "RemoveContainer" containerID="b59da60ecc764d40930bc401e680c533f6c1a12b9b5f935c27159b26ad047367" Oct 30 16:54:02.183345 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.183318 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:54:02.183427 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.183395 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:54:02.183464 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.183428 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:54:02.451226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.451183 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:02.573579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.573546 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk"] Oct 30 16:54:02.574797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.574760 6202 cpu_manager.go:414] "containerMap: removing container" podUID="970bb829-3799-40ba-a22d-f1e1ae061631" containerName="istio-proxy" Oct 30 16:54:02.574870 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.574800 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="970bb829-3799-40ba-a22d-f1e1ae061631" containerName="istio-proxy" Oct 30 16:54:02.575065 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.575048 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="970bb829-3799-40ba-a22d-f1e1ae061631" containerName="istio-proxy" Oct 30 16:54:02.578551 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.578528 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" Oct 30 16:54:02.583726 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.583677 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:02.597126 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.597090 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-7c4b84\"" Oct 30 16:54:02.604394 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.604366 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk"] Oct 30 16:54:02.771791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.769293 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkhpz\" (UniqueName: \"kubernetes.io/projected/be7ca116-75e7-468a-a943-4113a9fd941b-kube-api-access-xkhpz\") pod \"payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk\" (UID: \"be7ca116-75e7-468a-a943-4113a9fd941b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" Oct 30 16:54:02.771791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.769359 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be7ca116-75e7-468a-a943-4113a9fd941b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk\" (UID: \"be7ca116-75e7-468a-a943-4113a9fd941b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" Oct 30 16:54:02.808876 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.804729 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b"] Oct 30 16:54:02.823846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.815290 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" Oct 30 16:54:02.896792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.882557 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkhpz\" (UniqueName: \"kubernetes.io/projected/be7ca116-75e7-468a-a943-4113a9fd941b-kube-api-access-xkhpz\") pod \"payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk\" (UID: \"be7ca116-75e7-468a-a943-4113a9fd941b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" Oct 30 16:54:02.896792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.882612 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be7ca116-75e7-468a-a943-4113a9fd941b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk\" (UID: \"be7ca116-75e7-468a-a943-4113a9fd941b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" Oct 30 16:54:02.896792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.890885 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-7c4b84-dockercfg-hq6jl\"" Oct 30 16:54:02.898341 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.898217 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be7ca116-75e7-468a-a943-4113a9fd941b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk\" (UID: \"be7ca116-75e7-468a-a943-4113a9fd941b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" Oct 30 16:54:02.971754 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.971662 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b"] Oct 30 16:54:02.991066 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:02.990984 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkhpz\" (UniqueName: \"kubernetes.io/projected/be7ca116-75e7-468a-a943-4113a9fd941b-kube-api-access-xkhpz\") pod \"payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk\" (UID: \"be7ca116-75e7-468a-a943-4113a9fd941b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" Oct 30 16:54:03.042659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:03.042618 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwhlh\" (UniqueName: \"kubernetes.io/projected/e1cedd9a-5798-4463-a6b5-a8f199716991-kube-api-access-rwhlh\") pod \"payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b\" (UID: \"e1cedd9a-5798-4463-a6b5-a8f199716991\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" Oct 30 16:54:03.043260 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:03.043237 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1cedd9a-5798-4463-a6b5-a8f199716991-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b\" (UID: \"e1cedd9a-5798-4463-a6b5-a8f199716991\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" Oct 30 16:54:03.145342 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:03.145240 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1cedd9a-5798-4463-a6b5-a8f199716991-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b\" (UID: \"e1cedd9a-5798-4463-a6b5-a8f199716991\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" Oct 30 16:54:03.145856 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:03.145530 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwhlh\" (UniqueName: \"kubernetes.io/projected/e1cedd9a-5798-4463-a6b5-a8f199716991-kube-api-access-rwhlh\") pod \"payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b\" (UID: \"e1cedd9a-5798-4463-a6b5-a8f199716991\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" Oct 30 16:54:03.146113 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:03.146089 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1cedd9a-5798-4463-a6b5-a8f199716991-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b\" (UID: \"e1cedd9a-5798-4463-a6b5-a8f199716991\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" Oct 30 16:54:03.169247 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:03.169214 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwhlh\" (UniqueName: \"kubernetes.io/projected/e1cedd9a-5798-4463-a6b5-a8f199716991-kube-api-access-rwhlh\") pod \"payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b\" (UID: \"e1cedd9a-5798-4463-a6b5-a8f199716991\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" Oct 30 16:54:03.194126 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:03.194081 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" event={"ID":"2b6965bb-5f5c-40ca-b098-9f67c7433381","Type":"ContainerStarted","Data":"63374dbb7df5d4ccf540a17f8b99f53e02ca35ef4f36eb8ccd7e7e6e4fbc0acf"} Oct 30 16:54:03.194276 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:03.194173 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" Oct 30 16:54:03.323243 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:03.323208 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" Oct 30 16:54:03.487790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:03.487484 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:03.545050 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:03.540453 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk"] Oct 30 16:54:03.545445 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:54:03.545416 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7ca116_75e7_468a_a943_4113a9fd941b.slice/crio-05f146434887754822783f02bcae8489b9cf40325363a96bba7e76a978e1cbfb WatchSource:0}: Error finding container 05f146434887754822783f02bcae8489b9cf40325363a96bba7e76a978e1cbfb: Status 404 returned error can't find the container with id 05f146434887754822783f02bcae8489b9cf40325363a96bba7e76a978e1cbfb Oct 30 16:54:03.662809 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:03.662241 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b"] Oct 30 16:54:03.917316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:03.917237 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:54:03.917460 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:03.917384 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 16:54:03.917460 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:03.917425 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" podUID="2b6965bb-5f5c-40ca-b098-9f67c7433381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 16:54:04.204223 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:04.204130 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" event={"ID":"be7ca116-75e7-468a-a943-4113a9fd941b","Type":"ContainerStarted","Data":"6c45adfa6ff6a5b1f69168048a40d02c0fa60b9e3326f8949abc798a0f0a269c"} Oct 30 16:54:04.204223 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:04.204185 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" event={"ID":"be7ca116-75e7-468a-a943-4113a9fd941b","Type":"ContainerStarted","Data":"05f146434887754822783f02bcae8489b9cf40325363a96bba7e76a978e1cbfb"} Oct 30 16:54:04.204223 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:04.204208 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" Oct 30 16:54:04.206156 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:04.206129 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" event={"ID":"e1cedd9a-5798-4463-a6b5-a8f199716991","Type":"ContainerStarted","Data":"73d775ca11fc54629d826aaf35d3659503995932e95768b116a0c82aa3c8672d"} Oct 30 16:54:04.206306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:04.206169 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" event={"ID":"e1cedd9a-5798-4463-a6b5-a8f199716991","Type":"ContainerStarted","Data":"0629e1cf14918eaf36a0e58b492f95fcd5e5d1d6f32ff55b8be996a4b52251f7"} Oct 30 16:54:04.206306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:04.206211 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" Oct 30 16:54:04.266791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:04.251939 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" podStartSLOduration=2.2519188359999998 podStartE2EDuration="2.251918836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:04.25036205 +0000 UTC m=+1957.960816793" watchObservedRunningTime="2026-10-30 16:54:04.251918836 +0000 UTC m=+1957.962373582" Oct 30 16:54:04.291042 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:04.290978 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" podStartSLOduration=2.29096449 podStartE2EDuration="2.29096449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:04.290386279 +0000 UTC m=+1958.000841021" watchObservedRunningTime="2026-10-30 16:54:04.29096449 +0000 UTC m=+1958.001419232" Oct 30 16:54:04.918013 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:04.917982 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 16:54:04.918280 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:04.918042 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" podUID="2b6965bb-5f5c-40ca-b098-9f67c7433381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 16:54:05.748892 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:05.748836 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:05.919003 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:05.918970 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 16:54:05.919183 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:05.919025 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" podUID="2b6965bb-5f5c-40ca-b098-9f67c7433381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 16:54:06.918022 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:06.917982 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 16:54:06.918500 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:06.918042 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" podUID="2b6965bb-5f5c-40ca-b098-9f67c7433381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 16:54:07.511847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:07.511806 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:07.917727 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:07.917682 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 16:54:07.917959 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:07.917735 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" podUID="2b6965bb-5f5c-40ca-b098-9f67c7433381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 16:54:08.918290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:08.918257 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 16:54:08.918682 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:08.918312 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" podUID="2b6965bb-5f5c-40ca-b098-9f67c7433381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 16:54:09.449095 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:09.449042 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:09.923387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:09.923350 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:54:09.923976 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:09.923946 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:54:09.924758 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:09.924739 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 16:54:11.156001 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:11.155961 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:12.772237 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:12.772194 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:14.071097 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:54:14.071058 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:54:14.608650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:14.608616 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:15.217556 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:15.217529 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" Oct 30 16:54:15.218063 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:15.217988 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" Oct 30 16:54:16.234162 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:16.234111 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:17.717218 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:17.717171 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:19.635714 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:19.635683 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:21.208065 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:21.208035 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:21.472997 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:21.472886 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:21.650761 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:21.650724 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:23.065482 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:23.065248 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:24.682206 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:24.682170 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:26.220477 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:26.220416 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:27.166759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:27.166702 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:27.940277 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:27.940172 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:28.126296 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:28.126254 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:28.284692 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:28.284598 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:29.894406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:29.894329 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:31.474915 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:31.474868 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:31.537801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:31.520178 6202 scope.go:122] "RemoveContainer" containerID="1661f13d6ad63ce25ebacb6d57b6c874e7d90f2505083857d31f2394da060641" Oct 30 16:54:32.988217 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:32.988181 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:33.113876 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:33.113554 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:33.663448 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:33.663406 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:34.261718 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.261679 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:34.280367 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.280331 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd"] Oct 30 16:54:34.287240 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.287219 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd" Oct 30 16:54:34.356790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.320886 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:54:34.358998 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.330866 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:54:34.358998 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.330922 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:54:34.358998 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.338498 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:54:34.367907 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.367818 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd"] Oct 30 16:54:34.433663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.433629 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:34.457485 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.453838 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e571fb5-wpwpd\" (UID: \"70e2da8d-fe73-4d43-9db4-2a7acd7d9706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd" Oct 30 16:54:34.457485 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.453883 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55lkp\" (UniqueName: \"kubernetes.io/projected/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-kube-api-access-55lkp\") pod \"maas-api-revoke-sub-4e571fb5-wpwpd\" (UID: \"70e2da8d-fe73-4d43-9db4-2a7acd7d9706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd" Oct 30 16:54:34.457485 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.453926 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e571fb5-wpwpd\" (UID: \"70e2da8d-fe73-4d43-9db4-2a7acd7d9706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd" Oct 30 16:54:34.556376 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.556280 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e571fb5-wpwpd\" (UID: \"70e2da8d-fe73-4d43-9db4-2a7acd7d9706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd" Oct 30 16:54:34.556659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.556628 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55lkp\" (UniqueName: \"kubernetes.io/projected/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-kube-api-access-55lkp\") pod \"maas-api-revoke-sub-4e571fb5-wpwpd\" (UID: \"70e2da8d-fe73-4d43-9db4-2a7acd7d9706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd" Oct 30 16:54:34.556867 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.556826 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e571fb5-wpwpd\" (UID: \"70e2da8d-fe73-4d43-9db4-2a7acd7d9706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd" Oct 30 16:54:34.558940 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.558889 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e571fb5-wpwpd\" (UID: \"70e2da8d-fe73-4d43-9db4-2a7acd7d9706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd" Oct 30 16:54:34.600920 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.591235 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55lkp\" (UniqueName: \"kubernetes.io/projected/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-kube-api-access-55lkp\") pod \"maas-api-revoke-sub-4e571fb5-wpwpd\" (UID: \"70e2da8d-fe73-4d43-9db4-2a7acd7d9706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd" Oct 30 16:54:34.617975 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.615891 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e571fb5-wpwpd\" (UID: \"70e2da8d-fe73-4d43-9db4-2a7acd7d9706\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd" Oct 30 16:54:34.632791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:34.631455 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd" Oct 30 16:54:35.012391 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:35.012360 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd"] Oct 30 16:54:35.012912 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:54:35.012884 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e2da8d_fe73_4d43_9db4_2a7acd7d9706.slice/crio-ae8c6e06e0c09b6d073e4fb034960013450bb2f5ef3963a42e8fffcdca62cf02 WatchSource:0}: Error finding container ae8c6e06e0c09b6d073e4fb034960013450bb2f5ef3963a42e8fffcdca62cf02: Status 404 returned error can't find the container with id ae8c6e06e0c09b6d073e4fb034960013450bb2f5ef3963a42e8fffcdca62cf02 Oct 30 16:54:35.081790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:35.076516 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:35.694934 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:35.694896 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd" event={"ID":"70e2da8d-fe73-4d43-9db4-2a7acd7d9706","Type":"ContainerStarted","Data":"a1d8db655965566d92b3ac4d43003fbd35d49c82dace75769e2e65d1aba800bc"} Oct 30 16:54:35.695486 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:35.694969 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd" event={"ID":"70e2da8d-fe73-4d43-9db4-2a7acd7d9706","Type":"ContainerStarted","Data":"ae8c6e06e0c09b6d073e4fb034960013450bb2f5ef3963a42e8fffcdca62cf02"} Oct 30 16:54:35.742082 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:35.741956 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd" podStartSLOduration=1.74193416 podStartE2EDuration="1.74193416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:35.738419301 +0000 UTC m=+1989.448874047" watchObservedRunningTime="2026-10-30 16:54:35.74193416 +0000 UTC m=+1989.452388904" Oct 30 16:54:36.668516 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:36.668435 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:36.733621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:36.733580 6202 generic.go:356] "Generic (PLEG): container finished" podID="70e2da8d-fe73-4d43-9db4-2a7acd7d9706" containerID="a1d8db655965566d92b3ac4d43003fbd35d49c82dace75769e2e65d1aba800bc" exitCode=0 Oct 30 16:54:36.733621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:36.733632 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd" event={"ID":"70e2da8d-fe73-4d43-9db4-2a7acd7d9706","Type":"ContainerDied","Data":"a1d8db655965566d92b3ac4d43003fbd35d49c82dace75769e2e65d1aba800bc"} Oct 30 16:54:37.990532 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:37.990497 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd" Oct 30 16:54:38.052188 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.050780 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-kube-api-access-55lkp\" (UniqueName: \"kubernetes.io/projected/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-kube-api-access-55lkp\") pod \"70e2da8d-fe73-4d43-9db4-2a7acd7d9706\" (UID: \"70e2da8d-fe73-4d43-9db4-2a7acd7d9706\") " Oct 30 16:54:38.052188 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.051050 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-maas-api-xks-ca\") pod \"70e2da8d-fe73-4d43-9db4-2a7acd7d9706\" (UID: \"70e2da8d-fe73-4d43-9db4-2a7acd7d9706\") " Oct 30 16:54:38.052188 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.051170 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-maas-api-openshift-ca\") pod \"70e2da8d-fe73-4d43-9db4-2a7acd7d9706\" (UID: \"70e2da8d-fe73-4d43-9db4-2a7acd7d9706\") " Oct 30 16:54:38.068789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.052988 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-maas-api-openshift-ca" pod "70e2da8d-fe73-4d43-9db4-2a7acd7d9706" (UID: "70e2da8d-fe73-4d43-9db4-2a7acd7d9706"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:54:38.069537 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.069502 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-kube-api-access-55lkp" pod "70e2da8d-fe73-4d43-9db4-2a7acd7d9706" (UID: "70e2da8d-fe73-4d43-9db4-2a7acd7d9706"). InnerVolumeSpecName "kube-api-access-55lkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:54:38.073004 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.072972 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-maas-api-xks-ca" pod "70e2da8d-fe73-4d43-9db4-2a7acd7d9706" (UID: "70e2da8d-fe73-4d43-9db4-2a7acd7d9706"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:54:38.154079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.154046 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:54:38.154079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.154076 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:54:38.154079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.154086 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55lkp\" (UniqueName: \"kubernetes.io/projected/70e2da8d-fe73-4d43-9db4-2a7acd7d9706-kube-api-access-55lkp\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:54:38.286588 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.286521 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:38.565847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.565805 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:38.706631 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.706540 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:38.765097 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.765060 6202 generic.go:356] "Generic (PLEG): container finished" podID="323c4d64-a380-4e62-9779-1a1a87ae3308" containerID="381811ac503f75277f36a76ce038e7b9d3cc91bcf5273179879969a09538c8c7" exitCode=0 Oct 30 16:54:38.765309 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.765146 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" event={"ID":"323c4d64-a380-4e62-9779-1a1a87ae3308","Type":"ContainerDied","Data":"381811ac503f75277f36a76ce038e7b9d3cc91bcf5273179879969a09538c8c7"} Oct 30 16:54:38.765675 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.765653 6202 scope.go:122] "RemoveContainer" containerID="381811ac503f75277f36a76ce038e7b9d3cc91bcf5273179879969a09538c8c7" Oct 30 16:54:38.767150 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.767131 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd" Oct 30 16:54:38.767254 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.767162 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd" event={"ID":"70e2da8d-fe73-4d43-9db4-2a7acd7d9706","Type":"ContainerDied","Data":"ae8c6e06e0c09b6d073e4fb034960013450bb2f5ef3963a42e8fffcdca62cf02"} Oct 30 16:54:38.767254 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.767207 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae8c6e06e0c09b6d073e4fb034960013450bb2f5ef3963a42e8fffcdca62cf02" Oct 30 16:54:38.768189 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.768148 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:54:38.768283 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.768247 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:54:38.768334 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:38.768318 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:54:39.026205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:39.026128 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:54:39.781342 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:39.781008 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" event={"ID":"323c4d64-a380-4e62-9779-1a1a87ae3308","Type":"ContainerStarted","Data":"dde14991980aa79145cacb4f5218fd017677b3d0af5c194ae9937a3467003515"} Oct 30 16:54:40.016150 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.016106 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:40.168150 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.168114 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:40.169507 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.169477 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986"] Oct 30 16:54:40.170422 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.170405 6202 cpu_manager.go:414] "containerMap: removing container" podUID="70e2da8d-fe73-4d43-9db4-2a7acd7d9706" containerName="revoke-keys" Oct 30 16:54:40.170491 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.170423 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e2da8d-fe73-4d43-9db4-2a7acd7d9706" containerName="revoke-keys" Oct 30 16:54:40.170603 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.170593 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e2da8d-fe73-4d43-9db4-2a7acd7d9706" containerName="revoke-keys" Oct 30 16:54:40.173601 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.173584 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986" Oct 30 16:54:40.182226 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.182198 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986"] Oct 30 16:54:40.189188 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.189164 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f92caf\"" Oct 30 16:54:40.214800 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.214752 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v"] Oct 30 16:54:40.218478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.218458 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" Oct 30 16:54:40.234988 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.234959 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f92caf-dockercfg-xqfhn\"" Oct 30 16:54:40.302798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.301492 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v"] Oct 30 16:54:40.434947 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.397568 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:40.450790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.437578 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkbb2\" (UniqueName: \"kubernetes.io/projected/4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e-kube-api-access-qkbb2\") pod \"payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v\" (UID: \"4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" Oct 30 16:54:40.450790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.437671 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r24zx\" (UniqueName: \"kubernetes.io/projected/7d0e1b07-3b0f-4baf-bc98-005127e74f31-kube-api-access-r24zx\") pod \"payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986\" (UID: \"7d0e1b07-3b0f-4baf-bc98-005127e74f31\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986" Oct 30 16:54:40.450790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.437808 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0e1b07-3b0f-4baf-bc98-005127e74f31-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986\" (UID: \"7d0e1b07-3b0f-4baf-bc98-005127e74f31\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986" Oct 30 16:54:40.450790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.437989 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v\" (UID: \"4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" Oct 30 16:54:40.715134 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.715034 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v\" (UID: \"4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" Oct 30 16:54:40.717730 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.717701 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkbb2\" (UniqueName: \"kubernetes.io/projected/4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e-kube-api-access-qkbb2\") pod \"payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v\" (UID: \"4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" Oct 30 16:54:40.718162 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.718135 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r24zx\" (UniqueName: \"kubernetes.io/projected/7d0e1b07-3b0f-4baf-bc98-005127e74f31-kube-api-access-r24zx\") pod \"payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986\" (UID: \"7d0e1b07-3b0f-4baf-bc98-005127e74f31\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986" Oct 30 16:54:40.719600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.719577 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0e1b07-3b0f-4baf-bc98-005127e74f31-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986\" (UID: \"7d0e1b07-3b0f-4baf-bc98-005127e74f31\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986" Oct 30 16:54:40.722931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.722905 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0e1b07-3b0f-4baf-bc98-005127e74f31-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986\" (UID: \"7d0e1b07-3b0f-4baf-bc98-005127e74f31\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986" Oct 30 16:54:40.746595 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.745921 6202 patch_prober.go:29] interesting pod/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 16:54:40.746595 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.745996 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" podUID="323c4d64-a380-4e62-9779-1a1a87ae3308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 16:54:40.748401 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.748300 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v\" (UID: \"4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" Oct 30 16:54:40.812835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.804371 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:40.812835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.812092 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r24zx\" (UniqueName: \"kubernetes.io/projected/7d0e1b07-3b0f-4baf-bc98-005127e74f31-kube-api-access-r24zx\") pod \"payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986\" (UID: \"7d0e1b07-3b0f-4baf-bc98-005127e74f31\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986" Oct 30 16:54:40.819462 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.819429 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkbb2\" (UniqueName: \"kubernetes.io/projected/4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e-kube-api-access-qkbb2\") pod \"payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v\" (UID: \"4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" Oct 30 16:54:40.822302 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.822275 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:54:40.836817 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.836615 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" Oct 30 16:54:40.933082 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:40.932760 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:41.103145 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:41.103100 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986" Oct 30 16:54:41.271312 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:41.271159 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v"] Oct 30 16:54:41.475718 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:41.475686 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986"] Oct 30 16:54:41.475874 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:54:41.475854 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0e1b07_3b0f_4baf_bc98_005127e74f31.slice/crio-7ba3b337529bbece690fd2bbdfcf160caaf2d125ee2f25c5f5fa550dd319434d WatchSource:0}: Error finding container 7ba3b337529bbece690fd2bbdfcf160caaf2d125ee2f25c5f5fa550dd319434d: Status 404 returned error can't find the container with id 7ba3b337529bbece690fd2bbdfcf160caaf2d125ee2f25c5f5fa550dd319434d Oct 30 16:54:41.600837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:41.600343 6202 patch_prober.go:29] interesting pod/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 16:54:41.600837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:41.600407 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" podUID="323c4d64-a380-4e62-9779-1a1a87ae3308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 16:54:41.832756 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:41.832723 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" event={"ID":"4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e","Type":"ContainerStarted","Data":"c5e0a07769c0bbe69a12e96456ea392d9520705924ee1357c30206eff0195aca"} Oct 30 16:54:41.832953 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:41.832805 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" event={"ID":"4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e","Type":"ContainerStarted","Data":"9af67f606229102b2f8add6d466080d75a1f3f75af9ea4fe551c2a4565c5d409"} Oct 30 16:54:41.832953 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:41.832838 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" Oct 30 16:54:41.835146 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:41.835115 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986" event={"ID":"7d0e1b07-3b0f-4baf-bc98-005127e74f31","Type":"ContainerStarted","Data":"d0eaef2c4f7f51f35e9c14194b41cbaa14965663bde83415f3fd2fd2e917156e"} Oct 30 16:54:41.835297 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:41.835168 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986" event={"ID":"7d0e1b07-3b0f-4baf-bc98-005127e74f31","Type":"ContainerStarted","Data":"7ba3b337529bbece690fd2bbdfcf160caaf2d125ee2f25c5f5fa550dd319434d"} Oct 30 16:54:41.835297 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:41.835193 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986" Oct 30 16:54:41.894561 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:41.894513 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" podStartSLOduration=1.894497992 podStartE2EDuration="1.894497992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:41.889522003 +0000 UTC m=+1995.599976746" watchObservedRunningTime="2026-10-30 16:54:41.894497992 +0000 UTC m=+1995.604952735" Oct 30 16:54:41.933174 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:41.933127 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986" podStartSLOduration=1.933102009 podStartE2EDuration="1.933102009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:41.930507289 +0000 UTC m=+1995.640962031" watchObservedRunningTime="2026-10-30 16:54:41.933102009 +0000 UTC m=+1995.643556749" Oct 30 16:54:42.094598 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:42.094518 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:42.600458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:42.600423 6202 patch_prober.go:29] interesting pod/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 16:54:42.600978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:42.600483 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" podUID="323c4d64-a380-4e62-9779-1a1a87ae3308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 16:54:43.600321 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:43.600282 6202 patch_prober.go:29] interesting pod/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 16:54:43.600509 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:43.600342 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" podUID="323c4d64-a380-4e62-9779-1a1a87ae3308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 16:54:43.783902 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:43.783863 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:44.094159 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:54:44.094117 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:54:44.600296 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:44.600264 6202 patch_prober.go:29] interesting pod/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 16:54:44.600476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:44.600344 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" podUID="323c4d64-a380-4e62-9779-1a1a87ae3308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 16:54:45.318862 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:45.318821 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:45.450557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:45.450471 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:45.605483 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:45.605449 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:54:45.606317 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:45.606293 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:54:45.607458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:45.607435 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 16:54:45.913853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:45.913734 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf"] Oct 30 16:54:45.917653 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:45.917634 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf" Oct 30 16:54:45.933378 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:45.933349 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:54:45.933496 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:45.933474 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:54:45.933568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:45.933519 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:54:45.933568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:45.933529 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:54:45.938204 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:45.938181 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf"] Oct 30 16:54:45.954250 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:45.953935 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23364ade-95d2-4603-bc01-aec45b1c4c56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edc52165-k4kdf\" (UID: \"23364ade-95d2-4603-bc01-aec45b1c4c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf" Oct 30 16:54:45.954250 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:45.953996 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwxsp\" (UniqueName: \"kubernetes.io/projected/23364ade-95d2-4603-bc01-aec45b1c4c56-kube-api-access-xwxsp\") pod \"maas-api-revoke-sub-edc52165-k4kdf\" (UID: \"23364ade-95d2-4603-bc01-aec45b1c4c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf" Oct 30 16:54:45.954250 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:45.954051 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23364ade-95d2-4603-bc01-aec45b1c4c56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edc52165-k4kdf\" (UID: \"23364ade-95d2-4603-bc01-aec45b1c4c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf" Oct 30 16:54:46.055968 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:46.055927 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23364ade-95d2-4603-bc01-aec45b1c4c56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edc52165-k4kdf\" (UID: \"23364ade-95d2-4603-bc01-aec45b1c4c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf" Oct 30 16:54:46.056169 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:46.055990 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwxsp\" (UniqueName: \"kubernetes.io/projected/23364ade-95d2-4603-bc01-aec45b1c4c56-kube-api-access-xwxsp\") pod \"maas-api-revoke-sub-edc52165-k4kdf\" (UID: \"23364ade-95d2-4603-bc01-aec45b1c4c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf" Oct 30 16:54:46.056169 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:46.056044 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23364ade-95d2-4603-bc01-aec45b1c4c56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edc52165-k4kdf\" (UID: \"23364ade-95d2-4603-bc01-aec45b1c4c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf" Oct 30 16:54:46.056938 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:46.056908 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23364ade-95d2-4603-bc01-aec45b1c4c56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edc52165-k4kdf\" (UID: \"23364ade-95d2-4603-bc01-aec45b1c4c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf" Oct 30 16:54:46.059422 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:46.059396 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23364ade-95d2-4603-bc01-aec45b1c4c56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edc52165-k4kdf\" (UID: \"23364ade-95d2-4603-bc01-aec45b1c4c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf" Oct 30 16:54:46.080506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:46.080472 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwxsp\" (UniqueName: \"kubernetes.io/projected/23364ade-95d2-4603-bc01-aec45b1c4c56-kube-api-access-xwxsp\") pod \"maas-api-revoke-sub-edc52165-k4kdf\" (UID: \"23364ade-95d2-4603-bc01-aec45b1c4c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf" Oct 30 16:54:46.234222 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:46.234134 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf" Oct 30 16:54:46.547691 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:54:46.547660 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23364ade_95d2_4603_bc01_aec45b1c4c56.slice/crio-c1d9704ca517b819832a5be4e07346b8f8a8ea9daacf7cd641b05977958529ac WatchSource:0}: Error finding container c1d9704ca517b819832a5be4e07346b8f8a8ea9daacf7cd641b05977958529ac: Status 404 returned error can't find the container with id c1d9704ca517b819832a5be4e07346b8f8a8ea9daacf7cd641b05977958529ac Oct 30 16:54:46.548143 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:46.548089 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf"] Oct 30 16:54:46.921339 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:46.921296 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf" event={"ID":"23364ade-95d2-4603-bc01-aec45b1c4c56","Type":"ContainerStarted","Data":"c1d9704ca517b819832a5be4e07346b8f8a8ea9daacf7cd641b05977958529ac"} Oct 30 16:54:47.089969 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:47.089933 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:47.933246 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:47.933163 6202 generic.go:356] "Generic (PLEG): container finished" podID="23364ade-95d2-4603-bc01-aec45b1c4c56" containerID="dbb1c351fa2afcdb6a90414cfa5f868102d346e6112f4ffd8604e15070c2cc53" exitCode=0 Oct 30 16:54:47.933596 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:47.933245 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf" event={"ID":"23364ade-95d2-4603-bc01-aec45b1c4c56","Type":"ContainerDied","Data":"dbb1c351fa2afcdb6a90414cfa5f868102d346e6112f4ffd8604e15070c2cc53"} Oct 30 16:54:48.759692 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:48.759640 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:49.130264 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:49.130231 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf" Oct 30 16:54:49.207382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:49.207342 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23364ade-95d2-4603-bc01-aec45b1c4c56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23364ade-95d2-4603-bc01-aec45b1c4c56-maas-api-openshift-ca\") pod \"23364ade-95d2-4603-bc01-aec45b1c4c56\" (UID: \"23364ade-95d2-4603-bc01-aec45b1c4c56\") " Oct 30 16:54:49.207560 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:49.207460 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23364ade-95d2-4603-bc01-aec45b1c4c56-kube-api-access-xwxsp\" (UniqueName: \"kubernetes.io/projected/23364ade-95d2-4603-bc01-aec45b1c4c56-kube-api-access-xwxsp\") pod \"23364ade-95d2-4603-bc01-aec45b1c4c56\" (UID: \"23364ade-95d2-4603-bc01-aec45b1c4c56\") " Oct 30 16:54:49.207643 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:49.207626 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23364ade-95d2-4603-bc01-aec45b1c4c56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23364ade-95d2-4603-bc01-aec45b1c4c56-maas-api-xks-ca\") pod \"23364ade-95d2-4603-bc01-aec45b1c4c56\" (UID: \"23364ade-95d2-4603-bc01-aec45b1c4c56\") " Oct 30 16:54:49.207724 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:49.207689 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23364ade-95d2-4603-bc01-aec45b1c4c56-maas-api-openshift-ca" pod "23364ade-95d2-4603-bc01-aec45b1c4c56" (UID: "23364ade-95d2-4603-bc01-aec45b1c4c56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:54:49.208441 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:49.208410 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23364ade-95d2-4603-bc01-aec45b1c4c56-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:54:49.210029 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:49.210000 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23364ade-95d2-4603-bc01-aec45b1c4c56-kube-api-access-xwxsp" pod "23364ade-95d2-4603-bc01-aec45b1c4c56" (UID: "23364ade-95d2-4603-bc01-aec45b1c4c56"). InnerVolumeSpecName "kube-api-access-xwxsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:54:49.210149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:49.210089 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23364ade-95d2-4603-bc01-aec45b1c4c56-maas-api-xks-ca" pod "23364ade-95d2-4603-bc01-aec45b1c4c56" (UID: "23364ade-95d2-4603-bc01-aec45b1c4c56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:54:49.310467 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:49.310426 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwxsp\" (UniqueName: \"kubernetes.io/projected/23364ade-95d2-4603-bc01-aec45b1c4c56-kube-api-access-xwxsp\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:54:49.310467 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:49.310464 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23364ade-95d2-4603-bc01-aec45b1c4c56-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:54:49.953191 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:49.953160 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf" Oct 30 16:54:49.953458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:49.953194 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf" event={"ID":"23364ade-95d2-4603-bc01-aec45b1c4c56","Type":"ContainerDied","Data":"c1d9704ca517b819832a5be4e07346b8f8a8ea9daacf7cd641b05977958529ac"} Oct 30 16:54:49.953458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:49.953258 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1d9704ca517b819832a5be4e07346b8f8a8ea9daacf7cd641b05977958529ac" Oct 30 16:54:50.237149 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:50.237069 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:50.354991 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:50.354953 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:50.482492 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:50.482458 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:51.997005 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:51.996970 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:52.861373 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:52.861338 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986" Oct 30 16:54:52.861710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:52.861674 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" Oct 30 16:54:53.754142 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:53.754103 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:55.407249 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:55.407211 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:57.043049 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:57.043010 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:58.729894 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:54:58.729855 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:00.324544 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:00.324495 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:01.929979 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:01.929941 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:03.755329 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:03.755293 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:05.282478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:05.282399 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:06.931522 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:06.931469 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:09.115315 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:09.115259 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:55:09.146949 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:09.146882 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:55:09.169501 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:09.169459 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:09.434024 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:09.433923 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:09.666797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:09.650196 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 16:55:09.928896 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:09.928851 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj"] Oct 30 16:55:09.935050 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:09.934955 6202 cpu_manager.go:414] "containerMap: removing container" podUID="23364ade-95d2-4603-bc01-aec45b1c4c56" containerName="revoke-keys" Oct 30 16:55:09.935287 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:09.935268 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23364ade-95d2-4603-bc01-aec45b1c4c56" containerName="revoke-keys" Oct 30 16:55:09.935652 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:09.935638 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="23364ade-95d2-4603-bc01-aec45b1c4c56" containerName="revoke-keys" Oct 30 16:55:09.985011 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:09.984975 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj"] Oct 30 16:55:09.985376 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:09.985354 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj" Oct 30 16:55:10.001674 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:10.001647 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:55:10.015209 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:10.015186 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:55:10.015408 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:10.015394 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:55:10.015568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:10.015556 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:55:10.025535 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:10.025476 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj\" (UID: \"0e18e65b-ed87-44e6-a59c-d6ae3960aacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj" Oct 30 16:55:10.025679 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:10.025595 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj\" (UID: \"0e18e65b-ed87-44e6-a59c-d6ae3960aacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj" Oct 30 16:55:10.025679 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:10.025641 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js7qq\" (UniqueName: \"kubernetes.io/projected/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-kube-api-access-js7qq\") pod \"maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj\" (UID: \"0e18e65b-ed87-44e6-a59c-d6ae3960aacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj" Oct 30 16:55:10.126834 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:10.126792 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj\" (UID: \"0e18e65b-ed87-44e6-a59c-d6ae3960aacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj" Oct 30 16:55:10.127363 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:10.126912 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj\" (UID: \"0e18e65b-ed87-44e6-a59c-d6ae3960aacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj" Oct 30 16:55:10.127363 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:10.126953 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js7qq\" (UniqueName: \"kubernetes.io/projected/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-kube-api-access-js7qq\") pod \"maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj\" (UID: \"0e18e65b-ed87-44e6-a59c-d6ae3960aacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj" Oct 30 16:55:10.128353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:10.128324 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj\" (UID: \"0e18e65b-ed87-44e6-a59c-d6ae3960aacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj" Oct 30 16:55:10.131480 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:10.131445 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj\" (UID: \"0e18e65b-ed87-44e6-a59c-d6ae3960aacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj" Oct 30 16:55:10.158314 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:10.158247 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js7qq\" (UniqueName: \"kubernetes.io/projected/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-kube-api-access-js7qq\") pod \"maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj\" (UID: \"0e18e65b-ed87-44e6-a59c-d6ae3960aacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj" Oct 30 16:55:10.329329 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:10.329288 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj" Oct 30 16:55:10.835872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:10.833350 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj"] Oct 30 16:55:11.112257 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:11.112202 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:11.361793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:11.361278 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj" event={"ID":"0e18e65b-ed87-44e6-a59c-d6ae3960aacf","Type":"ContainerStarted","Data":"8bef94cf0799d374d74bcf6468d1e63f52ce065321cfe27530b68164b96167a4"} Oct 30 16:55:12.409318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:12.409279 6202 generic.go:356] "Generic (PLEG): container finished" podID="0e18e65b-ed87-44e6-a59c-d6ae3960aacf" containerID="9bb9461572ce3c78001e17323b934618d651b9b0f116399ec909b09bc3f0a69e" exitCode=0 Oct 30 16:55:12.416284 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:12.416252 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj" event={"ID":"0e18e65b-ed87-44e6-a59c-d6ae3960aacf","Type":"ContainerDied","Data":"9bb9461572ce3c78001e17323b934618d651b9b0f116399ec909b09bc3f0a69e"} Oct 30 16:55:12.961792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:12.959140 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:13.814575 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:13.814548 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj" Oct 30 16:55:13.916641 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:13.916602 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-maas-api-openshift-ca\") pod \"0e18e65b-ed87-44e6-a59c-d6ae3960aacf\" (UID: \"0e18e65b-ed87-44e6-a59c-d6ae3960aacf\") " Oct 30 16:55:13.916888 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:13.916712 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-kube-api-access-js7qq\" (UniqueName: \"kubernetes.io/projected/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-kube-api-access-js7qq\") pod \"0e18e65b-ed87-44e6-a59c-d6ae3960aacf\" (UID: \"0e18e65b-ed87-44e6-a59c-d6ae3960aacf\") " Oct 30 16:55:13.916963 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:13.916884 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-maas-api-xks-ca\") pod \"0e18e65b-ed87-44e6-a59c-d6ae3960aacf\" (UID: \"0e18e65b-ed87-44e6-a59c-d6ae3960aacf\") " Oct 30 16:55:13.917821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:13.917757 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-maas-api-openshift-ca" pod "0e18e65b-ed87-44e6-a59c-d6ae3960aacf" (UID: "0e18e65b-ed87-44e6-a59c-d6ae3960aacf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:55:13.920028 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:13.919999 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-maas-api-xks-ca" pod "0e18e65b-ed87-44e6-a59c-d6ae3960aacf" (UID: "0e18e65b-ed87-44e6-a59c-d6ae3960aacf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:55:13.923323 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:13.923291 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-kube-api-access-js7qq" pod "0e18e65b-ed87-44e6-a59c-d6ae3960aacf" (UID: "0e18e65b-ed87-44e6-a59c-d6ae3960aacf"). InnerVolumeSpecName "kube-api-access-js7qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:14.019452 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:14.019419 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:14.019452 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:14.019451 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:14.019665 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:14.019466 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js7qq\" (UniqueName: \"kubernetes.io/projected/0e18e65b-ed87-44e6-a59c-d6ae3960aacf-kube-api-access-js7qq\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:14.107440 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:55:14.107397 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:55:14.430676 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:14.430648 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj" Oct 30 16:55:14.430884 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:14.430679 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj" event={"ID":"0e18e65b-ed87-44e6-a59c-d6ae3960aacf","Type":"ContainerDied","Data":"8bef94cf0799d374d74bcf6468d1e63f52ce065321cfe27530b68164b96167a4"} Oct 30 16:55:14.430884 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:14.430749 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bef94cf0799d374d74bcf6468d1e63f52ce065321cfe27530b68164b96167a4" Oct 30 16:55:14.520807 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:14.520757 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:16.081474 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:16.081429 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:17.774835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:17.751813 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:18.479725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:18.479622 6202 generic.go:356] "Generic (PLEG): container finished" podID="c7320e48-7c58-40b0-bd5b-03b9f4bb551f" containerID="d579cd1a477fd869c51061a833073b281c5b2e206e1911192dfc00a4835ac8b1" exitCode=0 Oct 30 16:55:18.479725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:18.479698 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" event={"ID":"c7320e48-7c58-40b0-bd5b-03b9f4bb551f","Type":"ContainerDied","Data":"d579cd1a477fd869c51061a833073b281c5b2e206e1911192dfc00a4835ac8b1"} Oct 30 16:55:18.480356 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:18.480337 6202 scope.go:122] "RemoveContainer" containerID="d579cd1a477fd869c51061a833073b281c5b2e206e1911192dfc00a4835ac8b1" Oct 30 16:55:18.483152 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:18.483117 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:55:18.483232 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:18.483213 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:55:18.483281 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:18.483260 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:55:19.435384 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:19.435342 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:19.519031 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:19.518995 6202 generic.go:356] "Generic (PLEG): container finished" podID="318e8454-0cf7-485e-b774-9ce1b368cac6" containerID="fcaf838b4520c9a9d6431c3843ebbe8f76a404c6e31aa532650a80b032f843c7" exitCode=0 Oct 30 16:55:19.519222 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:19.519083 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" event={"ID":"318e8454-0cf7-485e-b774-9ce1b368cac6","Type":"ContainerDied","Data":"fcaf838b4520c9a9d6431c3843ebbe8f76a404c6e31aa532650a80b032f843c7"} Oct 30 16:55:19.522540 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:19.522015 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" event={"ID":"c7320e48-7c58-40b0-bd5b-03b9f4bb551f","Type":"ContainerStarted","Data":"a93ec556596a6308c936c62b119db9f06348514a9c1c8723762d98d97f4707eb"} Oct 30 16:55:19.523368 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:19.523344 6202 scope.go:122] "RemoveContainer" containerID="fcaf838b4520c9a9d6431c3843ebbe8f76a404c6e31aa532650a80b032f843c7" Oct 30 16:55:19.526358 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:19.526314 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:55:19.526448 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:19.526405 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:55:19.526448 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:19.526440 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:55:20.030631 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:20.030546 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw"] Oct 30 16:55:20.091134 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:20.083478 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn"] Oct 30 16:55:20.194857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:20.189286 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:20.194857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:20.193152 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:55:20.201108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:20.200571 6202 patch_prober.go:29] interesting pod/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Oct 30 16:55:20.201108 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:20.200628 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" podUID="c7320e48-7c58-40b0-bd5b-03b9f4bb551f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Oct 30 16:55:20.326670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:20.326635 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:20.533318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:20.533280 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" event={"ID":"318e8454-0cf7-485e-b774-9ce1b368cac6","Type":"ContainerStarted","Data":"bb4aa15c5ed737c36f2aa3d71aca44d0126b5debb4eac2aa352fde9ff2989172"} Oct 30 16:55:20.765035 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:20.764879 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:55:21.196264 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:21.196222 6202 patch_prober.go:29] interesting pod/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Oct 30 16:55:21.196565 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:21.196535 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" podUID="c7320e48-7c58-40b0-bd5b-03b9f4bb551f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Oct 30 16:55:21.313943 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:21.313911 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:55:21.315353 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:21.315325 6202 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 16:55:21.315423 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:21.315386 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" podUID="318e8454-0cf7-485e-b774-9ce1b368cac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 16:55:21.472033 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:21.464343 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:22.196790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:22.196227 6202 patch_prober.go:29] interesting pod/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Oct 30 16:55:22.196790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:22.196288 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" podUID="c7320e48-7c58-40b0-bd5b-03b9f4bb551f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Oct 30 16:55:22.313061 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:22.313024 6202 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 16:55:22.313252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:22.313080 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" podUID="318e8454-0cf7-485e-b774-9ce1b368cac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 16:55:23.193727 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:23.193241 6202 patch_prober.go:29] interesting pod/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Oct 30 16:55:23.193727 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:23.193308 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" podUID="c7320e48-7c58-40b0-bd5b-03b9f4bb551f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Oct 30 16:55:23.283793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:23.281335 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:23.313233 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:23.313185 6202 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 16:55:23.313593 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:23.313250 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" podUID="318e8454-0cf7-485e-b774-9ce1b368cac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 16:55:23.404311 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:23.404271 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:23.592855 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:23.579948 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:55:24.209919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:24.207507 6202 patch_prober.go:29] interesting pod/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Oct 30 16:55:24.209919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:24.207588 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" podUID="c7320e48-7c58-40b0-bd5b-03b9f4bb551f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Oct 30 16:55:24.313134 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:24.313099 6202 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 16:55:24.313644 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:24.313617 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" podUID="318e8454-0cf7-485e-b774-9ce1b368cac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 16:55:25.030930 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:25.030881 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" podUID="4e4a192c-bbec-4ec7-98c8-378faf14b730" containerName="payload-processing" containerID="cri-o://7e9d2f40a50c1f5ab0e3783f71432301bfb88d55447cc5427dccaa71058bd56a" gracePeriod=25 Oct 30 16:55:25.090653 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:25.090607 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" podUID="1dcea868-43f0-4503-9f06-04bb9db08e9b" containerName="payload-pre-processing" containerID="cri-o://a5e9283318dd1a34b10715db8bd65156a41ca264f1de1b3a20bcf5516f24490c" gracePeriod=25 Oct 30 16:55:25.211799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:25.206490 6202 patch_prober.go:29] interesting pod/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Oct 30 16:55:25.211799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:25.206554 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" podUID="c7320e48-7c58-40b0-bd5b-03b9f4bb551f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Oct 30 16:55:25.284861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:25.274436 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:25.317801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:25.314169 6202 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 16:55:25.317801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:25.314275 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" podUID="318e8454-0cf7-485e-b774-9ce1b368cac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 16:55:26.198358 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:26.198326 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:55:26.199410 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:26.199389 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 16:55:26.313866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:26.313828 6202 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 16:55:26.314044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:26.313892 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" podUID="318e8454-0cf7-485e-b774-9ce1b368cac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 16:55:26.860332 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:26.860295 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:27.318759 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:27.318677 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:55:27.319711 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:27.319685 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 16:55:28.400362 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.400326 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g"] Oct 30 16:55:28.401202 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.401186 6202 cpu_manager.go:414] "containerMap: removing container" podUID="0e18e65b-ed87-44e6-a59c-d6ae3960aacf" containerName="revoke-keys" Oct 30 16:55:28.401260 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.401202 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e18e65b-ed87-44e6-a59c-d6ae3960aacf" containerName="revoke-keys" Oct 30 16:55:28.401377 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.401368 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e18e65b-ed87-44e6-a59c-d6ae3960aacf" containerName="revoke-keys" Oct 30 16:55:28.404377 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.404346 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" Oct 30 16:55:28.420153 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.419994 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 16:55:28.432093 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.432067 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5jjc6\"" Oct 30 16:55:28.432276 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.432130 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:55:28.432276 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.432164 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f92caf-serving-cert\"" Oct 30 16:55:28.432429 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.432210 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:55:28.441601 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.441560 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g"] Oct 30 16:55:28.503312 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.503280 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:28.524380 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.524353 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkj9r\" (UniqueName: \"kubernetes.io/projected/73af482c-6523-4268-9ddf-8c4eeefd5f32-kube-api-access-nkj9r\") pod \"maas-api-e2e-inf-b-f92caf-68bf68464c-5899g\" (UID: \"73af482c-6523-4268-9ddf-8c4eeefd5f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" Oct 30 16:55:28.524530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.524518 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73af482c-6523-4268-9ddf-8c4eeefd5f32-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f92caf-68bf68464c-5899g\" (UID: \"73af482c-6523-4268-9ddf-8c4eeefd5f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" Oct 30 16:55:28.524576 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.524550 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73af482c-6523-4268-9ddf-8c4eeefd5f32-metrics-certs\") pod \"maas-api-e2e-inf-b-f92caf-68bf68464c-5899g\" (UID: \"73af482c-6523-4268-9ddf-8c4eeefd5f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" Oct 30 16:55:28.524625 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.524600 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73af482c-6523-4268-9ddf-8c4eeefd5f32-maas-api-tls\") pod \"maas-api-e2e-inf-b-f92caf-68bf68464c-5899g\" (UID: \"73af482c-6523-4268-9ddf-8c4eeefd5f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" Oct 30 16:55:28.620830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.620710 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:28.634917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.626891 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73af482c-6523-4268-9ddf-8c4eeefd5f32-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f92caf-68bf68464c-5899g\" (UID: \"73af482c-6523-4268-9ddf-8c4eeefd5f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" Oct 30 16:55:28.634917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.626958 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73af482c-6523-4268-9ddf-8c4eeefd5f32-metrics-certs\") pod \"maas-api-e2e-inf-b-f92caf-68bf68464c-5899g\" (UID: \"73af482c-6523-4268-9ddf-8c4eeefd5f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" Oct 30 16:55:28.634917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.627036 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73af482c-6523-4268-9ddf-8c4eeefd5f32-maas-api-tls\") pod \"maas-api-e2e-inf-b-f92caf-68bf68464c-5899g\" (UID: \"73af482c-6523-4268-9ddf-8c4eeefd5f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" Oct 30 16:55:28.634917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.627228 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkj9r\" (UniqueName: \"kubernetes.io/projected/73af482c-6523-4268-9ddf-8c4eeefd5f32-kube-api-access-nkj9r\") pod \"maas-api-e2e-inf-b-f92caf-68bf68464c-5899g\" (UID: \"73af482c-6523-4268-9ddf-8c4eeefd5f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" Oct 30 16:55:28.636289 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.635901 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73af482c-6523-4268-9ddf-8c4eeefd5f32-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f92caf-68bf68464c-5899g\" (UID: \"73af482c-6523-4268-9ddf-8c4eeefd5f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" Oct 30 16:55:28.640792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.637863 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73af482c-6523-4268-9ddf-8c4eeefd5f32-metrics-certs\") pod \"maas-api-e2e-inf-b-f92caf-68bf68464c-5899g\" (UID: \"73af482c-6523-4268-9ddf-8c4eeefd5f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" Oct 30 16:55:28.643382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.643354 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73af482c-6523-4268-9ddf-8c4eeefd5f32-maas-api-tls\") pod \"maas-api-e2e-inf-b-f92caf-68bf68464c-5899g\" (UID: \"73af482c-6523-4268-9ddf-8c4eeefd5f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" Oct 30 16:55:28.662152 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.650841 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkj9r\" (UniqueName: \"kubernetes.io/projected/73af482c-6523-4268-9ddf-8c4eeefd5f32-kube-api-access-nkj9r\") pod \"maas-api-e2e-inf-b-f92caf-68bf68464c-5899g\" (UID: \"73af482c-6523-4268-9ddf-8c4eeefd5f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" Oct 30 16:55:28.724697 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.724671 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" Oct 30 16:55:28.739248 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:28.739213 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:29.029627 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:55:29.029596 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73af482c_6523_4268_9ddf_8c4eeefd5f32.slice/crio-b7aa138796a15a8d7e123995b28d4a52c9d43d55f40824c2749662e0723759b5 WatchSource:0}: Error finding container b7aa138796a15a8d7e123995b28d4a52c9d43d55f40824c2749662e0723759b5: Status 404 returned error can't find the container with id b7aa138796a15a8d7e123995b28d4a52c9d43d55f40824c2749662e0723759b5 Oct 30 16:55:29.031300 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:29.031280 6202 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:55:29.031797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:29.031735 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g"] Oct 30 16:55:29.774575 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:29.774533 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" event={"ID":"73af482c-6523-4268-9ddf-8c4eeefd5f32","Type":"ContainerStarted","Data":"b7aa138796a15a8d7e123995b28d4a52c9d43d55f40824c2749662e0723759b5"} Oct 30 16:55:30.162311 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:30.162275 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:30.172161 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:30.172124 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" podUID="1dcea868-43f0-4503-9f06-04bb9db08e9b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:55:31.183287 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:31.183247 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" podUID="4e4a192c-bbec-4ec7-98c8-378faf14b730" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:55:31.641753 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:31.641724 6202 scope.go:122] "RemoveContainer" containerID="a802f2d979da69e617142f7ca05762877607a7c8db1027a8e07ea49106864bd8" Oct 30 16:55:31.661849 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:31.661818 6202 scope.go:122] "RemoveContainer" containerID="3befbad68f60aba6d6aaeb4103fc58e3f5f2430438055520b1f236284282dc28" Oct 30 16:55:31.672275 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:31.672251 6202 scope.go:122] "RemoveContainer" containerID="87ccf2487430f92873e24ccd858631799acb3e30db5ed76f3b6c23fae9e2daa8" Oct 30 16:55:31.788431 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:31.788389 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:31.798028 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:31.797993 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" event={"ID":"73af482c-6523-4268-9ddf-8c4eeefd5f32","Type":"ContainerStarted","Data":"0a86dd533395e5be7f50b6355cb92a01d747da5176c2d72ee44ccc42676f6f1e"} Oct 30 16:55:31.798178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:31.798078 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" Oct 30 16:55:31.848798 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:31.848224 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" podStartSLOduration=1.205918631 podStartE2EDuration="3.848203593s" totalImagesPullingTime="2.642284962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:31.846613735 +0000 UTC m=+2045.557068519" watchObservedRunningTime="2026-10-30 16:55:31.848203593 +0000 UTC m=+2045.558658338" Oct 30 16:55:32.456089 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:32.456052 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:33.112701 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:33.112665 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5"] Oct 30 16:55:33.117887 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:33.117859 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5" Oct 30 16:55:33.137732 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:33.137703 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5"] Oct 30 16:55:33.144009 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:33.143969 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:55:33.144180 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:33.144098 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:55:33.169636 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:33.169607 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92v4x\" (UniqueName: \"kubernetes.io/projected/91a6b947-98de-4432-bdb4-8e70f3719e69-kube-api-access-92v4x\") pod \"maas-api-revoke-sub-65a767a9-sbvr5\" (UID: \"91a6b947-98de-4432-bdb4-8e70f3719e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5" Oct 30 16:55:33.169861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:33.169645 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a6b947-98de-4432-bdb4-8e70f3719e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a767a9-sbvr5\" (UID: \"91a6b947-98de-4432-bdb4-8e70f3719e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5" Oct 30 16:55:33.169961 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:33.169858 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a6b947-98de-4432-bdb4-8e70f3719e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a767a9-sbvr5\" (UID: \"91a6b947-98de-4432-bdb4-8e70f3719e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5" Oct 30 16:55:33.309795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:33.287914 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92v4x\" (UniqueName: \"kubernetes.io/projected/91a6b947-98de-4432-bdb4-8e70f3719e69-kube-api-access-92v4x\") pod \"maas-api-revoke-sub-65a767a9-sbvr5\" (UID: \"91a6b947-98de-4432-bdb4-8e70f3719e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5" Oct 30 16:55:33.309795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:33.287971 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a6b947-98de-4432-bdb4-8e70f3719e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a767a9-sbvr5\" (UID: \"91a6b947-98de-4432-bdb4-8e70f3719e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5" Oct 30 16:55:33.309795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:33.288092 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a6b947-98de-4432-bdb4-8e70f3719e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a767a9-sbvr5\" (UID: \"91a6b947-98de-4432-bdb4-8e70f3719e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5" Oct 30 16:55:33.309795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:33.289227 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a6b947-98de-4432-bdb4-8e70f3719e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a767a9-sbvr5\" (UID: \"91a6b947-98de-4432-bdb4-8e70f3719e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5" Oct 30 16:55:33.357828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:33.349881 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92v4x\" (UniqueName: \"kubernetes.io/projected/91a6b947-98de-4432-bdb4-8e70f3719e69-kube-api-access-92v4x\") pod \"maas-api-revoke-sub-65a767a9-sbvr5\" (UID: \"91a6b947-98de-4432-bdb4-8e70f3719e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5" Oct 30 16:55:33.381846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:33.361307 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a6b947-98de-4432-bdb4-8e70f3719e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a767a9-sbvr5\" (UID: \"91a6b947-98de-4432-bdb4-8e70f3719e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5" Oct 30 16:55:33.436792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:33.436033 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5" Oct 30 16:55:33.636036 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:33.631082 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:33.838134 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:33.838100 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5"] Oct 30 16:55:33.838340 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:55:33.838310 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a6b947_98de_4432_bdb4_8e70f3719e69.slice/crio-40173cda3c7e0a9bc6201d2c1f863bbdd58784e21537872faea5e3b1929e9261 WatchSource:0}: Error finding container 40173cda3c7e0a9bc6201d2c1f863bbdd58784e21537872faea5e3b1929e9261: Status 404 returned error can't find the container with id 40173cda3c7e0a9bc6201d2c1f863bbdd58784e21537872faea5e3b1929e9261 Oct 30 16:55:34.828887 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:34.828851 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5" event={"ID":"91a6b947-98de-4432-bdb4-8e70f3719e69","Type":"ContainerStarted","Data":"00d9dfec16320a461356bac36b742f8624ab9805223e9447e20990b817ec0118"} Oct 30 16:55:34.829392 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:34.828921 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5" event={"ID":"91a6b947-98de-4432-bdb4-8e70f3719e69","Type":"ContainerStarted","Data":"40173cda3c7e0a9bc6201d2c1f863bbdd58784e21537872faea5e3b1929e9261"} Oct 30 16:55:34.871675 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:34.871614 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5" podStartSLOduration=1.8715951259999999 podStartE2EDuration="1.871595126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:34.868396905 +0000 UTC m=+2048.578851650" watchObservedRunningTime="2026-10-30 16:55:34.871595126 +0000 UTC m=+2048.582049873" Oct 30 16:55:35.035690 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:35.035656 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:35.503076 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:35.503044 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:35.839447 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:35.839410 6202 generic.go:356] "Generic (PLEG): container finished" podID="91a6b947-98de-4432-bdb4-8e70f3719e69" containerID="00d9dfec16320a461356bac36b742f8624ab9805223e9447e20990b817ec0118" exitCode=0 Oct 30 16:55:35.839881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:35.839488 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5" event={"ID":"91a6b947-98de-4432-bdb4-8e70f3719e69","Type":"ContainerDied","Data":"00d9dfec16320a461356bac36b742f8624ab9805223e9447e20990b817ec0118"} Oct 30 16:55:37.030549 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:37.030515 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg"] Oct 30 16:55:37.031042 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:37.030796 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" podUID="ec24f992-d313-4d6d-987d-98c4c8361df8" containerName="istio-proxy" containerID="cri-o://32bafbb010b706a8dc04ec9b5b727d70a5946780147d60e6acbd9924c36cde66" gracePeriod=30 Oct 30 16:55:37.043826 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:37.043147 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5" Oct 30 16:55:37.190878 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:37.190733 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91a6b947-98de-4432-bdb4-8e70f3719e69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a6b947-98de-4432-bdb4-8e70f3719e69-maas-api-openshift-ca\") pod \"91a6b947-98de-4432-bdb4-8e70f3719e69\" (UID: \"91a6b947-98de-4432-bdb4-8e70f3719e69\") " Oct 30 16:55:37.191064 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:37.191016 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91a6b947-98de-4432-bdb4-8e70f3719e69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a6b947-98de-4432-bdb4-8e70f3719e69-maas-api-xks-ca\") pod \"91a6b947-98de-4432-bdb4-8e70f3719e69\" (UID: \"91a6b947-98de-4432-bdb4-8e70f3719e69\") " Oct 30 16:55:37.191133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:37.191083 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a6b947-98de-4432-bdb4-8e70f3719e69-kube-api-access-92v4x\" (UniqueName: \"kubernetes.io/projected/91a6b947-98de-4432-bdb4-8e70f3719e69-kube-api-access-92v4x\") pod \"91a6b947-98de-4432-bdb4-8e70f3719e69\" (UID: \"91a6b947-98de-4432-bdb4-8e70f3719e69\") " Oct 30 16:55:37.192784 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:37.192734 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a6b947-98de-4432-bdb4-8e70f3719e69-maas-api-openshift-ca" pod "91a6b947-98de-4432-bdb4-8e70f3719e69" (UID: "91a6b947-98de-4432-bdb4-8e70f3719e69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:55:37.194659 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:37.194627 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a6b947-98de-4432-bdb4-8e70f3719e69-maas-api-xks-ca" pod "91a6b947-98de-4432-bdb4-8e70f3719e69" (UID: "91a6b947-98de-4432-bdb4-8e70f3719e69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:55:37.195235 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:37.195210 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a6b947-98de-4432-bdb4-8e70f3719e69-kube-api-access-92v4x" pod "91a6b947-98de-4432-bdb4-8e70f3719e69" (UID: "91a6b947-98de-4432-bdb4-8e70f3719e69"). InnerVolumeSpecName "kube-api-access-92v4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:37.279743 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:37.279708 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:37.294568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:37.294007 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a6b947-98de-4432-bdb4-8e70f3719e69-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:37.294568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:37.294041 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a6b947-98de-4432-bdb4-8e70f3719e69-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:37.294568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:37.294054 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92v4x\" (UniqueName: \"kubernetes.io/projected/91a6b947-98de-4432-bdb4-8e70f3719e69-kube-api-access-92v4x\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:37.816085 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:37.816014 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" Oct 30 16:55:37.861554 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:37.861526 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5" Oct 30 16:55:37.861777 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:37.861586 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5" event={"ID":"91a6b947-98de-4432-bdb4-8e70f3719e69","Type":"ContainerDied","Data":"40173cda3c7e0a9bc6201d2c1f863bbdd58784e21537872faea5e3b1929e9261"} Oct 30 16:55:37.861777 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:37.861647 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40173cda3c7e0a9bc6201d2c1f863bbdd58784e21537872faea5e3b1929e9261" Oct 30 16:55:38.589338 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:38.589302 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:39.139871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:39.139834 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:40.099342 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:40.099300 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:40.185550 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:40.183876 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" podUID="1dcea868-43f0-4503-9f06-04bb9db08e9b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:55:40.408726 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:40.408644 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:40.867576 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:40.867534 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:41.184680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:41.184566 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" podUID="4e4a192c-bbec-4ec7-98c8-378faf14b730" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:55:42.126853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.126823 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:55:42.127053 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.126873 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" podUID="ec24f992-d313-4d6d-987d-98c4c8361df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 16:55:42.327730 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.327704 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:55:42.458021 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.457982 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:42.488159 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.488124 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec24f992-d313-4d6d-987d-98c4c8361df8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec24f992-d313-4d6d-987d-98c4c8361df8-istiod-ca-cert\") pod \"ec24f992-d313-4d6d-987d-98c4c8361df8\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " Oct 30 16:55:42.488159 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.488163 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-token\" (UniqueName: \"kubernetes.io/projected/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-token\") pod \"ec24f992-d313-4d6d-987d-98c4c8361df8\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " Oct 30 16:55:42.488380 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.488194 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-data\") pod \"ec24f992-d313-4d6d-987d-98c4c8361df8\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " Oct 30 16:55:42.488380 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.488332 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec24f992-d313-4d6d-987d-98c4c8361df8-kube-api-access-bcd7t\" (UniqueName: \"kubernetes.io/projected/ec24f992-d313-4d6d-987d-98c4c8361df8-kube-api-access-bcd7t\") pod \"ec24f992-d313-4d6d-987d-98c4c8361df8\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " Oct 30 16:55:42.488380 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.488371 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-envoy\") pod \"ec24f992-d313-4d6d-987d-98c4c8361df8\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " Oct 30 16:55:42.488577 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.488544 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-workload-socket\") pod \"ec24f992-d313-4d6d-987d-98c4c8361df8\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " Oct 30 16:55:42.488650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.488552 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-data" pod "ec24f992-d313-4d6d-987d-98c4c8361df8" (UID: "ec24f992-d313-4d6d-987d-98c4c8361df8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:42.488650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.488604 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec24f992-d313-4d6d-987d-98c4c8361df8-istiod-ca-cert" pod "ec24f992-d313-4d6d-987d-98c4c8361df8" (UID: "ec24f992-d313-4d6d-987d-98c4c8361df8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:55:42.488650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.488614 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-credential-socket\") pod \"ec24f992-d313-4d6d-987d-98c4c8361df8\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " Oct 30 16:55:42.488650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.488650 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-workload-certs\") pod \"ec24f992-d313-4d6d-987d-98c4c8361df8\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " Oct 30 16:55:42.488923 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.488839 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-workload-socket" pod "ec24f992-d313-4d6d-987d-98c4c8361df8" (UID: "ec24f992-d313-4d6d-987d-98c4c8361df8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:42.488923 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.488862 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-credential-socket" pod "ec24f992-d313-4d6d-987d-98c4c8361df8" (UID: "ec24f992-d313-4d6d-987d-98c4c8361df8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:42.488923 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.488885 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-podinfo\") pod \"ec24f992-d313-4d6d-987d-98c4c8361df8\" (UID: \"ec24f992-d313-4d6d-987d-98c4c8361df8\") " Oct 30 16:55:42.489090 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.488975 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-workload-certs" pod "ec24f992-d313-4d6d-987d-98c4c8361df8" (UID: "ec24f992-d313-4d6d-987d-98c4c8361df8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:42.489804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.489780 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:42.489921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.489808 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:42.489921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.489823 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:42.489921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.489836 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec24f992-d313-4d6d-987d-98c4c8361df8-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:42.489921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.489849 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:42.490940 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.490909 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-token" pod "ec24f992-d313-4d6d-987d-98c4c8361df8" (UID: "ec24f992-d313-4d6d-987d-98c4c8361df8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:42.491094 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.491073 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-envoy" pod "ec24f992-d313-4d6d-987d-98c4c8361df8" (UID: "ec24f992-d313-4d6d-987d-98c4c8361df8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:55:42.491163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.491119 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec24f992-d313-4d6d-987d-98c4c8361df8-kube-api-access-bcd7t" pod "ec24f992-d313-4d6d-987d-98c4c8361df8" (UID: "ec24f992-d313-4d6d-987d-98c4c8361df8"). InnerVolumeSpecName "kube-api-access-bcd7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:42.491307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.491290 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-podinfo" pod "ec24f992-d313-4d6d-987d-98c4c8361df8" (UID: "ec24f992-d313-4d6d-987d-98c4c8361df8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:55:42.592301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.592265 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:42.592301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.592294 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:42.592301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.592303 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcd7t\" (UniqueName: \"kubernetes.io/projected/ec24f992-d313-4d6d-987d-98c4c8361df8-kube-api-access-bcd7t\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:42.592624 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.592314 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec24f992-d313-4d6d-987d-98c4c8361df8-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:42.916190 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.916103 6202 generic.go:356] "Generic (PLEG): container finished" podID="ec24f992-d313-4d6d-987d-98c4c8361df8" containerID="32bafbb010b706a8dc04ec9b5b727d70a5946780147d60e6acbd9924c36cde66" exitCode=0 Oct 30 16:55:42.916190 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.916168 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" Oct 30 16:55:42.916452 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.916192 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" event={"ID":"ec24f992-d313-4d6d-987d-98c4c8361df8","Type":"ContainerDied","Data":"32bafbb010b706a8dc04ec9b5b727d70a5946780147d60e6acbd9924c36cde66"} Oct 30 16:55:42.916452 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.916260 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg" event={"ID":"ec24f992-d313-4d6d-987d-98c4c8361df8","Type":"ContainerDied","Data":"c697988d07bd9e2d04a3baaa38806c768bed35c78552947c9e3ea5c473efc950"} Oct 30 16:55:42.916452 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.916289 6202 scope.go:122] "RemoveContainer" containerID="32bafbb010b706a8dc04ec9b5b727d70a5946780147d60e6acbd9924c36cde66" Oct 30 16:55:42.929395 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.929194 6202 scope.go:122] "RemoveContainer" containerID="32bafbb010b706a8dc04ec9b5b727d70a5946780147d60e6acbd9924c36cde66" Oct 30 16:55:42.929660 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:55:42.929564 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32bafbb010b706a8dc04ec9b5b727d70a5946780147d60e6acbd9924c36cde66\": container with ID starting with 32bafbb010b706a8dc04ec9b5b727d70a5946780147d60e6acbd9924c36cde66 not found: ID does not exist" containerID="32bafbb010b706a8dc04ec9b5b727d70a5946780147d60e6acbd9924c36cde66" Oct 30 16:55:42.929660 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.929599 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32bafbb010b706a8dc04ec9b5b727d70a5946780147d60e6acbd9924c36cde66"} err="failed to get container status \"32bafbb010b706a8dc04ec9b5b727d70a5946780147d60e6acbd9924c36cde66\": rpc error: code = NotFound desc = could not find container \"32bafbb010b706a8dc04ec9b5b727d70a5946780147d60e6acbd9924c36cde66\": container with ID starting with 32bafbb010b706a8dc04ec9b5b727d70a5946780147d60e6acbd9924c36cde66 not found: ID does not exist" Oct 30 16:55:42.984878 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.984841 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg"] Oct 30 16:55:42.996237 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:42.996203 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-l29jg"] Oct 30 16:55:44.108698 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:55:44.108656 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:55:44.178695 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:44.178657 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:44.414223 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:44.414134 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec24f992-d313-4d6d-987d-98c4c8361df8" path="/var/lib/kubelet/pods/ec24f992-d313-4d6d-987d-98c4c8361df8/volumes" Oct 30 16:55:45.431298 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:45.431270 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" Oct 30 16:55:45.481233 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:45.481207 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" Oct 30 16:55:45.548359 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:45.548265 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4a192c-bbec-4ec7-98c8-378faf14b730-kube-api-access-fm9v2\" (UniqueName: \"kubernetes.io/projected/4e4a192c-bbec-4ec7-98c8-378faf14b730-kube-api-access-fm9v2\") pod \"4e4a192c-bbec-4ec7-98c8-378faf14b730\" (UID: \"4e4a192c-bbec-4ec7-98c8-378faf14b730\") " Oct 30 16:55:45.548740 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:45.548716 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4a192c-bbec-4ec7-98c8-378faf14b730-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4a192c-bbec-4ec7-98c8-378faf14b730-plugins-config-volume\") pod \"4e4a192c-bbec-4ec7-98c8-378faf14b730\" (UID: \"4e4a192c-bbec-4ec7-98c8-378faf14b730\") " Oct 30 16:55:45.550462 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:45.549348 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4a192c-bbec-4ec7-98c8-378faf14b730-plugins-config-volume" pod "4e4a192c-bbec-4ec7-98c8-378faf14b730" (UID: "4e4a192c-bbec-4ec7-98c8-378faf14b730"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:55:45.550462 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:45.549594 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4a192c-bbec-4ec7-98c8-378faf14b730-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:45.553122 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:45.553098 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4a192c-bbec-4ec7-98c8-378faf14b730-kube-api-access-fm9v2" pod "4e4a192c-bbec-4ec7-98c8-378faf14b730" (UID: "4e4a192c-bbec-4ec7-98c8-378faf14b730"). InnerVolumeSpecName "kube-api-access-fm9v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:45.650734 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:45.650692 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dcea868-43f0-4503-9f06-04bb9db08e9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcea868-43f0-4503-9f06-04bb9db08e9b-plugins-config-volume\") pod \"1dcea868-43f0-4503-9f06-04bb9db08e9b\" (UID: \"1dcea868-43f0-4503-9f06-04bb9db08e9b\") " Oct 30 16:55:45.650974 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:45.650850 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dcea868-43f0-4503-9f06-04bb9db08e9b-kube-api-access-x7lb9\" (UniqueName: \"kubernetes.io/projected/1dcea868-43f0-4503-9f06-04bb9db08e9b-kube-api-access-x7lb9\") pod \"1dcea868-43f0-4503-9f06-04bb9db08e9b\" (UID: \"1dcea868-43f0-4503-9f06-04bb9db08e9b\") " Oct 30 16:55:45.651215 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:45.651188 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dcea868-43f0-4503-9f06-04bb9db08e9b-plugins-config-volume" pod "1dcea868-43f0-4503-9f06-04bb9db08e9b" (UID: "1dcea868-43f0-4503-9f06-04bb9db08e9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:55:45.651662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:45.651609 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm9v2\" (UniqueName: \"kubernetes.io/projected/4e4a192c-bbec-4ec7-98c8-378faf14b730-kube-api-access-fm9v2\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:45.651662 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:45.651634 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dcea868-43f0-4503-9f06-04bb9db08e9b-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:45.653349 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:45.653318 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dcea868-43f0-4503-9f06-04bb9db08e9b-kube-api-access-x7lb9" pod "1dcea868-43f0-4503-9f06-04bb9db08e9b" (UID: "1dcea868-43f0-4503-9f06-04bb9db08e9b"). InnerVolumeSpecName "kube-api-access-x7lb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:45.752950 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:45.752913 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7lb9\" (UniqueName: \"kubernetes.io/projected/1dcea868-43f0-4503-9f06-04bb9db08e9b-kube-api-access-x7lb9\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:55:45.770553 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:45.770519 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:46.036261 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.036227 6202 generic.go:356] "Generic (PLEG): container finished" podID="4e4a192c-bbec-4ec7-98c8-378faf14b730" containerID="7e9d2f40a50c1f5ab0e3783f71432301bfb88d55447cc5427dccaa71058bd56a" exitCode=0 Oct 30 16:55:46.036468 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.036271 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" Oct 30 16:55:46.036468 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.036277 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" event={"ID":"4e4a192c-bbec-4ec7-98c8-378faf14b730","Type":"ContainerDied","Data":"7e9d2f40a50c1f5ab0e3783f71432301bfb88d55447cc5427dccaa71058bd56a"} Oct 30 16:55:46.036468 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.036344 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw" event={"ID":"4e4a192c-bbec-4ec7-98c8-378faf14b730","Type":"ContainerDied","Data":"3f419a2527c65ae9de14b29a7d9a27096e1a62904ea9eb0e082a78af1ef7e078"} Oct 30 16:55:46.036468 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.036376 6202 scope.go:122] "RemoveContainer" containerID="7e9d2f40a50c1f5ab0e3783f71432301bfb88d55447cc5427dccaa71058bd56a" Oct 30 16:55:46.042781 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.038169 6202 generic.go:356] "Generic (PLEG): container finished" podID="1dcea868-43f0-4503-9f06-04bb9db08e9b" containerID="a5e9283318dd1a34b10715db8bd65156a41ca264f1de1b3a20bcf5516f24490c" exitCode=0 Oct 30 16:55:46.042781 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.038212 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" Oct 30 16:55:46.042781 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.038298 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" event={"ID":"1dcea868-43f0-4503-9f06-04bb9db08e9b","Type":"ContainerDied","Data":"a5e9283318dd1a34b10715db8bd65156a41ca264f1de1b3a20bcf5516f24490c"} Oct 30 16:55:46.042781 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.038371 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn" event={"ID":"1dcea868-43f0-4503-9f06-04bb9db08e9b","Type":"ContainerDied","Data":"ecf474154abd90e57e583c923f1bfe299fbb80ba2c68c4f3d1b27ed8e959ccba"} Oct 30 16:55:46.066509 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.066473 6202 scope.go:122] "RemoveContainer" containerID="7e9d2f40a50c1f5ab0e3783f71432301bfb88d55447cc5427dccaa71058bd56a" Oct 30 16:55:46.066911 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:55:46.066886 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e9d2f40a50c1f5ab0e3783f71432301bfb88d55447cc5427dccaa71058bd56a\": container with ID starting with 7e9d2f40a50c1f5ab0e3783f71432301bfb88d55447cc5427dccaa71058bd56a not found: ID does not exist" containerID="7e9d2f40a50c1f5ab0e3783f71432301bfb88d55447cc5427dccaa71058bd56a" Oct 30 16:55:46.067026 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.066917 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e9d2f40a50c1f5ab0e3783f71432301bfb88d55447cc5427dccaa71058bd56a"} err="failed to get container status \"7e9d2f40a50c1f5ab0e3783f71432301bfb88d55447cc5427dccaa71058bd56a\": rpc error: code = NotFound desc = could not find container \"7e9d2f40a50c1f5ab0e3783f71432301bfb88d55447cc5427dccaa71058bd56a\": container with ID starting with 7e9d2f40a50c1f5ab0e3783f71432301bfb88d55447cc5427dccaa71058bd56a not found: ID does not exist" Oct 30 16:55:46.067026 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.066936 6202 scope.go:122] "RemoveContainer" containerID="a5e9283318dd1a34b10715db8bd65156a41ca264f1de1b3a20bcf5516f24490c" Oct 30 16:55:46.078230 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.078203 6202 scope.go:122] "RemoveContainer" containerID="a5e9283318dd1a34b10715db8bd65156a41ca264f1de1b3a20bcf5516f24490c" Oct 30 16:55:46.078574 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:55:46.078547 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e9283318dd1a34b10715db8bd65156a41ca264f1de1b3a20bcf5516f24490c\": container with ID starting with a5e9283318dd1a34b10715db8bd65156a41ca264f1de1b3a20bcf5516f24490c not found: ID does not exist" containerID="a5e9283318dd1a34b10715db8bd65156a41ca264f1de1b3a20bcf5516f24490c" Oct 30 16:55:46.078787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.078585 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e9283318dd1a34b10715db8bd65156a41ca264f1de1b3a20bcf5516f24490c"} err="failed to get container status \"a5e9283318dd1a34b10715db8bd65156a41ca264f1de1b3a20bcf5516f24490c\": rpc error: code = NotFound desc = could not find container \"a5e9283318dd1a34b10715db8bd65156a41ca264f1de1b3a20bcf5516f24490c\": container with ID starting with a5e9283318dd1a34b10715db8bd65156a41ca264f1de1b3a20bcf5516f24490c not found: ID does not exist" Oct 30 16:55:46.106848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.106808 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn"] Oct 30 16:55:46.111265 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.111230 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-529bc7-744bd88bb7-2xcrn"] Oct 30 16:55:46.145848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.145811 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw"] Oct 30 16:55:46.149972 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.149948 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-529bc7-9578665b4-c7jxw"] Oct 30 16:55:46.415683 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.415650 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dcea868-43f0-4503-9f06-04bb9db08e9b" path="/var/lib/kubelet/pods/1dcea868-43f0-4503-9f06-04bb9db08e9b/volumes" Oct 30 16:55:46.416194 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:46.416178 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4a192c-bbec-4ec7-98c8-378faf14b730" path="/var/lib/kubelet/pods/4e4a192c-bbec-4ec7-98c8-378faf14b730/volumes" Oct 30 16:55:47.363713 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:47.363659 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:47.879357 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:47.879318 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:48.000271 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.000235 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:48.281314 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.281194 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:48.368845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.368806 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9"] Oct 30 16:55:48.370223 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.370199 6202 cpu_manager.go:414] "containerMap: removing container" podUID="1dcea868-43f0-4503-9f06-04bb9db08e9b" containerName="payload-pre-processing" Oct 30 16:55:48.370317 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.370227 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dcea868-43f0-4503-9f06-04bb9db08e9b" containerName="payload-pre-processing" Oct 30 16:55:48.370317 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.370249 6202 cpu_manager.go:414] "containerMap: removing container" podUID="ec24f992-d313-4d6d-987d-98c4c8361df8" containerName="istio-proxy" Oct 30 16:55:48.370317 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.370257 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec24f992-d313-4d6d-987d-98c4c8361df8" containerName="istio-proxy" Oct 30 16:55:48.370317 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.370272 6202 cpu_manager.go:414] "containerMap: removing container" podUID="91a6b947-98de-4432-bdb4-8e70f3719e69" containerName="revoke-keys" Oct 30 16:55:48.370317 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.370282 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a6b947-98de-4432-bdb4-8e70f3719e69" containerName="revoke-keys" Oct 30 16:55:48.370567 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.370339 6202 cpu_manager.go:414] "containerMap: removing container" podUID="4e4a192c-bbec-4ec7-98c8-378faf14b730" containerName="payload-processing" Oct 30 16:55:48.370567 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.370349 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4a192c-bbec-4ec7-98c8-378faf14b730" containerName="payload-processing" Oct 30 16:55:48.370567 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.370540 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4a192c-bbec-4ec7-98c8-378faf14b730" containerName="payload-processing" Oct 30 16:55:48.370675 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.370567 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dcea868-43f0-4503-9f06-04bb9db08e9b" containerName="payload-pre-processing" Oct 30 16:55:48.370675 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.370578 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec24f992-d313-4d6d-987d-98c4c8361df8" containerName="istio-proxy" Oct 30 16:55:48.370675 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.370590 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a6b947-98de-4432-bdb4-8e70f3719e69" containerName="revoke-keys" Oct 30 16:55:48.375501 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.375479 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.384537 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.384507 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x"] Oct 30 16:55:48.390043 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.390020 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.394175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.394152 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-857kc\" (UniqueName: \"kubernetes.io/projected/33abe911-bb76-44ba-a61e-6c8c87c95bb8-kube-api-access-857kc\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.394291 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.394191 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.394291 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.394219 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.394455 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.394394 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.394651 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.394632 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.394735 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.394688 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.394864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.394811 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.394928 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.394904 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.394992 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.394932 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.419177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.419145 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9"] Oct 30 16:55:48.419323 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.419212 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x"] Oct 30 16:55:48.496332 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.496296 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.496332 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.496333 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.496542 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.496357 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/051d25d4-9b2f-482e-a9f7-1c630eec241c-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.496649 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.496539 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.496649 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.496573 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.496649 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.496605 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.496761 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.496655 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.496761 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.496720 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-857kc\" (UniqueName: \"kubernetes.io/projected/33abe911-bb76-44ba-a61e-6c8c87c95bb8-kube-api-access-857kc\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.496761 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.496749 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.496989 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.496760 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.496989 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.496837 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.496989 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.496913 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.496989 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.496931 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.497173 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.497005 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.497173 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.497006 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.497173 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.497085 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.497318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.497210 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.497318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.497245 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.497318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.497271 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.497444 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.497425 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.497646 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.497570 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.497720 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.497705 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nv4n\" (UniqueName: \"kubernetes.io/projected/051d25d4-9b2f-482e-a9f7-1c630eec241c-kube-api-access-5nv4n\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.498070 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.498043 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.499209 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.499191 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.499289 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.499240 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.520197 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.520169 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.520318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.520306 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-857kc\" (UniqueName: \"kubernetes.io/projected/33abe911-bb76-44ba-a61e-6c8c87c95bb8-kube-api-access-857kc\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.598739 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.598708 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.598895 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.598824 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nv4n\" (UniqueName: \"kubernetes.io/projected/051d25d4-9b2f-482e-a9f7-1c630eec241c-kube-api-access-5nv4n\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.598951 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.598898 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/051d25d4-9b2f-482e-a9f7-1c630eec241c-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.598989 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.598961 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.599096 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.599077 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.599179 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.599121 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.599179 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.599145 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.599299 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.599217 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.599299 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.599255 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.599299 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.599276 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.599470 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.599421 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.599535 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.599492 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.599624 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.599605 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.599720 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.599657 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/051d25d4-9b2f-482e-a9f7-1c630eec241c-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.601616 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.601585 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.601825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.601800 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.622195 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.622168 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.622334 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.622316 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nv4n\" (UniqueName: \"kubernetes.io/projected/051d25d4-9b2f-482e-a9f7-1c630eec241c-kube-api-access-5nv4n\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.696170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.696144 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:48.707709 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.707685 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:48.784836 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:48.784760 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:49.182791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:49.180700 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:55:49.182791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:49.180828 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:55:49.182791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:49.180875 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:55:49.200037 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:49.192706 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:55:49.200037 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:49.192819 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:55:49.200037 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:49.192862 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:55:49.288835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:49.278891 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9"] Oct 30 16:55:49.295444 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:49.291902 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x"] Oct 30 16:55:49.640094 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:49.639974 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:49.855484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:49.855445 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj"] Oct 30 16:55:49.862283 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:49.862250 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-529bc7-r27mj"] Oct 30 16:55:49.865217 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:49.865187 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:49.980423 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:49.980329 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:50.212837 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:50.212794 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" event={"ID":"33abe911-bb76-44ba-a61e-6c8c87c95bb8","Type":"ContainerStarted","Data":"ea18800c7daca6729a6b9b53ff51414e4f1a738ee007693550e4254796bbb51b"} Oct 30 16:55:50.213039 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:50.212886 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" event={"ID":"33abe911-bb76-44ba-a61e-6c8c87c95bb8","Type":"ContainerStarted","Data":"80291eeadd4eec71104c462fd5a9c172e324ae4fe90b780437f21649b91ef75d"} Oct 30 16:55:50.215450 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:50.215407 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" event={"ID":"051d25d4-9b2f-482e-a9f7-1c630eec241c","Type":"ContainerStarted","Data":"ac831c619ea4ce9946bad503fcde433894b9b9af5a4ccb0c894c8ad233ee147b"} Oct 30 16:55:50.215602 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:50.215476 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" event={"ID":"051d25d4-9b2f-482e-a9f7-1c630eec241c","Type":"ContainerStarted","Data":"58274fa80b0a7f894fd0776a5441368c263c3e21e40730fb6569c0be574aec59"} Oct 30 16:55:50.266851 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:50.266550 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" podStartSLOduration=2.266532177 podStartE2EDuration="2.266532177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:50.265330525 +0000 UTC m=+2063.975785270" watchObservedRunningTime="2026-10-30 16:55:50.266532177 +0000 UTC m=+2063.976986923" Oct 30 16:55:50.332564 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:50.332488 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" podStartSLOduration=2.3324629630000002 podStartE2EDuration="2.332462963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:50.316296778 +0000 UTC m=+2064.026751548" watchObservedRunningTime="2026-10-30 16:55:50.332462963 +0000 UTC m=+2064.042917709" Oct 30 16:55:50.380835 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:50.379183 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:50.421202 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:50.420944 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e18e65b-ed87-44e6-a59c-d6ae3960aacf" path="/var/lib/kubelet/pods/0e18e65b-ed87-44e6-a59c-d6ae3960aacf/volumes" Oct 30 16:55:50.699012 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:50.698955 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:50.702828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:50.701446 6202 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 16:55:50.702828 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:50.701497 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" podUID="33abe911-bb76-44ba-a61e-6c8c87c95bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 16:55:50.707960 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:50.707926 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:50.709318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:50.709291 6202 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Oct 30 16:55:50.709442 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:50.709347 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" podUID="051d25d4-9b2f-482e-a9f7-1c630eec241c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Oct 30 16:55:51.702794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:51.702230 6202 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 16:55:51.702794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:51.702291 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" podUID="33abe911-bb76-44ba-a61e-6c8c87c95bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 16:55:51.708118 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:51.708078 6202 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Oct 30 16:55:51.708303 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:51.708137 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" podUID="051d25d4-9b2f-482e-a9f7-1c630eec241c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Oct 30 16:55:51.829317 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:51.829277 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:52.699822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:52.698169 6202 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 16:55:52.699822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:52.698233 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" podUID="33abe911-bb76-44ba-a61e-6c8c87c95bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 16:55:52.708718 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:52.708684 6202 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Oct 30 16:55:52.709075 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:52.708731 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" podUID="051d25d4-9b2f-482e-a9f7-1c630eec241c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Oct 30 16:55:53.227219 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:53.227183 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:53.351792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:53.348170 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:53.475702 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:53.475661 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:53.703182 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:53.703143 6202 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 16:55:53.703490 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:53.703459 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" podUID="33abe911-bb76-44ba-a61e-6c8c87c95bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 16:55:53.708155 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:53.708126 6202 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Oct 30 16:55:53.708330 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:53.708302 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" podUID="051d25d4-9b2f-482e-a9f7-1c630eec241c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Oct 30 16:55:54.697348 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:54.697307 6202 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 16:55:54.698034 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:54.697998 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" podUID="33abe911-bb76-44ba-a61e-6c8c87c95bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 16:55:54.708127 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:54.708099 6202 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Oct 30 16:55:54.708243 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:54.708135 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" podUID="051d25d4-9b2f-482e-a9f7-1c630eec241c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Oct 30 16:55:55.102215 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:55.102118 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:55.697318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:55.697250 6202 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 16:55:55.697722 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:55.697685 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" podUID="33abe911-bb76-44ba-a61e-6c8c87c95bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 16:55:55.711155 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:55.711123 6202 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Oct 30 16:55:55.711406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:55.711380 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" podUID="051d25d4-9b2f-482e-a9f7-1c630eec241c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Oct 30 16:55:56.013515 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:56.013424 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:56.702137 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:56.702107 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:56.702648 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:56.702621 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:56.703279 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:56.703258 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 16:55:56.713323 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:56.713298 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:56.713592 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:56.713573 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:56.714228 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:56.714207 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 16:55:57.824983 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:57.824948 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:58.434044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:58.434002 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:58.548126 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:58.548090 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:58.881498 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:58.881460 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd"] Oct 30 16:55:58.882075 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:58.881698 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" podUID="1f65a98a-1301-4d39-a40f-df19ec451712" containerName="istio-proxy" containerID="cri-o://9b0e8e1e1ed81010e4e9aeb1debbb374f13786c5259e9c340974190a659f39bf" gracePeriod=30 Oct 30 16:55:58.921846 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:58.921804 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:59.436640 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:59.436600 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2"] Oct 30 16:55:59.442992 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:59.442964 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2" Oct 30 16:55:59.462918 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:59.462887 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2"] Oct 30 16:55:59.488577 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:59.488547 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:55:59.488754 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:59.488619 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:55:59.524650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:59.524623 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbwfd\" (UniqueName: \"kubernetes.io/projected/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-kube-api-access-hbwfd\") pod \"maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2\" (UID: \"97be2811-cc69-483b-8dcd-f31ed5ec8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2" Oct 30 16:55:59.524951 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:59.524928 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2\" (UID: \"97be2811-cc69-483b-8dcd-f31ed5ec8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2" Oct 30 16:55:59.525102 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:59.525083 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2\" (UID: \"97be2811-cc69-483b-8dcd-f31ed5ec8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2" Oct 30 16:55:59.626049 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:59.626017 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbwfd\" (UniqueName: \"kubernetes.io/projected/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-kube-api-access-hbwfd\") pod \"maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2\" (UID: \"97be2811-cc69-483b-8dcd-f31ed5ec8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2" Oct 30 16:55:59.626220 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:59.626206 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2\" (UID: \"97be2811-cc69-483b-8dcd-f31ed5ec8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2" Oct 30 16:55:59.626292 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:59.626280 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2\" (UID: \"97be2811-cc69-483b-8dcd-f31ed5ec8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2" Oct 30 16:55:59.626974 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:59.626936 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2\" (UID: \"97be2811-cc69-483b-8dcd-f31ed5ec8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2" Oct 30 16:55:59.628710 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:59.628689 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2\" (UID: \"97be2811-cc69-483b-8dcd-f31ed5ec8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2" Oct 30 16:55:59.649551 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:59.649522 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbwfd\" (UniqueName: \"kubernetes.io/projected/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-kube-api-access-hbwfd\") pod \"maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2\" (UID: \"97be2811-cc69-483b-8dcd-f31ed5ec8e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2" Oct 30 16:55:59.764788 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:59.764694 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2" Oct 30 16:55:59.780714 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:55:59.780680 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:00.074902 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:56:00.074859 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97be2811_cc69_483b_8dcd_f31ed5ec8e5b.slice/crio-728fef87036f3eac7e81a608b821701d38285e499bc5b6f5c8076948fed8a0f9 WatchSource:0}: Error finding container 728fef87036f3eac7e81a608b821701d38285e499bc5b6f5c8076948fed8a0f9: Status 404 returned error can't find the container with id 728fef87036f3eac7e81a608b821701d38285e499bc5b6f5c8076948fed8a0f9 Oct 30 16:56:00.076941 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:00.076875 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2"] Oct 30 16:56:00.459690 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:00.459660 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2" event={"ID":"97be2811-cc69-483b-8dcd-f31ed5ec8e5b","Type":"ContainerStarted","Data":"728fef87036f3eac7e81a608b821701d38285e499bc5b6f5c8076948fed8a0f9"} Oct 30 16:56:01.482887 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:01.482844 6202 generic.go:356] "Generic (PLEG): container finished" podID="97be2811-cc69-483b-8dcd-f31ed5ec8e5b" containerID="695b9484dd3f46853901e1ee9a29504e46b2022bafbde711458e566d9481e5f4" exitCode=0 Oct 30 16:56:01.483382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:01.483081 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2" event={"ID":"97be2811-cc69-483b-8dcd-f31ed5ec8e5b","Type":"ContainerDied","Data":"695b9484dd3f46853901e1ee9a29504e46b2022bafbde711458e566d9481e5f4"} Oct 30 16:56:01.649315 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:01.649254 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:02.675989 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:02.675960 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2" Oct 30 16:56:02.779642 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:02.779546 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-kube-api-access-hbwfd\" (UniqueName: \"kubernetes.io/projected/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-kube-api-access-hbwfd\") pod \"97be2811-cc69-483b-8dcd-f31ed5ec8e5b\" (UID: \"97be2811-cc69-483b-8dcd-f31ed5ec8e5b\") " Oct 30 16:56:02.779642 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:02.779626 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-maas-api-openshift-ca\") pod \"97be2811-cc69-483b-8dcd-f31ed5ec8e5b\" (UID: \"97be2811-cc69-483b-8dcd-f31ed5ec8e5b\") " Oct 30 16:56:02.779944 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:02.779890 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-maas-api-xks-ca\") pod \"97be2811-cc69-483b-8dcd-f31ed5ec8e5b\" (UID: \"97be2811-cc69-483b-8dcd-f31ed5ec8e5b\") " Oct 30 16:56:02.780191 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:02.780160 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-maas-api-openshift-ca" pod "97be2811-cc69-483b-8dcd-f31ed5ec8e5b" (UID: "97be2811-cc69-483b-8dcd-f31ed5ec8e5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:56:02.780706 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:02.780678 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:02.782355 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:02.782322 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-maas-api-xks-ca" pod "97be2811-cc69-483b-8dcd-f31ed5ec8e5b" (UID: "97be2811-cc69-483b-8dcd-f31ed5ec8e5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:56:02.782602 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:02.782581 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-kube-api-access-hbwfd" pod "97be2811-cc69-483b-8dcd-f31ed5ec8e5b" (UID: "97be2811-cc69-483b-8dcd-f31ed5ec8e5b"). InnerVolumeSpecName "kube-api-access-hbwfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:02.882352 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:02.882320 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbwfd\" (UniqueName: \"kubernetes.io/projected/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-kube-api-access-hbwfd\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:02.882352 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:02.882351 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97be2811-cc69-483b-8dcd-f31ed5ec8e5b-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:03.307938 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:03.307905 6202 patch_prober.go:29] interesting pod/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:56:03.308118 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:03.307961 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" podUID="1f65a98a-1301-4d39-a40f-df19ec451712" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:56:03.393429 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:03.393388 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:03.508446 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:03.508419 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2" Oct 30 16:56:03.508623 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:03.508450 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2" event={"ID":"97be2811-cc69-483b-8dcd-f31ed5ec8e5b","Type":"ContainerDied","Data":"728fef87036f3eac7e81a608b821701d38285e499bc5b6f5c8076948fed8a0f9"} Oct 30 16:56:03.508623 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:03.508499 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="728fef87036f3eac7e81a608b821701d38285e499bc5b6f5c8076948fed8a0f9" Oct 30 16:56:04.205514 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.205491 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:56:04.306774 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.306739 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-workload-socket\") pod \"1f65a98a-1301-4d39-a40f-df19ec451712\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " Oct 30 16:56:04.307023 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.306836 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-credential-socket\") pod \"1f65a98a-1301-4d39-a40f-df19ec451712\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " Oct 30 16:56:04.307023 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.306910 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f65a98a-1301-4d39-a40f-df19ec451712-istio-token\" (UniqueName: \"kubernetes.io/projected/1f65a98a-1301-4d39-a40f-df19ec451712-istio-token\") pod \"1f65a98a-1301-4d39-a40f-df19ec451712\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " Oct 30 16:56:04.307023 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.306937 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f65a98a-1301-4d39-a40f-df19ec451712-kube-api-access-blmth\" (UniqueName: \"kubernetes.io/projected/1f65a98a-1301-4d39-a40f-df19ec451712-kube-api-access-blmth\") pod \"1f65a98a-1301-4d39-a40f-df19ec451712\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " Oct 30 16:56:04.307191 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.307106 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f65a98a-1301-4d39-a40f-df19ec451712-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f65a98a-1301-4d39-a40f-df19ec451712-istio-podinfo\") pod \"1f65a98a-1301-4d39-a40f-df19ec451712\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " Oct 30 16:56:04.307191 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.307160 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-workload-certs\") pod \"1f65a98a-1301-4d39-a40f-df19ec451712\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " Oct 30 16:56:04.307308 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.307248 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-istio-envoy\") pod \"1f65a98a-1301-4d39-a40f-df19ec451712\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " Oct 30 16:56:04.307308 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.307298 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f65a98a-1301-4d39-a40f-df19ec451712-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f65a98a-1301-4d39-a40f-df19ec451712-istiod-ca-cert\") pod \"1f65a98a-1301-4d39-a40f-df19ec451712\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " Oct 30 16:56:04.307491 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.307470 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-istio-data\") pod \"1f65a98a-1301-4d39-a40f-df19ec451712\" (UID: \"1f65a98a-1301-4d39-a40f-df19ec451712\") " Oct 30 16:56:04.307655 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.307623 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-credential-socket" pod "1f65a98a-1301-4d39-a40f-df19ec451712" (UID: "1f65a98a-1301-4d39-a40f-df19ec451712"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:04.307917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.307891 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-workload-certs" pod "1f65a98a-1301-4d39-a40f-df19ec451712" (UID: "1f65a98a-1301-4d39-a40f-df19ec451712"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:04.308152 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.308124 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f65a98a-1301-4d39-a40f-df19ec451712-istiod-ca-cert" pod "1f65a98a-1301-4d39-a40f-df19ec451712" (UID: "1f65a98a-1301-4d39-a40f-df19ec451712"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:56:04.308351 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.308331 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-istio-data" pod "1f65a98a-1301-4d39-a40f-df19ec451712" (UID: "1f65a98a-1301-4d39-a40f-df19ec451712"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:04.308485 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.308389 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-workload-socket" pod "1f65a98a-1301-4d39-a40f-df19ec451712" (UID: "1f65a98a-1301-4d39-a40f-df19ec451712"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:04.309256 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.309228 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:04.309364 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.309263 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f65a98a-1301-4d39-a40f-df19ec451712-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:04.309364 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.309278 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:04.309364 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.309295 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:04.309364 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.309309 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:04.310379 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.310345 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f65a98a-1301-4d39-a40f-df19ec451712-istio-token" pod "1f65a98a-1301-4d39-a40f-df19ec451712" (UID: "1f65a98a-1301-4d39-a40f-df19ec451712"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:04.310742 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.310701 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-istio-envoy" pod "1f65a98a-1301-4d39-a40f-df19ec451712" (UID: "1f65a98a-1301-4d39-a40f-df19ec451712"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:04.310742 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.310706 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f65a98a-1301-4d39-a40f-df19ec451712-kube-api-access-blmth" pod "1f65a98a-1301-4d39-a40f-df19ec451712" (UID: "1f65a98a-1301-4d39-a40f-df19ec451712"). InnerVolumeSpecName "kube-api-access-blmth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:04.312434 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.312395 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f65a98a-1301-4d39-a40f-df19ec451712-istio-podinfo" pod "1f65a98a-1301-4d39-a40f-df19ec451712" (UID: "1f65a98a-1301-4d39-a40f-df19ec451712"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:56:04.410406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.410377 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f65a98a-1301-4d39-a40f-df19ec451712-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:04.410406 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.410406 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f65a98a-1301-4d39-a40f-df19ec451712-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:04.410616 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.410414 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f65a98a-1301-4d39-a40f-df19ec451712-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:04.410616 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.410423 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blmth\" (UniqueName: \"kubernetes.io/projected/1f65a98a-1301-4d39-a40f-df19ec451712-kube-api-access-blmth\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:04.519203 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.519118 6202 generic.go:356] "Generic (PLEG): container finished" podID="1f65a98a-1301-4d39-a40f-df19ec451712" containerID="9b0e8e1e1ed81010e4e9aeb1debbb374f13786c5259e9c340974190a659f39bf" exitCode=0 Oct 30 16:56:04.519203 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.519163 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" Oct 30 16:56:04.519203 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.519189 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" event={"ID":"1f65a98a-1301-4d39-a40f-df19ec451712","Type":"ContainerDied","Data":"9b0e8e1e1ed81010e4e9aeb1debbb374f13786c5259e9c340974190a659f39bf"} Oct 30 16:56:04.519462 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.519240 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd" event={"ID":"1f65a98a-1301-4d39-a40f-df19ec451712","Type":"ContainerDied","Data":"97cb6202b36412ce941bc91ab8440dbf9a15736d0c1458b08340c51d5503bc10"} Oct 30 16:56:04.519462 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.519262 6202 scope.go:122] "RemoveContainer" containerID="9b0e8e1e1ed81010e4e9aeb1debbb374f13786c5259e9c340974190a659f39bf" Oct 30 16:56:04.532977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.532952 6202 scope.go:122] "RemoveContainer" containerID="805947c6ef9ee84b32d05c60294d386532b0a2401cdd92a499ee8a53249cf625" Oct 30 16:56:04.544439 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.544415 6202 scope.go:122] "RemoveContainer" containerID="9b0e8e1e1ed81010e4e9aeb1debbb374f13786c5259e9c340974190a659f39bf" Oct 30 16:56:04.544739 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:56:04.544719 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b0e8e1e1ed81010e4e9aeb1debbb374f13786c5259e9c340974190a659f39bf\": container with ID starting with 9b0e8e1e1ed81010e4e9aeb1debbb374f13786c5259e9c340974190a659f39bf not found: ID does not exist" containerID="9b0e8e1e1ed81010e4e9aeb1debbb374f13786c5259e9c340974190a659f39bf" Oct 30 16:56:04.544863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.544748 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b0e8e1e1ed81010e4e9aeb1debbb374f13786c5259e9c340974190a659f39bf"} err="failed to get container status \"9b0e8e1e1ed81010e4e9aeb1debbb374f13786c5259e9c340974190a659f39bf\": rpc error: code = NotFound desc = could not find container \"9b0e8e1e1ed81010e4e9aeb1debbb374f13786c5259e9c340974190a659f39bf\": container with ID starting with 9b0e8e1e1ed81010e4e9aeb1debbb374f13786c5259e9c340974190a659f39bf not found: ID does not exist" Oct 30 16:56:04.544863 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.544785 6202 scope.go:122] "RemoveContainer" containerID="805947c6ef9ee84b32d05c60294d386532b0a2401cdd92a499ee8a53249cf625" Oct 30 16:56:04.545043 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:56:04.545028 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805947c6ef9ee84b32d05c60294d386532b0a2401cdd92a499ee8a53249cf625\": container with ID starting with 805947c6ef9ee84b32d05c60294d386532b0a2401cdd92a499ee8a53249cf625 not found: ID does not exist" containerID="805947c6ef9ee84b32d05c60294d386532b0a2401cdd92a499ee8a53249cf625" Oct 30 16:56:04.545083 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.545058 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805947c6ef9ee84b32d05c60294d386532b0a2401cdd92a499ee8a53249cf625"} err="failed to get container status \"805947c6ef9ee84b32d05c60294d386532b0a2401cdd92a499ee8a53249cf625\": rpc error: code = NotFound desc = could not find container \"805947c6ef9ee84b32d05c60294d386532b0a2401cdd92a499ee8a53249cf625\": container with ID starting with 805947c6ef9ee84b32d05c60294d386532b0a2401cdd92a499ee8a53249cf625 not found: ID does not exist" Oct 30 16:56:04.573532 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.573506 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd"] Oct 30 16:56:04.577698 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.577675 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-529bc7-openshift-default-9f9b9959f-hvstd"] Oct 30 16:56:04.971317 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:04.971281 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:06.414021 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:06.413943 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f65a98a-1301-4d39-a40f-df19ec451712" path="/var/lib/kubelet/pods/1f65a98a-1301-4d39-a40f-df19ec451712/volumes" Oct 30 16:56:06.563741 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:06.563709 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:06.773845 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:06.773599 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:07.177447 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.177369 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v"] Oct 30 16:56:07.178266 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.178249 6202 cpu_manager.go:414] "containerMap: removing container" podUID="1f65a98a-1301-4d39-a40f-df19ec451712" containerName="istio-proxy" Oct 30 16:56:07.178266 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.178266 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f65a98a-1301-4d39-a40f-df19ec451712" containerName="istio-proxy" Oct 30 16:56:07.178374 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.178317 6202 cpu_manager.go:414] "containerMap: removing container" podUID="97be2811-cc69-483b-8dcd-f31ed5ec8e5b" containerName="revoke-keys" Oct 30 16:56:07.178374 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.178322 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97be2811-cc69-483b-8dcd-f31ed5ec8e5b" containerName="revoke-keys" Oct 30 16:56:07.178449 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.178440 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="97be2811-cc69-483b-8dcd-f31ed5ec8e5b" containerName="revoke-keys" Oct 30 16:56:07.178486 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.178454 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f65a98a-1301-4d39-a40f-df19ec451712" containerName="istio-proxy" Oct 30 16:56:07.185242 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.185206 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v" Oct 30 16:56:07.202886 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.202859 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v"] Oct 30 16:56:07.220796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.208077 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:56:07.220796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.208183 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:56:07.374318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.374270 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61969530-e89d-44aa-801b-78ad6fcc61b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b38945ae-stj5v\" (UID: \"61969530-e89d-44aa-801b-78ad6fcc61b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v" Oct 30 16:56:07.374509 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.374395 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61969530-e89d-44aa-801b-78ad6fcc61b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b38945ae-stj5v\" (UID: \"61969530-e89d-44aa-801b-78ad6fcc61b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v" Oct 30 16:56:07.374509 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.374453 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7kvz\" (UniqueName: \"kubernetes.io/projected/61969530-e89d-44aa-801b-78ad6fcc61b1-kube-api-access-z7kvz\") pod \"maas-api-revoke-sub-b38945ae-stj5v\" (UID: \"61969530-e89d-44aa-801b-78ad6fcc61b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v" Oct 30 16:56:07.476799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.476703 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61969530-e89d-44aa-801b-78ad6fcc61b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b38945ae-stj5v\" (UID: \"61969530-e89d-44aa-801b-78ad6fcc61b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v" Oct 30 16:56:07.477257 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.476818 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61969530-e89d-44aa-801b-78ad6fcc61b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b38945ae-stj5v\" (UID: \"61969530-e89d-44aa-801b-78ad6fcc61b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v" Oct 30 16:56:07.477257 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.476877 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7kvz\" (UniqueName: \"kubernetes.io/projected/61969530-e89d-44aa-801b-78ad6fcc61b1-kube-api-access-z7kvz\") pod \"maas-api-revoke-sub-b38945ae-stj5v\" (UID: \"61969530-e89d-44aa-801b-78ad6fcc61b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v" Oct 30 16:56:07.477508 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.477480 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61969530-e89d-44aa-801b-78ad6fcc61b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b38945ae-stj5v\" (UID: \"61969530-e89d-44aa-801b-78ad6fcc61b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v" Oct 30 16:56:07.479605 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.479577 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61969530-e89d-44aa-801b-78ad6fcc61b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b38945ae-stj5v\" (UID: \"61969530-e89d-44aa-801b-78ad6fcc61b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v" Oct 30 16:56:07.500097 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.500078 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7kvz\" (UniqueName: \"kubernetes.io/projected/61969530-e89d-44aa-801b-78ad6fcc61b1-kube-api-access-z7kvz\") pod \"maas-api-revoke-sub-b38945ae-stj5v\" (UID: \"61969530-e89d-44aa-801b-78ad6fcc61b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v" Oct 30 16:56:07.505949 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.505920 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v" Oct 30 16:56:07.816399 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:07.816369 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v"] Oct 30 16:56:07.817176 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:56:07.817156 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61969530_e89d_44aa_801b_78ad6fcc61b1.slice/crio-d5297eb19edd8cf9803c91b7cc1d11a550885444f53d35beeaf8b31253b496be WatchSource:0}: Error finding container d5297eb19edd8cf9803c91b7cc1d11a550885444f53d35beeaf8b31253b496be: Status 404 returned error can't find the container with id d5297eb19edd8cf9803c91b7cc1d11a550885444f53d35beeaf8b31253b496be Oct 30 16:56:08.382625 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:08.382592 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:08.575370 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:08.575333 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v" event={"ID":"61969530-e89d-44aa-801b-78ad6fcc61b1","Type":"ContainerStarted","Data":"ea65e5b21d07716e4fd88f9ad9bee9e524c74b2ca7fb2d9abaa648bb0fef951d"} Oct 30 16:56:08.575831 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:08.575405 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v" event={"ID":"61969530-e89d-44aa-801b-78ad6fcc61b1","Type":"ContainerStarted","Data":"d5297eb19edd8cf9803c91b7cc1d11a550885444f53d35beeaf8b31253b496be"} Oct 30 16:56:08.616205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:08.616150 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v" podStartSLOduration=1.6161334269999998 podStartE2EDuration="1.616133427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:56:08.615026598 +0000 UTC m=+2082.325481341" watchObservedRunningTime="2026-10-30 16:56:08.616133427 +0000 UTC m=+2082.326588173" Oct 30 16:56:09.591701 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:09.591664 6202 generic.go:356] "Generic (PLEG): container finished" podID="61969530-e89d-44aa-801b-78ad6fcc61b1" containerID="ea65e5b21d07716e4fd88f9ad9bee9e524c74b2ca7fb2d9abaa648bb0fef951d" exitCode=0 Oct 30 16:56:09.592229 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:09.591739 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v" event={"ID":"61969530-e89d-44aa-801b-78ad6fcc61b1","Type":"ContainerDied","Data":"ea65e5b21d07716e4fd88f9ad9bee9e524c74b2ca7fb2d9abaa648bb0fef951d"} Oct 30 16:56:09.686074 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:09.686042 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:09.845541 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:09.845460 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c"] Oct 30 16:56:09.855975 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:09.855946 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:09.883693 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:09.883662 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92"] Oct 30 16:56:09.997703 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:09.997333 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:10.134882 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:10.134781 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:10.877522 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:10.877492 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v" Oct 30 16:56:10.958908 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:10.958820 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61969530-e89d-44aa-801b-78ad6fcc61b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61969530-e89d-44aa-801b-78ad6fcc61b1-maas-api-xks-ca\") pod \"61969530-e89d-44aa-801b-78ad6fcc61b1\" (UID: \"61969530-e89d-44aa-801b-78ad6fcc61b1\") " Oct 30 16:56:10.958908 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:10.958872 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61969530-e89d-44aa-801b-78ad6fcc61b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61969530-e89d-44aa-801b-78ad6fcc61b1-maas-api-openshift-ca\") pod \"61969530-e89d-44aa-801b-78ad6fcc61b1\" (UID: \"61969530-e89d-44aa-801b-78ad6fcc61b1\") " Oct 30 16:56:10.959169 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:10.959148 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61969530-e89d-44aa-801b-78ad6fcc61b1-kube-api-access-z7kvz\" (UniqueName: \"kubernetes.io/projected/61969530-e89d-44aa-801b-78ad6fcc61b1-kube-api-access-z7kvz\") pod \"61969530-e89d-44aa-801b-78ad6fcc61b1\" (UID: \"61969530-e89d-44aa-801b-78ad6fcc61b1\") " Oct 30 16:56:10.959295 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:10.959271 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61969530-e89d-44aa-801b-78ad6fcc61b1-maas-api-openshift-ca" pod "61969530-e89d-44aa-801b-78ad6fcc61b1" (UID: "61969530-e89d-44aa-801b-78ad6fcc61b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:56:10.959949 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:10.959924 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61969530-e89d-44aa-801b-78ad6fcc61b1-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:10.963190 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:10.963152 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61969530-e89d-44aa-801b-78ad6fcc61b1-maas-api-xks-ca" pod "61969530-e89d-44aa-801b-78ad6fcc61b1" (UID: "61969530-e89d-44aa-801b-78ad6fcc61b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:56:10.964726 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:10.964698 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61969530-e89d-44aa-801b-78ad6fcc61b1-kube-api-access-z7kvz" pod "61969530-e89d-44aa-801b-78ad6fcc61b1" (UID: "61969530-e89d-44aa-801b-78ad6fcc61b1"). InnerVolumeSpecName "kube-api-access-z7kvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:11.061376 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:11.061335 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7kvz\" (UniqueName: \"kubernetes.io/projected/61969530-e89d-44aa-801b-78ad6fcc61b1-kube-api-access-z7kvz\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:11.061376 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:11.061373 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61969530-e89d-44aa-801b-78ad6fcc61b1-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:11.500743 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:11.500704 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:11.617069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:11.617032 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:11.654298 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:11.654265 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v" Oct 30 16:56:11.654497 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:11.654304 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v" event={"ID":"61969530-e89d-44aa-801b-78ad6fcc61b1","Type":"ContainerDied","Data":"d5297eb19edd8cf9803c91b7cc1d11a550885444f53d35beeaf8b31253b496be"} Oct 30 16:56:11.654497 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:11.654354 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5297eb19edd8cf9803c91b7cc1d11a550885444f53d35beeaf8b31253b496be" Oct 30 16:56:13.209456 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:13.209418 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:13.322270 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:13.322226 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:14.076702 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:56:14.076661 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:56:14.846564 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:14.846513 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" podUID="bc3b37b0-b774-4172-917e-b887db4df343" containerName="payload-processing" containerID="cri-o://6ab1d6f25ac64ddc8136793275db54e2108a4aa4e808450c16d1557e2ee6b533" gracePeriod=25 Oct 30 16:56:14.906181 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:14.894248 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" podUID="c15c9a03-dd7f-4323-a9a5-83b76cd584ff" containerName="payload-pre-processing" containerID="cri-o://52aa1db6ba7d73b3dfb90dce6d83b02ea3f119aae4a3cc9df28d98385a2d0ee9" gracePeriod=25 Oct 30 16:56:14.939598 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:14.939560 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:16.377367 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:16.377288 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:16.520884 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:16.520846 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" podUID="c15c9a03-dd7f-4323-a9a5-83b76cd584ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:56:16.520884 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:16.520874 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" podUID="bc3b37b0-b774-4172-917e-b887db4df343" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:56:18.081559 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:18.081517 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:18.580940 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:18.580904 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:18.845051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:18.844961 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf"] Oct 30 16:56:18.845931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:18.845912 6202 cpu_manager.go:414] "containerMap: removing container" podUID="61969530-e89d-44aa-801b-78ad6fcc61b1" containerName="revoke-keys" Oct 30 16:56:18.846023 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:18.845933 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61969530-e89d-44aa-801b-78ad6fcc61b1" containerName="revoke-keys" Oct 30 16:56:18.846023 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:18.845955 6202 cpu_manager.go:414] "containerMap: removing container" podUID="1f65a98a-1301-4d39-a40f-df19ec451712" containerName="istio-proxy" Oct 30 16:56:18.846023 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:18.845960 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f65a98a-1301-4d39-a40f-df19ec451712" containerName="istio-proxy" Oct 30 16:56:18.846170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:18.846098 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="61969530-e89d-44aa-801b-78ad6fcc61b1" containerName="revoke-keys" Oct 30 16:56:18.846170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:18.846112 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f65a98a-1301-4d39-a40f-df19ec451712" containerName="istio-proxy" Oct 30 16:56:18.850588 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:18.850563 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf" Oct 30 16:56:18.870702 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:18.870674 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf"] Oct 30 16:56:18.877203 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:18.877167 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:56:18.877350 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:18.877237 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:56:18.948218 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:18.948181 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42gfh\" (UniqueName: \"kubernetes.io/projected/900ad7ef-2fd9-44da-9c9a-7f08341b3962-kube-api-access-42gfh\") pod \"maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf\" (UID: \"900ad7ef-2fd9-44da-9c9a-7f08341b3962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf" Oct 30 16:56:18.948469 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:18.948446 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900ad7ef-2fd9-44da-9c9a-7f08341b3962-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf\" (UID: \"900ad7ef-2fd9-44da-9c9a-7f08341b3962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf" Oct 30 16:56:18.948623 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:18.948601 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900ad7ef-2fd9-44da-9c9a-7f08341b3962-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf\" (UID: \"900ad7ef-2fd9-44da-9c9a-7f08341b3962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf" Oct 30 16:56:19.050246 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:19.050211 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900ad7ef-2fd9-44da-9c9a-7f08341b3962-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf\" (UID: \"900ad7ef-2fd9-44da-9c9a-7f08341b3962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf" Oct 30 16:56:19.050453 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:19.050301 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900ad7ef-2fd9-44da-9c9a-7f08341b3962-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf\" (UID: \"900ad7ef-2fd9-44da-9c9a-7f08341b3962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf" Oct 30 16:56:19.050453 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:19.050404 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gfh\" (UniqueName: \"kubernetes.io/projected/900ad7ef-2fd9-44da-9c9a-7f08341b3962-kube-api-access-42gfh\") pod \"maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf\" (UID: \"900ad7ef-2fd9-44da-9c9a-7f08341b3962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf" Oct 30 16:56:19.051027 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:19.050994 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900ad7ef-2fd9-44da-9c9a-7f08341b3962-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf\" (UID: \"900ad7ef-2fd9-44da-9c9a-7f08341b3962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf" Oct 30 16:56:19.053208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:19.053182 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900ad7ef-2fd9-44da-9c9a-7f08341b3962-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf\" (UID: \"900ad7ef-2fd9-44da-9c9a-7f08341b3962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf" Oct 30 16:56:19.074093 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:19.074067 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42gfh\" (UniqueName: \"kubernetes.io/projected/900ad7ef-2fd9-44da-9c9a-7f08341b3962-kube-api-access-42gfh\") pod \"maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf\" (UID: \"900ad7ef-2fd9-44da-9c9a-7f08341b3962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf" Oct 30 16:56:19.181632 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:19.181545 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf" Oct 30 16:56:19.508303 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:56:19.508272 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900ad7ef_2fd9_44da_9c9a_7f08341b3962.slice/crio-e88be765180192a62a3b3371fca0b977abf348951cd664cb91d5c9e07070a0d9 WatchSource:0}: Error finding container e88be765180192a62a3b3371fca0b977abf348951cd664cb91d5c9e07070a0d9: Status 404 returned error can't find the container with id e88be765180192a62a3b3371fca0b977abf348951cd664cb91d5c9e07070a0d9 Oct 30 16:56:19.508457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:19.508426 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf"] Oct 30 16:56:19.748632 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:19.748597 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf" event={"ID":"900ad7ef-2fd9-44da-9c9a-7f08341b3962","Type":"ContainerStarted","Data":"e88be765180192a62a3b3371fca0b977abf348951cd664cb91d5c9e07070a0d9"} Oct 30 16:56:19.936494 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:19.936454 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:20.760343 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:20.760305 6202 generic.go:356] "Generic (PLEG): container finished" podID="900ad7ef-2fd9-44da-9c9a-7f08341b3962" containerID="1228a3596b5c9528aea24640cc1982cca57f71ebb6f6ad3a4fa1a7b3734f8018" exitCode=0 Oct 30 16:56:20.760840 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:20.760394 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf" event={"ID":"900ad7ef-2fd9-44da-9c9a-7f08341b3962","Type":"ContainerDied","Data":"1228a3596b5c9528aea24640cc1982cca57f71ebb6f6ad3a4fa1a7b3734f8018"} Oct 30 16:56:21.617896 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:21.617858 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:21.954242 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:21.954212 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf" Oct 30 16:56:22.034812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:22.034691 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/900ad7ef-2fd9-44da-9c9a-7f08341b3962-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900ad7ef-2fd9-44da-9c9a-7f08341b3962-maas-api-xks-ca\") pod \"900ad7ef-2fd9-44da-9c9a-7f08341b3962\" (UID: \"900ad7ef-2fd9-44da-9c9a-7f08341b3962\") " Oct 30 16:56:22.035019 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:22.034821 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900ad7ef-2fd9-44da-9c9a-7f08341b3962-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900ad7ef-2fd9-44da-9c9a-7f08341b3962-maas-api-openshift-ca\") pod \"900ad7ef-2fd9-44da-9c9a-7f08341b3962\" (UID: \"900ad7ef-2fd9-44da-9c9a-7f08341b3962\") " Oct 30 16:56:22.035019 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:22.034857 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/900ad7ef-2fd9-44da-9c9a-7f08341b3962-kube-api-access-42gfh\" (UniqueName: \"kubernetes.io/projected/900ad7ef-2fd9-44da-9c9a-7f08341b3962-kube-api-access-42gfh\") pod \"900ad7ef-2fd9-44da-9c9a-7f08341b3962\" (UID: \"900ad7ef-2fd9-44da-9c9a-7f08341b3962\") " Oct 30 16:56:22.035343 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:22.035279 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900ad7ef-2fd9-44da-9c9a-7f08341b3962-maas-api-openshift-ca" pod "900ad7ef-2fd9-44da-9c9a-7f08341b3962" (UID: "900ad7ef-2fd9-44da-9c9a-7f08341b3962"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:56:22.035634 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:22.035612 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900ad7ef-2fd9-44da-9c9a-7f08341b3962-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:22.038914 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:22.038873 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/900ad7ef-2fd9-44da-9c9a-7f08341b3962-maas-api-xks-ca" pod "900ad7ef-2fd9-44da-9c9a-7f08341b3962" (UID: "900ad7ef-2fd9-44da-9c9a-7f08341b3962"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:56:22.039041 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:22.038918 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900ad7ef-2fd9-44da-9c9a-7f08341b3962-kube-api-access-42gfh" pod "900ad7ef-2fd9-44da-9c9a-7f08341b3962" (UID: "900ad7ef-2fd9-44da-9c9a-7f08341b3962"). InnerVolumeSpecName "kube-api-access-42gfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:22.137664 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:22.137627 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900ad7ef-2fd9-44da-9c9a-7f08341b3962-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:22.137664 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:22.137659 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42gfh\" (UniqueName: \"kubernetes.io/projected/900ad7ef-2fd9-44da-9c9a-7f08341b3962-kube-api-access-42gfh\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:22.784410 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:22.784334 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf" Oct 30 16:56:22.784580 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:22.784387 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf" event={"ID":"900ad7ef-2fd9-44da-9c9a-7f08341b3962","Type":"ContainerDied","Data":"e88be765180192a62a3b3371fca0b977abf348951cd664cb91d5c9e07070a0d9"} Oct 30 16:56:22.784580 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:22.784453 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e88be765180192a62a3b3371fca0b977abf348951cd664cb91d5c9e07070a0d9" Oct 30 16:56:23.457658 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:23.457604 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:25.288281 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:25.288240 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:26.535801 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:26.535736 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" podUID="bc3b37b0-b774-4172-917e-b887db4df343" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:56:26.541479 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:26.541438 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" podUID="c15c9a03-dd7f-4323-a9a5-83b76cd584ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:56:26.861078 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:26.861044 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:28.516489 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:28.516454 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:29.243614 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:29.243580 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs"] Oct 30 16:56:29.386390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:29.386355 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:29.561848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:29.560069 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:30.799019 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:30.798976 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:31.718005 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:31.717971 6202 scope.go:122] "RemoveContainer" containerID="d52a1feebf490c9bcb620547c8683b2e2323e3482f7f1f698149bc207a75a3dd" Oct 30 16:56:31.731341 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:31.731313 6202 scope.go:122] "RemoveContainer" containerID="8da7a577762116c664954402f8c6282459daaa5a3ffedadf2ba95d68e088ed02" Oct 30 16:56:31.745979 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:31.745958 6202 scope.go:122] "RemoveContainer" containerID="76b5a30126e26238df544f902cf98f8444366da240f3260dcb711fcf415a89cb" Oct 30 16:56:32.437857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:32.437820 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:34.116033 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:34.115990 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:34.244208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:34.244160 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" podUID="4c79977c-90b4-4d76-b6b4-b24aaf3df92b" containerName="payload-processing" containerID="cri-o://81e406a2b0c9d3c01725f0c6a3a92c20be48ca8983296dac27219f16e047ab15" gracePeriod=25 Oct 30 16:56:34.997904 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:34.997872 6202 generic.go:356] "Generic (PLEG): container finished" podID="bc3b37b0-b774-4172-917e-b887db4df343" containerID="6ab1d6f25ac64ddc8136793275db54e2108a4aa4e808450c16d1557e2ee6b533" exitCode=0 Oct 30 16:56:34.998085 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:34.997948 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" event={"ID":"bc3b37b0-b774-4172-917e-b887db4df343","Type":"ContainerDied","Data":"6ab1d6f25ac64ddc8136793275db54e2108a4aa4e808450c16d1557e2ee6b533"} Oct 30 16:56:35.214492 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:35.214464 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" Oct 30 16:56:35.265152 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:35.265070 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" Oct 30 16:56:35.343521 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:35.343484 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15c9a03-dd7f-4323-a9a5-83b76cd584ff-kube-api-access-c2t2p\" (UniqueName: \"kubernetes.io/projected/c15c9a03-dd7f-4323-a9a5-83b76cd584ff-kube-api-access-c2t2p\") pod \"c15c9a03-dd7f-4323-a9a5-83b76cd584ff\" (UID: \"c15c9a03-dd7f-4323-a9a5-83b76cd584ff\") " Oct 30 16:56:35.343728 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:35.343578 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc3b37b0-b774-4172-917e-b887db4df343-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3b37b0-b774-4172-917e-b887db4df343-plugins-config-volume\") pod \"bc3b37b0-b774-4172-917e-b887db4df343\" (UID: \"bc3b37b0-b774-4172-917e-b887db4df343\") " Oct 30 16:56:35.343728 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:35.343686 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15c9a03-dd7f-4323-a9a5-83b76cd584ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15c9a03-dd7f-4323-a9a5-83b76cd584ff-plugins-config-volume\") pod \"c15c9a03-dd7f-4323-a9a5-83b76cd584ff\" (UID: \"c15c9a03-dd7f-4323-a9a5-83b76cd584ff\") " Oct 30 16:56:35.343962 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:35.343941 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3b37b0-b774-4172-917e-b887db4df343-kube-api-access-zz8np\" (UniqueName: \"kubernetes.io/projected/bc3b37b0-b774-4172-917e-b887db4df343-kube-api-access-zz8np\") pod \"bc3b37b0-b774-4172-917e-b887db4df343\" (UID: \"bc3b37b0-b774-4172-917e-b887db4df343\") " Oct 30 16:56:35.344334 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:35.344306 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15c9a03-dd7f-4323-a9a5-83b76cd584ff-plugins-config-volume" pod "c15c9a03-dd7f-4323-a9a5-83b76cd584ff" (UID: "c15c9a03-dd7f-4323-a9a5-83b76cd584ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:56:35.344964 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:35.344920 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15c9a03-dd7f-4323-a9a5-83b76cd584ff-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:35.345105 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:35.345088 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3b37b0-b774-4172-917e-b887db4df343-plugins-config-volume" pod "bc3b37b0-b774-4172-917e-b887db4df343" (UID: "bc3b37b0-b774-4172-917e-b887db4df343"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:56:35.347173 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:35.347143 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15c9a03-dd7f-4323-a9a5-83b76cd584ff-kube-api-access-c2t2p" pod "c15c9a03-dd7f-4323-a9a5-83b76cd584ff" (UID: "c15c9a03-dd7f-4323-a9a5-83b76cd584ff"). InnerVolumeSpecName "kube-api-access-c2t2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:35.347350 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:35.347323 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3b37b0-b774-4172-917e-b887db4df343-kube-api-access-zz8np" pod "bc3b37b0-b774-4172-917e-b887db4df343" (UID: "bc3b37b0-b774-4172-917e-b887db4df343"). InnerVolumeSpecName "kube-api-access-zz8np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:35.446894 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:35.446805 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2t2p\" (UniqueName: \"kubernetes.io/projected/c15c9a03-dd7f-4323-a9a5-83b76cd584ff-kube-api-access-c2t2p\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:35.446894 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:35.446839 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3b37b0-b774-4172-917e-b887db4df343-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:35.446894 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:35.446855 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz8np\" (UniqueName: \"kubernetes.io/projected/bc3b37b0-b774-4172-917e-b887db4df343-kube-api-access-zz8np\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:35.863955 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:35.863916 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:35.979127 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:35.979091 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:36.014929 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:36.014755 6202 generic.go:356] "Generic (PLEG): container finished" podID="c15c9a03-dd7f-4323-a9a5-83b76cd584ff" containerID="52aa1db6ba7d73b3dfb90dce6d83b02ea3f119aae4a3cc9df28d98385a2d0ee9" exitCode=0 Oct 30 16:56:36.014929 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:36.014914 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" Oct 30 16:56:36.015301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:36.014948 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" event={"ID":"c15c9a03-dd7f-4323-a9a5-83b76cd584ff","Type":"ContainerDied","Data":"52aa1db6ba7d73b3dfb90dce6d83b02ea3f119aae4a3cc9df28d98385a2d0ee9"} Oct 30 16:56:36.015301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:36.015022 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92" event={"ID":"c15c9a03-dd7f-4323-a9a5-83b76cd584ff","Type":"ContainerDied","Data":"658f17bb7081dfe924352a03e204865d6c77dcd0e691b88e84851e48464d77e7"} Oct 30 16:56:36.015301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:36.015057 6202 scope.go:122] "RemoveContainer" containerID="52aa1db6ba7d73b3dfb90dce6d83b02ea3f119aae4a3cc9df28d98385a2d0ee9" Oct 30 16:56:36.017297 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:36.017273 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" event={"ID":"bc3b37b0-b774-4172-917e-b887db4df343","Type":"ContainerDied","Data":"7e182be153e7d8f0b959e4f9937554fffa38cb7675662c7aebb0843ad0d62fb1"} Oct 30 16:56:36.017444 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:36.017308 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c" Oct 30 16:56:36.032737 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:36.032711 6202 scope.go:122] "RemoveContainer" containerID="52aa1db6ba7d73b3dfb90dce6d83b02ea3f119aae4a3cc9df28d98385a2d0ee9" Oct 30 16:56:36.033199 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:56:36.033173 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52aa1db6ba7d73b3dfb90dce6d83b02ea3f119aae4a3cc9df28d98385a2d0ee9\": container with ID starting with 52aa1db6ba7d73b3dfb90dce6d83b02ea3f119aae4a3cc9df28d98385a2d0ee9 not found: ID does not exist" containerID="52aa1db6ba7d73b3dfb90dce6d83b02ea3f119aae4a3cc9df28d98385a2d0ee9" Oct 30 16:56:36.033282 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:36.033206 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52aa1db6ba7d73b3dfb90dce6d83b02ea3f119aae4a3cc9df28d98385a2d0ee9"} err="failed to get container status \"52aa1db6ba7d73b3dfb90dce6d83b02ea3f119aae4a3cc9df28d98385a2d0ee9\": rpc error: code = NotFound desc = could not find container \"52aa1db6ba7d73b3dfb90dce6d83b02ea3f119aae4a3cc9df28d98385a2d0ee9\": container with ID starting with 52aa1db6ba7d73b3dfb90dce6d83b02ea3f119aae4a3cc9df28d98385a2d0ee9 not found: ID does not exist" Oct 30 16:56:36.033282 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:36.033225 6202 scope.go:122] "RemoveContainer" containerID="6ab1d6f25ac64ddc8136793275db54e2108a4aa4e808450c16d1557e2ee6b533" Oct 30 16:56:36.117318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:36.117232 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92"] Oct 30 16:56:36.128488 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:36.128457 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3ee682-777b754b7f-kzc92"] Oct 30 16:56:36.155497 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:36.155463 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c"] Oct 30 16:56:36.164259 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:36.164233 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3ee682-79b689876d-dwr5c"] Oct 30 16:56:36.416499 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:36.416409 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc3b37b0-b774-4172-917e-b887db4df343" path="/var/lib/kubelet/pods/bc3b37b0-b774-4172-917e-b887db4df343/volumes" Oct 30 16:56:36.417012 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:36.416994 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15c9a03-dd7f-4323-a9a5-83b76cd584ff" path="/var/lib/kubelet/pods/c15c9a03-dd7f-4323-a9a5-83b76cd584ff/volumes" Oct 30 16:56:37.432402 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:37.432370 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:38.277062 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:38.277027 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:38.397679 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:38.397638 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf"] Oct 30 16:56:38.444839 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:38.437215 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:38.462198 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:38.462163 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff1189-bq8rf"] Oct 30 16:56:38.555068 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:38.554975 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:38.897361 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:38.897330 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:40.192224 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:40.192191 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:40.307948 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:40.307913 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:40.413175 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:40.413139 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="900ad7ef-2fd9-44da-9c9a-7f08341b3962" path="/var/lib/kubelet/pods/900ad7ef-2fd9-44da-9c9a-7f08341b3962/volumes" Oct 30 16:56:41.742739 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:41.742705 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:43.568157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:43.568116 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:43.916574 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:43.916539 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" podUID="4c79977c-90b4-4d76-b6b4-b24aaf3df92b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:56:44.070409 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:56:44.070370 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:56:45.138240 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:45.138201 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:45.432435 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:45.432353 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:45.550793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:45.548041 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:45.785136 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:45.785019 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:46.320739 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.320706 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg"] Oct 30 16:56:46.321628 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.321610 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c15c9a03-dd7f-4323-a9a5-83b76cd584ff" containerName="payload-pre-processing" Oct 30 16:56:46.321693 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.321631 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15c9a03-dd7f-4323-a9a5-83b76cd584ff" containerName="payload-pre-processing" Oct 30 16:56:46.321693 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.321670 6202 cpu_manager.go:414] "containerMap: removing container" podUID="900ad7ef-2fd9-44da-9c9a-7f08341b3962" containerName="revoke-keys" Oct 30 16:56:46.321693 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.321676 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="900ad7ef-2fd9-44da-9c9a-7f08341b3962" containerName="revoke-keys" Oct 30 16:56:46.321830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.321698 6202 cpu_manager.go:414] "containerMap: removing container" podUID="bc3b37b0-b774-4172-917e-b887db4df343" containerName="payload-processing" Oct 30 16:56:46.321830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.321704 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3b37b0-b774-4172-917e-b887db4df343" containerName="payload-processing" Oct 30 16:56:46.321830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.321823 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15c9a03-dd7f-4323-a9a5-83b76cd584ff" containerName="payload-pre-processing" Oct 30 16:56:46.321929 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.321839 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="900ad7ef-2fd9-44da-9c9a-7f08341b3962" containerName="revoke-keys" Oct 30 16:56:46.321929 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.321847 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3b37b0-b774-4172-917e-b887db4df343" containerName="payload-processing" Oct 30 16:56:46.327616 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.327598 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg" Oct 30 16:56:46.345868 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.345814 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg"] Oct 30 16:56:46.384481 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.384436 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5074fa9b-3fb9-4603-8c91-99bd249b0917-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg\" (UID: \"5074fa9b-3fb9-4603-8c91-99bd249b0917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg" Oct 30 16:56:46.384750 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.384720 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5074fa9b-3fb9-4603-8c91-99bd249b0917-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg\" (UID: \"5074fa9b-3fb9-4603-8c91-99bd249b0917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg" Oct 30 16:56:46.384889 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.384869 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpw47\" (UniqueName: \"kubernetes.io/projected/5074fa9b-3fb9-4603-8c91-99bd249b0917-kube-api-access-qpw47\") pod \"maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg\" (UID: \"5074fa9b-3fb9-4603-8c91-99bd249b0917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg" Oct 30 16:56:46.424186 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.424152 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:56:46.424186 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.424182 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:56:46.486253 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.486207 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5074fa9b-3fb9-4603-8c91-99bd249b0917-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg\" (UID: \"5074fa9b-3fb9-4603-8c91-99bd249b0917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg" Oct 30 16:56:46.486453 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.486343 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpw47\" (UniqueName: \"kubernetes.io/projected/5074fa9b-3fb9-4603-8c91-99bd249b0917-kube-api-access-qpw47\") pod \"maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg\" (UID: \"5074fa9b-3fb9-4603-8c91-99bd249b0917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg" Oct 30 16:56:46.486736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.486702 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5074fa9b-3fb9-4603-8c91-99bd249b0917-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg\" (UID: \"5074fa9b-3fb9-4603-8c91-99bd249b0917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg" Oct 30 16:56:46.487148 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.487121 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5074fa9b-3fb9-4603-8c91-99bd249b0917-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg\" (UID: \"5074fa9b-3fb9-4603-8c91-99bd249b0917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg" Oct 30 16:56:46.489303 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.489283 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5074fa9b-3fb9-4603-8c91-99bd249b0917-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg\" (UID: \"5074fa9b-3fb9-4603-8c91-99bd249b0917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg" Oct 30 16:56:46.510577 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.510552 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpw47\" (UniqueName: \"kubernetes.io/projected/5074fa9b-3fb9-4603-8c91-99bd249b0917-kube-api-access-qpw47\") pod \"maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg\" (UID: \"5074fa9b-3fb9-4603-8c91-99bd249b0917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg" Oct 30 16:56:46.643302 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.643271 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg" Oct 30 16:56:46.951594 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:56:46.951545 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5074fa9b_3fb9_4603_8c91_99bd249b0917.slice/crio-97e1ad8e6d5eaf56bd5b5263b2db375b68856d90f5c94817d441158692a11f0e WatchSource:0}: Error finding container 97e1ad8e6d5eaf56bd5b5263b2db375b68856d90f5c94817d441158692a11f0e: Status 404 returned error can't find the container with id 97e1ad8e6d5eaf56bd5b5263b2db375b68856d90f5c94817d441158692a11f0e Oct 30 16:56:46.955690 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.955661 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg"] Oct 30 16:56:46.976804 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:46.965205 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:47.149745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:47.149695 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg" event={"ID":"5074fa9b-3fb9-4603-8c91-99bd249b0917","Type":"ContainerStarted","Data":"97e1ad8e6d5eaf56bd5b5263b2db375b68856d90f5c94817d441158692a11f0e"} Oct 30 16:56:48.161513 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:48.161474 6202 generic.go:356] "Generic (PLEG): container finished" podID="5074fa9b-3fb9-4603-8c91-99bd249b0917" containerID="eb56478b60ff9cba2b4689b35781e709766a58830b9fd95ed7ba4a14348b62ab" exitCode=0 Oct 30 16:56:48.161913 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:48.161556 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg" event={"ID":"5074fa9b-3fb9-4603-8c91-99bd249b0917","Type":"ContainerDied","Data":"eb56478b60ff9cba2b4689b35781e709766a58830b9fd95ed7ba4a14348b62ab"} Oct 30 16:56:48.777624 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:48.777591 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:49.347909 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:49.347875 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg" Oct 30 16:56:49.446998 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:49.446916 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5074fa9b-3fb9-4603-8c91-99bd249b0917-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5074fa9b-3fb9-4603-8c91-99bd249b0917-maas-api-openshift-ca\") pod \"5074fa9b-3fb9-4603-8c91-99bd249b0917\" (UID: \"5074fa9b-3fb9-4603-8c91-99bd249b0917\") " Oct 30 16:56:49.447195 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:49.447180 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5074fa9b-3fb9-4603-8c91-99bd249b0917-kube-api-access-qpw47\" (UniqueName: \"kubernetes.io/projected/5074fa9b-3fb9-4603-8c91-99bd249b0917-kube-api-access-qpw47\") pod \"5074fa9b-3fb9-4603-8c91-99bd249b0917\" (UID: \"5074fa9b-3fb9-4603-8c91-99bd249b0917\") " Oct 30 16:56:49.447263 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:49.447212 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5074fa9b-3fb9-4603-8c91-99bd249b0917-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5074fa9b-3fb9-4603-8c91-99bd249b0917-maas-api-xks-ca\") pod \"5074fa9b-3fb9-4603-8c91-99bd249b0917\" (UID: \"5074fa9b-3fb9-4603-8c91-99bd249b0917\") " Oct 30 16:56:49.447365 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:49.447338 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5074fa9b-3fb9-4603-8c91-99bd249b0917-maas-api-openshift-ca" pod "5074fa9b-3fb9-4603-8c91-99bd249b0917" (UID: "5074fa9b-3fb9-4603-8c91-99bd249b0917"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:56:49.448212 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:49.448186 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5074fa9b-3fb9-4603-8c91-99bd249b0917-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:49.449611 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:49.449584 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5074fa9b-3fb9-4603-8c91-99bd249b0917-kube-api-access-qpw47" pod "5074fa9b-3fb9-4603-8c91-99bd249b0917" (UID: "5074fa9b-3fb9-4603-8c91-99bd249b0917"). InnerVolumeSpecName "kube-api-access-qpw47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:49.449698 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:49.449631 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5074fa9b-3fb9-4603-8c91-99bd249b0917-maas-api-xks-ca" pod "5074fa9b-3fb9-4603-8c91-99bd249b0917" (UID: "5074fa9b-3fb9-4603-8c91-99bd249b0917"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:56:49.550146 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:49.550047 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpw47\" (UniqueName: \"kubernetes.io/projected/5074fa9b-3fb9-4603-8c91-99bd249b0917-kube-api-access-qpw47\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:49.550146 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:49.550083 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5074fa9b-3fb9-4603-8c91-99bd249b0917-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:50.191007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:50.190961 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg" event={"ID":"5074fa9b-3fb9-4603-8c91-99bd249b0917","Type":"ContainerDied","Data":"97e1ad8e6d5eaf56bd5b5263b2db375b68856d90f5c94817d441158692a11f0e"} Oct 30 16:56:50.191193 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:50.191064 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97e1ad8e6d5eaf56bd5b5263b2db375b68856d90f5c94817d441158692a11f0e" Oct 30 16:56:50.191193 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:50.190979 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg" Oct 30 16:56:50.389235 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:50.389170 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:51.771319 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.771279 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj"] Oct 30 16:56:51.772211 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.772190 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5074fa9b-3fb9-4603-8c91-99bd249b0917" containerName="revoke-keys" Oct 30 16:56:51.772292 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.772213 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5074fa9b-3fb9-4603-8c91-99bd249b0917" containerName="revoke-keys" Oct 30 16:56:51.772396 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.772386 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5074fa9b-3fb9-4603-8c91-99bd249b0917" containerName="revoke-keys" Oct 30 16:56:51.776492 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.776474 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.799821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.799785 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj"] Oct 30 16:56:51.805611 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.805583 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-17f60f-openshift-default-dockercfg-65cvq\"" Oct 30 16:56:51.889215 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.889181 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-workload-certs\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.889457 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.889438 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-podinfo\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.889562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.889489 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-workload-socket\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.889650 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.889633 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-envoy\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.889704 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.889664 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-token\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.889829 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.889795 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-data\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.889977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.889914 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istiod-ca-cert\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.889977 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.889952 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmshr\" (UniqueName: \"kubernetes.io/projected/4cead7b4-5a36-4ed4-8550-3c454ace71a3-kube-api-access-wmshr\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.890153 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.890000 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-credential-socket\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.991326 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.991284 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-envoy\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.991326 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.991326 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-token\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.991658 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.991378 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-data\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.991658 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.991467 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istiod-ca-cert\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.991658 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.991500 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmshr\" (UniqueName: \"kubernetes.io/projected/4cead7b4-5a36-4ed4-8550-3c454ace71a3-kube-api-access-wmshr\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.991658 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.991543 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-credential-socket\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.991658 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.991613 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-workload-certs\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.992006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.991882 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-podinfo\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.992006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.991923 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-workload-socket\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.992006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.991954 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-data\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.992172 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.992023 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-credential-socket\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.992172 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.992151 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-workload-certs\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.992294 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.992247 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-workload-socket\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.992294 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.992288 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istiod-ca-cert\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.994535 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.994511 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-podinfo\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:51.994651 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:51.994594 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-envoy\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:52.001585 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:52.001516 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:52.015040 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:52.015014 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-token\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:52.015655 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:52.015637 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmshr\" (UniqueName: \"kubernetes.io/projected/4cead7b4-5a36-4ed4-8550-3c454ace71a3-kube-api-access-wmshr\") pod \"e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:52.080958 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:52.080918 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg"] Oct 30 16:56:52.099821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:52.095186 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71b55cf-26qkg"] Oct 30 16:56:52.099821 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:52.095494 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:52.413506 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:52.413428 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d90175c6-171e-41b4-9fe7-3370494ef31d" path="/var/lib/kubelet/pods/d90175c6-171e-41b4-9fe7-3370494ef31d/volumes" Oct 30 16:56:52.437625 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:52.437596 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj"] Oct 30 16:56:52.438238 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:56:52.438207 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cead7b4_5a36_4ed4_8550_3c454ace71a3.slice/crio-6d2d08694cd65a4f8893a582f21fb0f9344309a610c2e30d22c9e97b1291ec74 WatchSource:0}: Error finding container 6d2d08694cd65a4f8893a582f21fb0f9344309a610c2e30d22c9e97b1291ec74: Status 404 returned error can't find the container with id 6d2d08694cd65a4f8893a582f21fb0f9344309a610c2e30d22c9e97b1291ec74 Oct 30 16:56:52.441055 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:52.441019 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:56:52.441136 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:52.441089 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:56:52.441136 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:52.441118 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:56:53.224631 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:53.224596 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" event={"ID":"4cead7b4-5a36-4ed4-8550-3c454ace71a3","Type":"ContainerStarted","Data":"d663a7002638c3ee2565968279436a3b92be88aa1ed03a504f1839c2626c925c"} Oct 30 16:56:53.225133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:53.224654 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" event={"ID":"4cead7b4-5a36-4ed4-8550-3c454ace71a3","Type":"ContainerStarted","Data":"6d2d08694cd65a4f8893a582f21fb0f9344309a610c2e30d22c9e97b1291ec74"} Oct 30 16:56:53.296568 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:53.296517 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" podStartSLOduration=2.296502011 podStartE2EDuration="2.296502011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:56:53.294758523 +0000 UTC m=+2127.005213267" watchObservedRunningTime="2026-10-30 16:56:53.296502011 +0000 UTC m=+2127.006956754" Oct 30 16:56:53.428397 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:53.428355 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:53.953792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:53.952887 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" podUID="4c79977c-90b4-4d76-b6b4-b24aaf3df92b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:56:54.101785 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:54.100069 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:56:54.109991 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:54.107873 6202 patch_prober.go:29] interesting pod/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 16:56:54.109991 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:54.107940 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" podUID="4cead7b4-5a36-4ed4-8550-3c454ace71a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 16:56:54.176006 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:54.175962 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:54.365829 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:54.364798 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:54.387906 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:54.387870 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2"] Oct 30 16:56:54.400527 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:54.400494 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ee682-mngp2"] Oct 30 16:56:54.414536 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:54.414503 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97be2811-cc69-483b-8dcd-f31ed5ec8e5b" path="/var/lib/kubelet/pods/97be2811-cc69-483b-8dcd-f31ed5ec8e5b/volumes" Oct 30 16:56:54.480400 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:54.480347 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:54.691318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:54.689835 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:54.775795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:54.774011 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" Oct 30 16:56:54.897792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:54.893456 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c79977c-90b4-4d76-b6b4-b24aaf3df92b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c79977c-90b4-4d76-b6b4-b24aaf3df92b-plugins-config-volume\") pod \"4c79977c-90b4-4d76-b6b4-b24aaf3df92b\" (UID: \"4c79977c-90b4-4d76-b6b4-b24aaf3df92b\") " Oct 30 16:56:54.897792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:54.893635 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c79977c-90b4-4d76-b6b4-b24aaf3df92b-kube-api-access-7btft\" (UniqueName: \"kubernetes.io/projected/4c79977c-90b4-4d76-b6b4-b24aaf3df92b-kube-api-access-7btft\") pod \"4c79977c-90b4-4d76-b6b4-b24aaf3df92b\" (UID: \"4c79977c-90b4-4d76-b6b4-b24aaf3df92b\") " Oct 30 16:56:55.001953 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:54.912080 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c79977c-90b4-4d76-b6b4-b24aaf3df92b-plugins-config-volume" pod "4c79977c-90b4-4d76-b6b4-b24aaf3df92b" (UID: "4c79977c-90b4-4d76-b6b4-b24aaf3df92b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:56:55.001953 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:54.995439 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c79977c-90b4-4d76-b6b4-b24aaf3df92b-kube-api-access-7btft" pod "4c79977c-90b4-4d76-b6b4-b24aaf3df92b" (UID: "4c79977c-90b4-4d76-b6b4-b24aaf3df92b"). InnerVolumeSpecName "kube-api-access-7btft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:55.001953 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:54.996948 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7btft\" (UniqueName: \"kubernetes.io/projected/4c79977c-90b4-4d76-b6b4-b24aaf3df92b-kube-api-access-7btft\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:55.001953 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:54.996977 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c79977c-90b4-4d76-b6b4-b24aaf3df92b-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:56:55.002416 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:55.002319 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:55.106026 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:55.105555 6202 patch_prober.go:29] interesting pod/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 16:56:55.106026 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:55.105643 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" podUID="4cead7b4-5a36-4ed4-8550-3c454ace71a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 16:56:55.275442 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:55.275340 6202 generic.go:356] "Generic (PLEG): container finished" podID="4c79977c-90b4-4d76-b6b4-b24aaf3df92b" containerID="81e406a2b0c9d3c01725f0c6a3a92c20be48ca8983296dac27219f16e047ab15" exitCode=0 Oct 30 16:56:55.275442 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:55.275400 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" Oct 30 16:56:55.275442 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:55.275433 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" event={"ID":"4c79977c-90b4-4d76-b6b4-b24aaf3df92b","Type":"ContainerDied","Data":"81e406a2b0c9d3c01725f0c6a3a92c20be48ca8983296dac27219f16e047ab15"} Oct 30 16:56:55.275735 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:55.275489 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs" event={"ID":"4c79977c-90b4-4d76-b6b4-b24aaf3df92b","Type":"ContainerDied","Data":"d2bd4463af7327ca3a3e1cbdc3bd4a7bf2e66997d1ba23ea224c3edac4cc895e"} Oct 30 16:56:55.275735 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:55.275513 6202 scope.go:122] "RemoveContainer" containerID="81e406a2b0c9d3c01725f0c6a3a92c20be48ca8983296dac27219f16e047ab15" Oct 30 16:56:55.301401 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:55.301380 6202 scope.go:122] "RemoveContainer" containerID="81e406a2b0c9d3c01725f0c6a3a92c20be48ca8983296dac27219f16e047ab15" Oct 30 16:56:55.301743 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:56:55.301716 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e406a2b0c9d3c01725f0c6a3a92c20be48ca8983296dac27219f16e047ab15\": container with ID starting with 81e406a2b0c9d3c01725f0c6a3a92c20be48ca8983296dac27219f16e047ab15 not found: ID does not exist" containerID="81e406a2b0c9d3c01725f0c6a3a92c20be48ca8983296dac27219f16e047ab15" Oct 30 16:56:55.301844 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:55.301757 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e406a2b0c9d3c01725f0c6a3a92c20be48ca8983296dac27219f16e047ab15"} err="failed to get container status \"81e406a2b0c9d3c01725f0c6a3a92c20be48ca8983296dac27219f16e047ab15\": rpc error: code = NotFound desc = could not find container \"81e406a2b0c9d3c01725f0c6a3a92c20be48ca8983296dac27219f16e047ab15\": container with ID starting with 81e406a2b0c9d3c01725f0c6a3a92c20be48ca8983296dac27219f16e047ab15 not found: ID does not exist" Oct 30 16:56:55.348794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:55.348322 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs"] Oct 30 16:56:55.370825 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:55.357786 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ff1189-88d84f88b-btpbs"] Oct 30 16:56:56.099582 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:56.099540 6202 patch_prober.go:29] interesting pod/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 16:56:56.099822 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:56.099601 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" podUID="4cead7b4-5a36-4ed4-8550-3c454ace71a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 16:56:56.244793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:56.242966 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:56.415071 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:56.414990 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c79977c-90b4-4d76-b6b4-b24aaf3df92b" path="/var/lib/kubelet/pods/4c79977c-90b4-4d76-b6b4-b24aaf3df92b/volumes" Oct 30 16:56:56.817983 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:56.817883 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp"] Oct 30 16:56:56.936330 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:56.936236 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:57.122256 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:57.113409 6202 patch_prober.go:29] interesting pod/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 16:56:57.122256 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:57.113467 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" podUID="4cead7b4-5a36-4ed4-8550-3c454ace71a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 16:56:57.196142 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:57.187907 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:58.064912 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:58.064876 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:58.100287 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:58.100254 6202 patch_prober.go:29] interesting pod/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 16:56:58.100462 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:58.100319 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" podUID="4cead7b4-5a36-4ed4-8550-3c454ace71a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 16:56:59.099791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:59.099682 6202 patch_prober.go:29] interesting pod/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 16:56:59.099791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:59.099746 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" podUID="4cead7b4-5a36-4ed4-8550-3c454ace71a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 16:56:59.616573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:56:59.616539 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:00.104859 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:00.104826 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:57:00.105529 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:00.105508 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:57:00.106605 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:00.106585 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 16:57:01.353954 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:01.353898 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:01.500929 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:01.500880 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:01.824983 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:01.822925 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" podUID="bb3d3849-9fcb-414c-aece-8402ba38ab8d" containerName="payload-pre-processing" containerID="cri-o://ec0cbfb558beaf9bbbeba16624031b8e98b14d1f4b419699b53b492113055470" gracePeriod=25 Oct 30 16:57:01.911914 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:01.904865 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:03.079452 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:03.079421 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:04.574252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:04.574210 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:06.193280 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:06.193243 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:06.892199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:06.892166 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h"] Oct 30 16:57:06.892411 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:06.892387 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" podUID="6eac6c0b-6f5c-4e8c-8034-a045f3530426" containerName="istio-proxy" containerID="cri-o://07ccaeca3e5343427dad0b1bac0130ae5e0cb7bde596c44c4f61aa391ae49b94" gracePeriod=30 Oct 30 16:57:06.969973 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:06.969927 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" podUID="bb3d3849-9fcb-414c-aece-8402ba38ab8d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:57:07.668569 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:07.668532 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:09.334731 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:09.334698 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:10.855465 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:10.855427 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:10.971456 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:10.971419 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:12.194315 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.194290 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:57:12.273364 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.273332 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eac6c0b-6f5c-4e8c-8034-a045f3530426-kube-api-access-d5hs5\" (UniqueName: \"kubernetes.io/projected/6eac6c0b-6f5c-4e8c-8034-a045f3530426-kube-api-access-d5hs5\") pod \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " Oct 30 16:57:12.273547 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.273432 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-data\") pod \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " Oct 30 16:57:12.273547 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.273492 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-envoy\") pod \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " Oct 30 16:57:12.273679 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.273586 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-workload-certs\") pod \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " Oct 30 16:57:12.273679 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.273616 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-workload-socket\") pod \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " Oct 30 16:57:12.273818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.273686 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-data" pod "6eac6c0b-6f5c-4e8c-8034-a045f3530426" (UID: "6eac6c0b-6f5c-4e8c-8034-a045f3530426"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:12.273818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.273719 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-credential-socket\") pod \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " Oct 30 16:57:12.273941 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.273757 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-podinfo\") pod \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " Oct 30 16:57:12.273941 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.273919 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-workload-certs" pod "6eac6c0b-6f5c-4e8c-8034-a045f3530426" (UID: "6eac6c0b-6f5c-4e8c-8034-a045f3530426"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:12.274043 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.273946 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-workload-socket" pod "6eac6c0b-6f5c-4e8c-8034-a045f3530426" (UID: "6eac6c0b-6f5c-4e8c-8034-a045f3530426"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:12.274043 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.274026 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-token\" (UniqueName: \"kubernetes.io/projected/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-token\") pod \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " Oct 30 16:57:12.274148 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.274058 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istiod-ca-cert\") pod \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\" (UID: \"6eac6c0b-6f5c-4e8c-8034-a045f3530426\") " Oct 30 16:57:12.274148 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.274079 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-credential-socket" pod "6eac6c0b-6f5c-4e8c-8034-a045f3530426" (UID: "6eac6c0b-6f5c-4e8c-8034-a045f3530426"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:12.274485 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.274451 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istiod-ca-cert" pod "6eac6c0b-6f5c-4e8c-8034-a045f3530426" (UID: "6eac6c0b-6f5c-4e8c-8034-a045f3530426"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:57:12.274890 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.274830 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:12.274890 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.274857 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:12.274890 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.274872 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:12.275125 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.274883 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:12.275125 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.275110 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:12.276535 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.276511 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-podinfo" pod "6eac6c0b-6f5c-4e8c-8034-a045f3530426" (UID: "6eac6c0b-6f5c-4e8c-8034-a045f3530426"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:57:12.276974 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.276950 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-token" pod "6eac6c0b-6f5c-4e8c-8034-a045f3530426" (UID: "6eac6c0b-6f5c-4e8c-8034-a045f3530426"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:12.277402 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.277381 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-envoy" pod "6eac6c0b-6f5c-4e8c-8034-a045f3530426" (UID: "6eac6c0b-6f5c-4e8c-8034-a045f3530426"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:12.277702 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.277676 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eac6c0b-6f5c-4e8c-8034-a045f3530426-kube-api-access-d5hs5" pod "6eac6c0b-6f5c-4e8c-8034-a045f3530426" (UID: "6eac6c0b-6f5c-4e8c-8034-a045f3530426"). InnerVolumeSpecName "kube-api-access-d5hs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:12.326128 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.326084 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:12.380655 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.380594 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:12.383270 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.383247 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5hs5\" (UniqueName: \"kubernetes.io/projected/6eac6c0b-6f5c-4e8c-8034-a045f3530426-kube-api-access-d5hs5\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:12.383484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.383465 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:12.383596 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.383575 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eac6c0b-6f5c-4e8c-8034-a045f3530426-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:12.498426 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.498395 6202 generic.go:356] "Generic (PLEG): container finished" podID="6eac6c0b-6f5c-4e8c-8034-a045f3530426" containerID="07ccaeca3e5343427dad0b1bac0130ae5e0cb7bde596c44c4f61aa391ae49b94" exitCode=0 Oct 30 16:57:12.498614 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.498441 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" Oct 30 16:57:12.498614 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.498463 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" event={"ID":"6eac6c0b-6f5c-4e8c-8034-a045f3530426","Type":"ContainerDied","Data":"07ccaeca3e5343427dad0b1bac0130ae5e0cb7bde596c44c4f61aa391ae49b94"} Oct 30 16:57:12.498614 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.498508 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h" event={"ID":"6eac6c0b-6f5c-4e8c-8034-a045f3530426","Type":"ContainerDied","Data":"a54fdcf3254121bca01803a5ddce214d2856a4851790e7be09fb6ffe83c2c814"} Oct 30 16:57:12.498614 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.498532 6202 scope.go:122] "RemoveContainer" containerID="07ccaeca3e5343427dad0b1bac0130ae5e0cb7bde596c44c4f61aa391ae49b94" Oct 30 16:57:12.510533 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.510510 6202 scope.go:122] "RemoveContainer" containerID="07ccaeca3e5343427dad0b1bac0130ae5e0cb7bde596c44c4f61aa391ae49b94" Oct 30 16:57:12.510841 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:57:12.510820 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ccaeca3e5343427dad0b1bac0130ae5e0cb7bde596c44c4f61aa391ae49b94\": container with ID starting with 07ccaeca3e5343427dad0b1bac0130ae5e0cb7bde596c44c4f61aa391ae49b94 not found: ID does not exist" containerID="07ccaeca3e5343427dad0b1bac0130ae5e0cb7bde596c44c4f61aa391ae49b94" Oct 30 16:57:12.510912 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.510849 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ccaeca3e5343427dad0b1bac0130ae5e0cb7bde596c44c4f61aa391ae49b94"} err="failed to get container status \"07ccaeca3e5343427dad0b1bac0130ae5e0cb7bde596c44c4f61aa391ae49b94\": rpc error: code = NotFound desc = could not find container \"07ccaeca3e5343427dad0b1bac0130ae5e0cb7bde596c44c4f61aa391ae49b94\": container with ID starting with 07ccaeca3e5343427dad0b1bac0130ae5e0cb7bde596c44c4f61aa391ae49b94 not found: ID does not exist" Oct 30 16:57:12.554316 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.554288 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h"] Oct 30 16:57:12.560747 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.560718 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-kk25h"] Oct 30 16:57:12.614170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:12.614138 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:13.083949 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:13.083889 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:13.606872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:13.597118 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:14.056472 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.056391 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:14.077522 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:57:14.077463 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:57:14.342242 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.342210 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc"] Oct 30 16:57:14.343812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.343739 6202 cpu_manager.go:414] "containerMap: removing container" podUID="6eac6c0b-6f5c-4e8c-8034-a045f3530426" containerName="istio-proxy" Oct 30 16:57:14.343812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.343785 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eac6c0b-6f5c-4e8c-8034-a045f3530426" containerName="istio-proxy" Oct 30 16:57:14.344023 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.343840 6202 cpu_manager.go:414] "containerMap: removing container" podUID="4c79977c-90b4-4d76-b6b4-b24aaf3df92b" containerName="payload-processing" Oct 30 16:57:14.344023 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.343848 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c79977c-90b4-4d76-b6b4-b24aaf3df92b" containerName="payload-processing" Oct 30 16:57:14.344179 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.344051 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eac6c0b-6f5c-4e8c-8034-a045f3530426" containerName="istio-proxy" Oct 30 16:57:14.344179 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.344074 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c79977c-90b4-4d76-b6b4-b24aaf3df92b" containerName="payload-processing" Oct 30 16:57:14.349724 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.349699 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.368525 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.368499 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc"] Oct 30 16:57:14.372205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.372182 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-7c4b84\"/\"default-dockercfg-kjprt\"" Oct 30 16:57:14.372356 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.372198 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-7c4b84\"/\"auth-test-model-7c4b84-kserve-self-signed-certs\"" Oct 30 16:57:14.377692 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.377668 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-7c4b84\"/\"kube-root-ca.crt\"" Oct 30 16:57:14.378694 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.378677 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-7c4b84\"/\"openshift-service-ca.crt\"" Oct 30 16:57:14.405180 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.405137 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:14.415336 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.415307 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eac6c0b-6f5c-4e8c-8034-a045f3530426" path="/var/lib/kubelet/pods/6eac6c0b-6f5c-4e8c-8034-a045f3530426/volumes" Oct 30 16:57:14.425122 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.425096 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-home\") pod \"auth-test-model-7c4b84-kserve-59796579b9-95khc\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.425232 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.425143 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-tmp-dir\") pod \"auth-test-model-7c4b84-kserve-59796579b9-95khc\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.425280 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.425259 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-tls-certs\") pod \"auth-test-model-7c4b84-kserve-59796579b9-95khc\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.425352 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.425338 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-model-cache\") pod \"auth-test-model-7c4b84-kserve-59796579b9-95khc\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.425417 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.425403 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlbw2\" (UniqueName: \"kubernetes.io/projected/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-kube-api-access-dlbw2\") pod \"auth-test-model-7c4b84-kserve-59796579b9-95khc\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.425495 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.425484 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-dshm\") pod \"auth-test-model-7c4b84-kserve-59796579b9-95khc\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.526745 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.526716 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-tls-certs\") pod \"auth-test-model-7c4b84-kserve-59796579b9-95khc\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.526937 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.526794 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-model-cache\") pod \"auth-test-model-7c4b84-kserve-59796579b9-95khc\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.526937 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.526853 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlbw2\" (UniqueName: \"kubernetes.io/projected/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-kube-api-access-dlbw2\") pod \"auth-test-model-7c4b84-kserve-59796579b9-95khc\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.526937 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.526898 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-dshm\") pod \"auth-test-model-7c4b84-kserve-59796579b9-95khc\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.527099 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.527074 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-home\") pod \"auth-test-model-7c4b84-kserve-59796579b9-95khc\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.527278 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.527111 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-tmp-dir\") pod \"auth-test-model-7c4b84-kserve-59796579b9-95khc\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.527278 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.527231 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-model-cache\") pod \"auth-test-model-7c4b84-kserve-59796579b9-95khc\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.527435 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.527415 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-home\") pod \"auth-test-model-7c4b84-kserve-59796579b9-95khc\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.527667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.527479 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-tmp-dir\") pod \"auth-test-model-7c4b84-kserve-59796579b9-95khc\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.529255 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.529237 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-dshm\") pod \"auth-test-model-7c4b84-kserve-59796579b9-95khc\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.529686 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.529666 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-tls-certs\") pod \"auth-test-model-7c4b84-kserve-59796579b9-95khc\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.551177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.551150 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlbw2\" (UniqueName: \"kubernetes.io/projected/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-kube-api-access-dlbw2\") pod \"auth-test-model-7c4b84-kserve-59796579b9-95khc\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:14.681839 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:14.680830 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:15.054432 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:15.053925 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc"] Oct 30 16:57:15.527196 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:15.527161 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" event={"ID":"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6","Type":"ContainerStarted","Data":"b4c1685b048a5c527b3e4417252abeb4fd67583f1eb8fa17dad816b3e27acbf0"} Oct 30 16:57:15.527484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:15.527205 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" event={"ID":"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6","Type":"ContainerStarted","Data":"c9e6d67cbece072e6922983c9d20da9e86d4f111c42a42c8a6c9f0edabd8f2a9"} Oct 30 16:57:15.578943 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:15.578887 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" podStartSLOduration=1.5788717669999999 podStartE2EDuration="1.578871767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:15.574035152 +0000 UTC m=+2149.284489898" watchObservedRunningTime="2026-10-30 16:57:15.578871767 +0000 UTC m=+2149.289326510" Oct 30 16:57:15.681709 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:15.681672 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:15.890416 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:15.890344 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:16.968711 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:16.968667 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" podUID="bb3d3849-9fcb-414c-aece-8402ba38ab8d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:57:17.319383 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:17.319300 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:17.643539 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:17.643503 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:17.763355 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:17.763316 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:17.900466 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:17.900371 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:19.408490 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.408454 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:19.552473 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.552432 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:19.575884 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.575850 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk"] Oct 30 16:57:19.582528 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.581924 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" Oct 30 16:57:19.582697 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.582598 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7"] Oct 30 16:57:19.588502 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.588479 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" Oct 30 16:57:19.631871 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.631839 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-bfecdd\"" Oct 30 16:57:19.682857 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.675172 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/052f7204-67ca-4d46-8b48-5f44e951a8f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk\" (UID: \"052f7204-67ca-4d46-8b48-5f44e951a8f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" Oct 30 16:57:19.737140 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.737102 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdtws\" (UniqueName: \"kubernetes.io/projected/052f7204-67ca-4d46-8b48-5f44e951a8f1-kube-api-access-pdtws\") pod \"payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk\" (UID: \"052f7204-67ca-4d46-8b48-5f44e951a8f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" Oct 30 16:57:19.737501 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.737478 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf706793-9392-4041-8753-f65a7f1afe24-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7\" (UID: \"cf706793-9392-4041-8753-f65a7f1afe24\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" Oct 30 16:57:19.756793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.694023 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7"] Oct 30 16:57:19.756793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.734521 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-bfecdd-dockercfg-fxmjn\"" Oct 30 16:57:19.756793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.742966 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvz22\" (UniqueName: \"kubernetes.io/projected/cf706793-9392-4041-8753-f65a7f1afe24-kube-api-access-zvz22\") pod \"payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7\" (UID: \"cf706793-9392-4041-8753-f65a7f1afe24\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" Oct 30 16:57:19.884790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.856640 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvz22\" (UniqueName: \"kubernetes.io/projected/cf706793-9392-4041-8753-f65a7f1afe24-kube-api-access-zvz22\") pod \"payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7\" (UID: \"cf706793-9392-4041-8753-f65a7f1afe24\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" Oct 30 16:57:19.888797 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.885295 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/052f7204-67ca-4d46-8b48-5f44e951a8f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk\" (UID: \"052f7204-67ca-4d46-8b48-5f44e951a8f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" Oct 30 16:57:19.892156 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.892126 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdtws\" (UniqueName: \"kubernetes.io/projected/052f7204-67ca-4d46-8b48-5f44e951a8f1-kube-api-access-pdtws\") pod \"payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk\" (UID: \"052f7204-67ca-4d46-8b48-5f44e951a8f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" Oct 30 16:57:19.898986 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.898941 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf706793-9392-4041-8753-f65a7f1afe24-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7\" (UID: \"cf706793-9392-4041-8753-f65a7f1afe24\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" Oct 30 16:57:19.942976 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.900539 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf706793-9392-4041-8753-f65a7f1afe24-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7\" (UID: \"cf706793-9392-4041-8753-f65a7f1afe24\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" Oct 30 16:57:20.195034 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:19.944150 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/052f7204-67ca-4d46-8b48-5f44e951a8f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk\" (UID: \"052f7204-67ca-4d46-8b48-5f44e951a8f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" Oct 30 16:57:20.397796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:20.386368 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk"] Oct 30 16:57:20.432792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:20.417526 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdtws\" (UniqueName: \"kubernetes.io/projected/052f7204-67ca-4d46-8b48-5f44e951a8f1-kube-api-access-pdtws\") pod \"payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk\" (UID: \"052f7204-67ca-4d46-8b48-5f44e951a8f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" Oct 30 16:57:20.432792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:20.418358 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvz22\" (UniqueName: \"kubernetes.io/projected/cf706793-9392-4041-8753-f65a7f1afe24-kube-api-access-zvz22\") pod \"payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7\" (UID: \"cf706793-9392-4041-8753-f65a7f1afe24\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" Oct 30 16:57:20.564191 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:20.564093 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" Oct 30 16:57:20.591829 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:20.572594 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:20.592182 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:20.574134 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" Oct 30 16:57:21.091792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:21.086051 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:57:21.101791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:21.095090 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb"] Oct 30 16:57:21.101791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:21.095331 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" podUID="45641dd8-8e3a-4a54-872d-c7033e870619" containerName="istio-proxy" containerID="cri-o://1d33159d34621620b609288c1564981f16c04b7c2d6fb99bef60f461954cdf6d" gracePeriod=30 Oct 30 16:57:21.187868 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:21.174880 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded" Oct 30 16:57:21.187868 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:21.182995 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:57:21.187868 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:21.183527 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 16:57:21.187868 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:21.184303 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:57:21.203192 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:21.203147 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:57:22.000924 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:22.000833 6202 generic.go:356] "Generic (PLEG): container finished" podID="bb3d3849-9fcb-414c-aece-8402ba38ab8d" containerID="ec0cbfb558beaf9bbbeba16624031b8e98b14d1f4b419699b53b492113055470" exitCode=0 Oct 30 16:57:22.001306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:22.001089 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" event={"ID":"bb3d3849-9fcb-414c-aece-8402ba38ab8d","Type":"ContainerDied","Data":"ec0cbfb558beaf9bbbeba16624031b8e98b14d1f4b419699b53b492113055470"} Oct 30 16:57:22.080195 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:22.080167 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk"] Oct 30 16:57:22.084852 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:57:22.084809 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052f7204_67ca_4d46_8b48_5f44e951a8f1.slice/crio-b2c294902fb0e9e4c8b8db626b04347803fd9940f5422e8ad6bfca921b80d705 WatchSource:0}: Error finding container b2c294902fb0e9e4c8b8db626b04347803fd9940f5422e8ad6bfca921b80d705: Status 404 returned error can't find the container with id b2c294902fb0e9e4c8b8db626b04347803fd9940f5422e8ad6bfca921b80d705 Oct 30 16:57:22.136975 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:22.136939 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7"] Oct 30 16:57:22.137864 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:57:22.137565 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf706793_9392_4041_8753_f65a7f1afe24.slice/crio-c96f5c650e5d680e5d74b5a2d479f48d8b0c5def611f7847e74150d192634b4d WatchSource:0}: Error finding container c96f5c650e5d680e5d74b5a2d479f48d8b0c5def611f7847e74150d192634b4d: Status 404 returned error can't find the container with id c96f5c650e5d680e5d74b5a2d479f48d8b0c5def611f7847e74150d192634b4d Oct 30 16:57:22.307181 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:22.304659 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" Oct 30 16:57:22.432658 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:22.432625 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3d3849-9fcb-414c-aece-8402ba38ab8d-kube-api-access-9q4f5\" (UniqueName: \"kubernetes.io/projected/bb3d3849-9fcb-414c-aece-8402ba38ab8d-kube-api-access-9q4f5\") pod \"bb3d3849-9fcb-414c-aece-8402ba38ab8d\" (UID: \"bb3d3849-9fcb-414c-aece-8402ba38ab8d\") " Oct 30 16:57:22.432897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:22.432877 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3d3849-9fcb-414c-aece-8402ba38ab8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3d3849-9fcb-414c-aece-8402ba38ab8d-plugins-config-volume\") pod \"bb3d3849-9fcb-414c-aece-8402ba38ab8d\" (UID: \"bb3d3849-9fcb-414c-aece-8402ba38ab8d\") " Oct 30 16:57:22.433306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:22.433276 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3d3849-9fcb-414c-aece-8402ba38ab8d-plugins-config-volume" pod "bb3d3849-9fcb-414c-aece-8402ba38ab8d" (UID: "bb3d3849-9fcb-414c-aece-8402ba38ab8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:57:22.433561 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:22.433542 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3d3849-9fcb-414c-aece-8402ba38ab8d-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:22.434964 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:22.434945 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3d3849-9fcb-414c-aece-8402ba38ab8d-kube-api-access-9q4f5" pod "bb3d3849-9fcb-414c-aece-8402ba38ab8d" (UID: "bb3d3849-9fcb-414c-aece-8402ba38ab8d"). InnerVolumeSpecName "kube-api-access-9q4f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:22.535917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:22.535875 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q4f5\" (UniqueName: \"kubernetes.io/projected/bb3d3849-9fcb-414c-aece-8402ba38ab8d-kube-api-access-9q4f5\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:22.801935 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:22.801900 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:23.047791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:23.046660 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" event={"ID":"cf706793-9392-4041-8753-f65a7f1afe24","Type":"ContainerStarted","Data":"75b1301afbc1d6d370ce272a13cf03e3ca26b8949d5ccc25316e725ff22a56fd"} Oct 30 16:57:23.047791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:23.046828 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" event={"ID":"cf706793-9392-4041-8753-f65a7f1afe24","Type":"ContainerStarted","Data":"c96f5c650e5d680e5d74b5a2d479f48d8b0c5def611f7847e74150d192634b4d"} Oct 30 16:57:23.047791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:23.046887 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" Oct 30 16:57:23.048625 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:23.048595 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" event={"ID":"bb3d3849-9fcb-414c-aece-8402ba38ab8d","Type":"ContainerDied","Data":"0830b4e23a42ddd600ae11afdcf6f011a0b1abdb298d524c070b3312e886941b"} Oct 30 16:57:23.048849 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:23.048832 6202 scope.go:122] "RemoveContainer" containerID="ec0cbfb558beaf9bbbeba16624031b8e98b14d1f4b419699b53b492113055470" Oct 30 16:57:23.049110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:23.049097 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp" Oct 30 16:57:23.063146 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:23.059837 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" event={"ID":"052f7204-67ca-4d46-8b48-5f44e951a8f1","Type":"ContainerStarted","Data":"070515d09bed052bee8ec9fa902a8ba75eaa17ad24eb57f3f3a68b7959da17e5"} Oct 30 16:57:23.063146 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:23.059893 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" event={"ID":"052f7204-67ca-4d46-8b48-5f44e951a8f1","Type":"ContainerStarted","Data":"b2c294902fb0e9e4c8b8db626b04347803fd9940f5422e8ad6bfca921b80d705"} Oct 30 16:57:23.063146 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:23.060191 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" Oct 30 16:57:23.112178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:23.111539 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" podStartSLOduration=4.111520924 podStartE2EDuration="4.111520924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:23.111043177 +0000 UTC m=+2156.821497922" watchObservedRunningTime="2026-10-30 16:57:23.111520924 +0000 UTC m=+2156.821975669" Oct 30 16:57:23.151832 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:23.151798 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp"] Oct 30 16:57:23.158435 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:23.158411 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-480a60-d9c56f8b7-hljvp"] Oct 30 16:57:23.202420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:23.202342 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" podStartSLOduration=4.202324706 podStartE2EDuration="4.202324706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:23.200412253 +0000 UTC m=+2156.910866998" watchObservedRunningTime="2026-10-30 16:57:23.202324706 +0000 UTC m=+2156.912779458" Oct 30 16:57:24.416401 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:24.416362 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3d3849-9fcb-414c-aece-8402ba38ab8d" path="/var/lib/kubelet/pods/bb3d3849-9fcb-414c-aece-8402ba38ab8d/volumes" Oct 30 16:57:24.437065 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:24.437035 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:24.682155 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:24.681860 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:24.695579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:24.695549 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:25.094933 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:25.094906 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 16:57:25.982246 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:25.982210 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:26.375254 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.375226 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:57:26.516412 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.516321 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-workload-socket\") pod \"45641dd8-8e3a-4a54-872d-c7033e870619\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " Oct 30 16:57:26.516570 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.516450 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45641dd8-8e3a-4a54-872d-c7033e870619-istio-token\" (UniqueName: \"kubernetes.io/projected/45641dd8-8e3a-4a54-872d-c7033e870619-istio-token\") pod \"45641dd8-8e3a-4a54-872d-c7033e870619\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " Oct 30 16:57:26.516629 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.516614 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45641dd8-8e3a-4a54-872d-c7033e870619-kube-api-access-5znlf\" (UniqueName: \"kubernetes.io/projected/45641dd8-8e3a-4a54-872d-c7033e870619-kube-api-access-5znlf\") pod \"45641dd8-8e3a-4a54-872d-c7033e870619\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " Oct 30 16:57:26.516667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.516618 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-workload-socket" pod "45641dd8-8e3a-4a54-872d-c7033e870619" (UID: "45641dd8-8e3a-4a54-872d-c7033e870619"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:26.516824 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.516759 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45641dd8-8e3a-4a54-872d-c7033e870619-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45641dd8-8e3a-4a54-872d-c7033e870619-istio-podinfo\") pod \"45641dd8-8e3a-4a54-872d-c7033e870619\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " Oct 30 16:57:26.516928 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.516912 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-workload-certs\") pod \"45641dd8-8e3a-4a54-872d-c7033e870619\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " Oct 30 16:57:26.517078 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.517060 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45641dd8-8e3a-4a54-872d-c7033e870619-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45641dd8-8e3a-4a54-872d-c7033e870619-istiod-ca-cert\") pod \"45641dd8-8e3a-4a54-872d-c7033e870619\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " Oct 30 16:57:26.517252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.517227 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-workload-certs" pod "45641dd8-8e3a-4a54-872d-c7033e870619" (UID: "45641dd8-8e3a-4a54-872d-c7033e870619"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:26.518208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.517423 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45641dd8-8e3a-4a54-872d-c7033e870619-istiod-ca-cert" pod "45641dd8-8e3a-4a54-872d-c7033e870619" (UID: "45641dd8-8e3a-4a54-872d-c7033e870619"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:57:26.518208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.517460 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-credential-socket\") pod \"45641dd8-8e3a-4a54-872d-c7033e870619\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " Oct 30 16:57:26.518208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.517555 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-credential-socket" pod "45641dd8-8e3a-4a54-872d-c7033e870619" (UID: "45641dd8-8e3a-4a54-872d-c7033e870619"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:26.518208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.517576 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-istio-data\") pod \"45641dd8-8e3a-4a54-872d-c7033e870619\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " Oct 30 16:57:26.518208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.517620 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-istio-envoy\") pod \"45641dd8-8e3a-4a54-872d-c7033e870619\" (UID: \"45641dd8-8e3a-4a54-872d-c7033e870619\") " Oct 30 16:57:26.518208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.517953 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-istio-data" pod "45641dd8-8e3a-4a54-872d-c7033e870619" (UID: "45641dd8-8e3a-4a54-872d-c7033e870619"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:26.518644 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.518434 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:26.518644 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.518454 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45641dd8-8e3a-4a54-872d-c7033e870619-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:26.518644 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.518467 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:26.518644 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.518480 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:26.518644 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.518493 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:26.519842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.519805 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45641dd8-8e3a-4a54-872d-c7033e870619-kube-api-access-5znlf" pod "45641dd8-8e3a-4a54-872d-c7033e870619" (UID: "45641dd8-8e3a-4a54-872d-c7033e870619"). InnerVolumeSpecName "kube-api-access-5znlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:26.519964 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.519942 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45641dd8-8e3a-4a54-872d-c7033e870619-istio-token" pod "45641dd8-8e3a-4a54-872d-c7033e870619" (UID: "45641dd8-8e3a-4a54-872d-c7033e870619"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:26.520053 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.520033 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45641dd8-8e3a-4a54-872d-c7033e870619-istio-podinfo" pod "45641dd8-8e3a-4a54-872d-c7033e870619" (UID: "45641dd8-8e3a-4a54-872d-c7033e870619"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:57:26.521092 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.521068 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-istio-envoy" pod "45641dd8-8e3a-4a54-872d-c7033e870619" (UID: "45641dd8-8e3a-4a54-872d-c7033e870619"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:26.620071 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.620039 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45641dd8-8e3a-4a54-872d-c7033e870619-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:26.620071 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.620067 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45641dd8-8e3a-4a54-872d-c7033e870619-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:26.620071 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.620076 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45641dd8-8e3a-4a54-872d-c7033e870619-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:26.620306 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:26.620084 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5znlf\" (UniqueName: \"kubernetes.io/projected/45641dd8-8e3a-4a54-872d-c7033e870619-kube-api-access-5znlf\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:27.109689 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:27.109654 6202 generic.go:356] "Generic (PLEG): container finished" podID="45641dd8-8e3a-4a54-872d-c7033e870619" containerID="1d33159d34621620b609288c1564981f16c04b7c2d6fb99bef60f461954cdf6d" exitCode=0 Oct 30 16:57:27.110170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:27.109710 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" Oct 30 16:57:27.110170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:27.109743 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" event={"ID":"45641dd8-8e3a-4a54-872d-c7033e870619","Type":"ContainerDied","Data":"1d33159d34621620b609288c1564981f16c04b7c2d6fb99bef60f461954cdf6d"} Oct 30 16:57:27.110170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:27.109822 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb" event={"ID":"45641dd8-8e3a-4a54-872d-c7033e870619","Type":"ContainerDied","Data":"671fc68c4ac352e6e08db6249ffd691ff2be4da305a71b5d29a40aa4f6a4ba43"} Oct 30 16:57:27.110170 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:27.109851 6202 scope.go:122] "RemoveContainer" containerID="1d33159d34621620b609288c1564981f16c04b7c2d6fb99bef60f461954cdf6d" Oct 30 16:57:27.122053 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:27.122034 6202 scope.go:122] "RemoveContainer" containerID="1d33159d34621620b609288c1564981f16c04b7c2d6fb99bef60f461954cdf6d" Oct 30 16:57:27.122343 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:57:27.122320 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d33159d34621620b609288c1564981f16c04b7c2d6fb99bef60f461954cdf6d\": container with ID starting with 1d33159d34621620b609288c1564981f16c04b7c2d6fb99bef60f461954cdf6d not found: ID does not exist" containerID="1d33159d34621620b609288c1564981f16c04b7c2d6fb99bef60f461954cdf6d" Oct 30 16:57:27.122390 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:27.122351 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d33159d34621620b609288c1564981f16c04b7c2d6fb99bef60f461954cdf6d"} err="failed to get container status \"1d33159d34621620b609288c1564981f16c04b7c2d6fb99bef60f461954cdf6d\": rpc error: code = NotFound desc = could not find container \"1d33159d34621620b609288c1564981f16c04b7c2d6fb99bef60f461954cdf6d\": container with ID starting with 1d33159d34621620b609288c1564981f16c04b7c2d6fb99bef60f461954cdf6d not found: ID does not exist" Oct 30 16:57:27.176323 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:27.176281 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb"] Oct 30 16:57:27.188251 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:27.188222 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nx9kb"] Oct 30 16:57:27.575430 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:27.575393 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:28.413899 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:28.413863 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45641dd8-8e3a-4a54-872d-c7033e870619" path="/var/lib/kubelet/pods/45641dd8-8e3a-4a54-872d-c7033e870619/volumes" Oct 30 16:57:29.281468 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:29.281421 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:30.217016 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:30.216978 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:30.586301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:30.586260 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:31.246681 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:31.246624 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:31.370155 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:31.370104 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:31.394938 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:31.394905 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg"] Oct 30 16:57:31.406501 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:31.406472 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480a60-kzcfg"] Oct 30 16:57:31.510842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:31.509607 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:31.815698 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:31.815613 6202 scope.go:122] "RemoveContainer" containerID="00f11cabc6a2810ee94901abd0e45963fb186346725e2c9ccd62984f96bcbde5" Oct 30 16:57:32.412799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:32.412750 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5074fa9b-3fb9-4603-8c91-99bd249b0917" path="/var/lib/kubelet/pods/5074fa9b-3fb9-4603-8c91-99bd249b0917/volumes" Oct 30 16:57:33.152609 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:33.152573 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:34.076600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:34.076571 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" Oct 30 16:57:34.077095 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:34.076654 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" Oct 30 16:57:34.965248 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:34.965213 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:36.603044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:36.602957 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:38.279044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:38.279007 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:39.994010 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:39.993975 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:40.128191 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:40.128141 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:40.485642 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:40.485590 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:40.606923 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:40.606881 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:41.938088 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:41.938054 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:42.180367 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:42.180327 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:43.883205 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:43.883166 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:44.076957 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:57:44.076919 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:57:44.432928 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.432893 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj"] Oct 30 16:57:44.433778 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.433742 6202 cpu_manager.go:414] "containerMap: removing container" podUID="bb3d3849-9fcb-414c-aece-8402ba38ab8d" containerName="payload-pre-processing" Oct 30 16:57:44.433903 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.433782 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3d3849-9fcb-414c-aece-8402ba38ab8d" containerName="payload-pre-processing" Oct 30 16:57:44.433903 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.433801 6202 cpu_manager.go:414] "containerMap: removing container" podUID="45641dd8-8e3a-4a54-872d-c7033e870619" containerName="istio-proxy" Oct 30 16:57:44.433903 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.433806 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45641dd8-8e3a-4a54-872d-c7033e870619" containerName="istio-proxy" Oct 30 16:57:44.434073 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.433944 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3d3849-9fcb-414c-aece-8402ba38ab8d" containerName="payload-pre-processing" Oct 30 16:57:44.434073 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.433953 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="45641dd8-8e3a-4a54-872d-c7033e870619" containerName="istio-proxy" Oct 30 16:57:44.441215 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.441191 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.454068 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.454041 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istiod-ca-cert\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.454213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.454136 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-workload-certs\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.454213 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.454158 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-envoy\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.454341 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.454300 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-workload-socket\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.454411 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.454346 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-token\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.454497 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.454474 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-data\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.454559 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.454510 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmltb\" (UniqueName: \"kubernetes.io/projected/4775a051-8cf1-4771-ab83-0b31f6f9f40e-kube-api-access-vmltb\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.454620 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.454560 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-credential-socket\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.454826 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.454802 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-podinfo\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.465941 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.465917 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj"] Oct 30 16:57:44.473022 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.473000 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a59502-openshift-default-dockercfg-st59z\"" Oct 30 16:57:44.556720 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.556677 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-podinfo\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.556931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.556795 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istiod-ca-cert\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.556931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.556884 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-workload-certs\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.556931 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.556913 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-envoy\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.557139 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.557008 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-workload-socket\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.557139 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.557050 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-token\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.557139 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.557097 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-data\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.557139 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.557129 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmltb\" (UniqueName: \"kubernetes.io/projected/4775a051-8cf1-4771-ab83-0b31f6f9f40e-kube-api-access-vmltb\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.557374 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.557169 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-credential-socket\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.557557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.557527 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-credential-socket\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.557670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.557570 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-workload-certs\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.557670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.557604 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-data\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.557870 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.557846 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-workload-socket\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.558537 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.558511 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istiod-ca-cert\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.560131 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.560102 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-envoy\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.561008 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.560981 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-podinfo\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.581111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.581082 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-token\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.581697 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.581673 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmltb\" (UniqueName: \"kubernetes.io/projected/4775a051-8cf1-4771-ab83-0b31f6f9f40e-kube-api-access-vmltb\") pod \"e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.756455 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.756374 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:44.900060 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.900030 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb"] Oct 30 16:57:44.906260 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.906240 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb" Oct 30 16:57:44.922717 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.922693 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb"] Oct 30 16:57:44.923571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.923442 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:57:44.923571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.923510 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:57:44.964123 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.964084 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6brw\" (UniqueName: \"kubernetes.io/projected/f3638dac-29cc-4e5e-bc9b-3de82be10a96-kube-api-access-b6brw\") pod \"maas-api-revoke-sub-efa2c7f7-crqbb\" (UID: \"f3638dac-29cc-4e5e-bc9b-3de82be10a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb" Oct 30 16:57:44.964367 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.964342 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3638dac-29cc-4e5e-bc9b-3de82be10a96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa2c7f7-crqbb\" (UID: \"f3638dac-29cc-4e5e-bc9b-3de82be10a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb" Oct 30 16:57:44.964501 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:44.964480 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3638dac-29cc-4e5e-bc9b-3de82be10a96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa2c7f7-crqbb\" (UID: \"f3638dac-29cc-4e5e-bc9b-3de82be10a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb" Oct 30 16:57:45.066013 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:45.065918 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3638dac-29cc-4e5e-bc9b-3de82be10a96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa2c7f7-crqbb\" (UID: \"f3638dac-29cc-4e5e-bc9b-3de82be10a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb" Oct 30 16:57:45.066199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:45.066056 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6brw\" (UniqueName: \"kubernetes.io/projected/f3638dac-29cc-4e5e-bc9b-3de82be10a96-kube-api-access-b6brw\") pod \"maas-api-revoke-sub-efa2c7f7-crqbb\" (UID: \"f3638dac-29cc-4e5e-bc9b-3de82be10a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb" Oct 30 16:57:45.066277 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:45.066252 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3638dac-29cc-4e5e-bc9b-3de82be10a96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa2c7f7-crqbb\" (UID: \"f3638dac-29cc-4e5e-bc9b-3de82be10a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb" Oct 30 16:57:45.066647 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:45.066626 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3638dac-29cc-4e5e-bc9b-3de82be10a96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa2c7f7-crqbb\" (UID: \"f3638dac-29cc-4e5e-bc9b-3de82be10a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb" Oct 30 16:57:45.068867 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:45.068841 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3638dac-29cc-4e5e-bc9b-3de82be10a96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa2c7f7-crqbb\" (UID: \"f3638dac-29cc-4e5e-bc9b-3de82be10a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb" Oct 30 16:57:45.086339 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:57:45.086304 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4775a051_8cf1_4771_ab83_0b31f6f9f40e.slice/crio-93cdca363853d97caa04ef9a23f6d96655098ac3b6e2e1f22fc4428af2d358eb WatchSource:0}: Error finding container 93cdca363853d97caa04ef9a23f6d96655098ac3b6e2e1f22fc4428af2d358eb: Status 404 returned error can't find the container with id 93cdca363853d97caa04ef9a23f6d96655098ac3b6e2e1f22fc4428af2d358eb Oct 30 16:57:45.087986 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:45.087962 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj"] Oct 30 16:57:45.089681 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:45.089643 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:57:45.089842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:45.089721 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:57:45.089842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:45.089754 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:57:45.090066 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:45.090048 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6brw\" (UniqueName: \"kubernetes.io/projected/f3638dac-29cc-4e5e-bc9b-3de82be10a96-kube-api-access-b6brw\") pod \"maas-api-revoke-sub-efa2c7f7-crqbb\" (UID: \"f3638dac-29cc-4e5e-bc9b-3de82be10a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb" Oct 30 16:57:45.229942 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:45.229910 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb" Oct 30 16:57:45.361892 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:45.361851 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" event={"ID":"4775a051-8cf1-4771-ab83-0b31f6f9f40e","Type":"ContainerStarted","Data":"59fdb85be3a3aac56d9ef48c7fa6adea7eb8f7ae4f30b1099a15723f2a94f38d"} Oct 30 16:57:45.362088 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:45.361933 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" event={"ID":"4775a051-8cf1-4771-ab83-0b31f6f9f40e","Type":"ContainerStarted","Data":"93cdca363853d97caa04ef9a23f6d96655098ac3b6e2e1f22fc4428af2d358eb"} Oct 30 16:57:45.412274 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:45.412224 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" podStartSLOduration=1.412209161 podStartE2EDuration="1.412209161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:45.40833049 +0000 UTC m=+2179.118785246" watchObservedRunningTime="2026-10-30 16:57:45.412209161 +0000 UTC m=+2179.122663905" Oct 30 16:57:45.558089 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:57:45.558046 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3638dac_29cc_4e5e_bc9b_3de82be10a96.slice/crio-ffe68c8580ccc743aa1a69fcca288fbcbbc6b5b35a10d3aaf33d8bba5429f21a WatchSource:0}: Error finding container ffe68c8580ccc743aa1a69fcca288fbcbbc6b5b35a10d3aaf33d8bba5429f21a: Status 404 returned error can't find the container with id ffe68c8580ccc743aa1a69fcca288fbcbbc6b5b35a10d3aaf33d8bba5429f21a Oct 30 16:57:45.559286 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:45.559051 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb"] Oct 30 16:57:45.667580 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:45.667542 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:45.767793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:45.762866 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:46.376368 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:46.376325 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb" event={"ID":"f3638dac-29cc-4e5e-bc9b-3de82be10a96","Type":"ContainerStarted","Data":"4132baddde47a724fc3ece7d40a9e318dda0ec3dbcc511b8636d9aa0867521f3"} Oct 30 16:57:46.376799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:46.376384 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb" event={"ID":"f3638dac-29cc-4e5e-bc9b-3de82be10a96","Type":"ContainerStarted","Data":"ffe68c8580ccc743aa1a69fcca288fbcbbc6b5b35a10d3aaf33d8bba5429f21a"} Oct 30 16:57:46.417918 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:46.417868 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb" podStartSLOduration=2.417854215 podStartE2EDuration="2.417854215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:46.416511715 +0000 UTC m=+2180.126966437" watchObservedRunningTime="2026-10-30 16:57:46.417854215 +0000 UTC m=+2180.128308958" Oct 30 16:57:46.770793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:46.769088 6202 patch_prober.go:29] interesting pod/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 30 16:57:46.770793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:46.769168 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" podUID="4775a051-8cf1-4771-ab83-0b31f6f9f40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 30 16:57:47.288509 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:47.288472 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:47.415799 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:47.415747 6202 generic.go:356] "Generic (PLEG): container finished" podID="f3638dac-29cc-4e5e-bc9b-3de82be10a96" containerID="4132baddde47a724fc3ece7d40a9e318dda0ec3dbcc511b8636d9aa0867521f3" exitCode=0 Oct 30 16:57:47.416252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:47.416223 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb" event={"ID":"f3638dac-29cc-4e5e-bc9b-3de82be10a96","Type":"ContainerDied","Data":"4132baddde47a724fc3ece7d40a9e318dda0ec3dbcc511b8636d9aa0867521f3"} Oct 30 16:57:47.757359 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:47.757319 6202 patch_prober.go:29] interesting pod/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 30 16:57:47.757685 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:47.757640 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" podUID="4775a051-8cf1-4771-ab83-0b31f6f9f40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 30 16:57:48.685995 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:48.685922 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb" Oct 30 16:57:48.759793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:48.746002 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3638dac-29cc-4e5e-bc9b-3de82be10a96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3638dac-29cc-4e5e-bc9b-3de82be10a96-maas-api-xks-ca\") pod \"f3638dac-29cc-4e5e-bc9b-3de82be10a96\" (UID: \"f3638dac-29cc-4e5e-bc9b-3de82be10a96\") " Oct 30 16:57:48.759793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:48.746210 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3638dac-29cc-4e5e-bc9b-3de82be10a96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3638dac-29cc-4e5e-bc9b-3de82be10a96-maas-api-openshift-ca\") pod \"f3638dac-29cc-4e5e-bc9b-3de82be10a96\" (UID: \"f3638dac-29cc-4e5e-bc9b-3de82be10a96\") " Oct 30 16:57:48.759793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:48.746458 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3638dac-29cc-4e5e-bc9b-3de82be10a96-kube-api-access-b6brw\" (UniqueName: \"kubernetes.io/projected/f3638dac-29cc-4e5e-bc9b-3de82be10a96-kube-api-access-b6brw\") pod \"f3638dac-29cc-4e5e-bc9b-3de82be10a96\" (UID: \"f3638dac-29cc-4e5e-bc9b-3de82be10a96\") " Oct 30 16:57:48.783794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:48.761950 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3638dac-29cc-4e5e-bc9b-3de82be10a96-maas-api-openshift-ca" pod "f3638dac-29cc-4e5e-bc9b-3de82be10a96" (UID: "f3638dac-29cc-4e5e-bc9b-3de82be10a96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:57:48.783794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:48.762453 6202 patch_prober.go:29] interesting pod/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 30 16:57:48.783794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:48.762506 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" podUID="4775a051-8cf1-4771-ab83-0b31f6f9f40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 30 16:57:48.783794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:48.774975 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3638dac-29cc-4e5e-bc9b-3de82be10a96-maas-api-xks-ca" pod "f3638dac-29cc-4e5e-bc9b-3de82be10a96" (UID: "f3638dac-29cc-4e5e-bc9b-3de82be10a96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:57:48.796336 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:48.791494 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3638dac-29cc-4e5e-bc9b-3de82be10a96-kube-api-access-b6brw" pod "f3638dac-29cc-4e5e-bc9b-3de82be10a96" (UID: "f3638dac-29cc-4e5e-bc9b-3de82be10a96"). InnerVolumeSpecName "kube-api-access-b6brw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:48.850794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:48.850395 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3638dac-29cc-4e5e-bc9b-3de82be10a96-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:48.850794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:48.850463 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6brw\" (UniqueName: \"kubernetes.io/projected/f3638dac-29cc-4e5e-bc9b-3de82be10a96-kube-api-access-b6brw\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:48.850794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:48.850481 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3638dac-29cc-4e5e-bc9b-3de82be10a96-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:48.975246 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:48.975143 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:49.319740 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:49.319650 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:49.437133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:49.437093 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb" Oct 30 16:57:49.437308 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:49.437101 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb" event={"ID":"f3638dac-29cc-4e5e-bc9b-3de82be10a96","Type":"ContainerDied","Data":"ffe68c8580ccc743aa1a69fcca288fbcbbc6b5b35a10d3aaf33d8bba5429f21a"} Oct 30 16:57:49.437308 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:49.437260 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffe68c8580ccc743aa1a69fcca288fbcbbc6b5b35a10d3aaf33d8bba5429f21a" Oct 30 16:57:49.503013 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:49.502979 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:49.726320 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:49.726281 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f"] Oct 30 16:57:49.726865 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:49.726545 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" podUID="3554e861-457c-4806-99ee-ad9e27786362" containerName="istio-proxy" containerID="cri-o://cad1712e838221e01ef37e03a0caf99d554bbaa897a1c914f260d016ddcb1a14" gracePeriod=30 Oct 30 16:57:49.777333 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:49.777301 6202 patch_prober.go:29] interesting pod/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 30 16:57:49.777585 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:49.777555 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" podUID="4775a051-8cf1-4771-ab83-0b31f6f9f40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 30 16:57:49.778505 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:49.778452 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:50.762793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:50.761890 6202 patch_prober.go:29] interesting pod/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 30 16:57:50.762793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:50.761962 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" podUID="4775a051-8cf1-4771-ab83-0b31f6f9f40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 30 16:57:50.988116 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:50.988060 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:51.820832 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:51.820782 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:51.822588 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:51.822563 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:51.826322 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:51.826298 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:57:51.845987 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:51.845948 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:52.621286 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:52.621250 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:54.224309 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:54.224272 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:55.048128 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.048098 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:57:55.123297 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.123260 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-istio-data\") pod \"3554e861-457c-4806-99ee-ad9e27786362\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " Oct 30 16:57:55.123297 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.123301 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3554e861-457c-4806-99ee-ad9e27786362-kube-api-access-nmt96\" (UniqueName: \"kubernetes.io/projected/3554e861-457c-4806-99ee-ad9e27786362-kube-api-access-nmt96\") pod \"3554e861-457c-4806-99ee-ad9e27786362\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " Oct 30 16:57:55.123562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.123370 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3554e861-457c-4806-99ee-ad9e27786362-istio-token\" (UniqueName: \"kubernetes.io/projected/3554e861-457c-4806-99ee-ad9e27786362-istio-token\") pod \"3554e861-457c-4806-99ee-ad9e27786362\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " Oct 30 16:57:55.123562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.123485 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-workload-socket\") pod \"3554e861-457c-4806-99ee-ad9e27786362\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " Oct 30 16:57:55.123562 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.123546 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-istio-data" pod "3554e861-457c-4806-99ee-ad9e27786362" (UID: "3554e861-457c-4806-99ee-ad9e27786362"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:55.123708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.123564 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-istio-envoy\") pod \"3554e861-457c-4806-99ee-ad9e27786362\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " Oct 30 16:57:55.123708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.123620 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3554e861-457c-4806-99ee-ad9e27786362-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3554e861-457c-4806-99ee-ad9e27786362-istio-podinfo\") pod \"3554e861-457c-4806-99ee-ad9e27786362\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " Oct 30 16:57:55.123827 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.123722 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-workload-socket" pod "3554e861-457c-4806-99ee-ad9e27786362" (UID: "3554e861-457c-4806-99ee-ad9e27786362"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:55.123878 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.123822 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3554e861-457c-4806-99ee-ad9e27786362-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3554e861-457c-4806-99ee-ad9e27786362-istiod-ca-cert\") pod \"3554e861-457c-4806-99ee-ad9e27786362\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " Oct 30 16:57:55.124049 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.124020 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-workload-certs\") pod \"3554e861-457c-4806-99ee-ad9e27786362\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " Oct 30 16:57:55.124201 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.124070 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-credential-socket\") pod \"3554e861-457c-4806-99ee-ad9e27786362\" (UID: \"3554e861-457c-4806-99ee-ad9e27786362\") " Oct 30 16:57:55.124497 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.124261 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-workload-certs" pod "3554e861-457c-4806-99ee-ad9e27786362" (UID: "3554e861-457c-4806-99ee-ad9e27786362"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:55.124497 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.124289 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3554e861-457c-4806-99ee-ad9e27786362-istiod-ca-cert" pod "3554e861-457c-4806-99ee-ad9e27786362" (UID: "3554e861-457c-4806-99ee-ad9e27786362"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:57:55.124497 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.124474 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-credential-socket" pod "3554e861-457c-4806-99ee-ad9e27786362" (UID: "3554e861-457c-4806-99ee-ad9e27786362"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:55.125076 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.125046 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3554e861-457c-4806-99ee-ad9e27786362-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:55.125187 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.125082 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:55.125187 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.125097 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:55.125187 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.125113 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:55.125187 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.125128 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:55.126220 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.126197 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3554e861-457c-4806-99ee-ad9e27786362-istio-token" pod "3554e861-457c-4806-99ee-ad9e27786362" (UID: "3554e861-457c-4806-99ee-ad9e27786362"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:55.126337 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.126311 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3554e861-457c-4806-99ee-ad9e27786362-istio-podinfo" pod "3554e861-457c-4806-99ee-ad9e27786362" (UID: "3554e861-457c-4806-99ee-ad9e27786362"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:57:55.126404 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.126333 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-istio-envoy" pod "3554e861-457c-4806-99ee-ad9e27786362" (UID: "3554e861-457c-4806-99ee-ad9e27786362"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:55.126894 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.126874 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3554e861-457c-4806-99ee-ad9e27786362-kube-api-access-nmt96" pod "3554e861-457c-4806-99ee-ad9e27786362" (UID: "3554e861-457c-4806-99ee-ad9e27786362"). InnerVolumeSpecName "kube-api-access-nmt96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:55.226488 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.226455 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3554e861-457c-4806-99ee-ad9e27786362-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:55.226488 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.226486 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmt96\" (UniqueName: \"kubernetes.io/projected/3554e861-457c-4806-99ee-ad9e27786362-kube-api-access-nmt96\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:55.226488 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.226498 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3554e861-457c-4806-99ee-ad9e27786362-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:55.226983 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.226506 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3554e861-457c-4806-99ee-ad9e27786362-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:57:55.531412 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.531371 6202 generic.go:356] "Generic (PLEG): container finished" podID="3554e861-457c-4806-99ee-ad9e27786362" containerID="cad1712e838221e01ef37e03a0caf99d554bbaa897a1c914f260d016ddcb1a14" exitCode=0 Oct 30 16:57:55.531600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.531433 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" Oct 30 16:57:55.531600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.531458 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" event={"ID":"3554e861-457c-4806-99ee-ad9e27786362","Type":"ContainerDied","Data":"cad1712e838221e01ef37e03a0caf99d554bbaa897a1c914f260d016ddcb1a14"} Oct 30 16:57:55.531600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.531516 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f" event={"ID":"3554e861-457c-4806-99ee-ad9e27786362","Type":"ContainerDied","Data":"29d7dba35aaf5d9afc5991cca87b64f74d995838170310ff35b3f4cb75406839"} Oct 30 16:57:55.531600 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.531542 6202 scope.go:122] "RemoveContainer" containerID="cad1712e838221e01ef37e03a0caf99d554bbaa897a1c914f260d016ddcb1a14" Oct 30 16:57:55.544685 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.544657 6202 scope.go:122] "RemoveContainer" containerID="08461321eda010962c4ec4af030200c3f2ca572dcb0544a345b68550ea5b7a08" Oct 30 16:57:55.560387 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.560363 6202 scope.go:122] "RemoveContainer" containerID="cad1712e838221e01ef37e03a0caf99d554bbaa897a1c914f260d016ddcb1a14" Oct 30 16:57:55.560792 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:57:55.560751 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad1712e838221e01ef37e03a0caf99d554bbaa897a1c914f260d016ddcb1a14\": container with ID starting with cad1712e838221e01ef37e03a0caf99d554bbaa897a1c914f260d016ddcb1a14 not found: ID does not exist" containerID="cad1712e838221e01ef37e03a0caf99d554bbaa897a1c914f260d016ddcb1a14" Oct 30 16:57:55.560874 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.560804 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad1712e838221e01ef37e03a0caf99d554bbaa897a1c914f260d016ddcb1a14"} err="failed to get container status \"cad1712e838221e01ef37e03a0caf99d554bbaa897a1c914f260d016ddcb1a14\": rpc error: code = NotFound desc = could not find container \"cad1712e838221e01ef37e03a0caf99d554bbaa897a1c914f260d016ddcb1a14\": container with ID starting with cad1712e838221e01ef37e03a0caf99d554bbaa897a1c914f260d016ddcb1a14 not found: ID does not exist" Oct 30 16:57:55.560921 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.560888 6202 scope.go:122] "RemoveContainer" containerID="08461321eda010962c4ec4af030200c3f2ca572dcb0544a345b68550ea5b7a08" Oct 30 16:57:55.561282 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:57:55.561255 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08461321eda010962c4ec4af030200c3f2ca572dcb0544a345b68550ea5b7a08\": container with ID starting with 08461321eda010962c4ec4af030200c3f2ca572dcb0544a345b68550ea5b7a08 not found: ID does not exist" containerID="08461321eda010962c4ec4af030200c3f2ca572dcb0544a345b68550ea5b7a08" Oct 30 16:57:55.561393 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.561292 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08461321eda010962c4ec4af030200c3f2ca572dcb0544a345b68550ea5b7a08"} err="failed to get container status \"08461321eda010962c4ec4af030200c3f2ca572dcb0544a345b68550ea5b7a08\": rpc error: code = NotFound desc = could not find container \"08461321eda010962c4ec4af030200c3f2ca572dcb0544a345b68550ea5b7a08\": container with ID starting with 08461321eda010962c4ec4af030200c3f2ca572dcb0544a345b68550ea5b7a08 not found: ID does not exist" Oct 30 16:57:55.587786 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.587728 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f"] Oct 30 16:57:55.599618 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.599582 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-480a60-openshift-default-fb894cdbc-8tt8f"] Oct 30 16:57:55.945907 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:55.945869 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:56.412952 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:56.412919 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3554e861-457c-4806-99ee-ad9e27786362" path="/var/lib/kubelet/pods/3554e861-457c-4806-99ee-ad9e27786362/volumes" Oct 30 16:57:57.565872 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:57.565840 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:59.224539 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:57:59.213174 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:00.873564 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:00.873525 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:02.442492 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:02.442457 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:03.927796 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:03.927740 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:04.044879 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:04.044839 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:05.735875 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:05.735833 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:07.450321 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:07.450281 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:08.969623 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:08.969583 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:10.620966 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:10.620927 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:12.142579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:12.142528 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:13.873201 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:13.873162 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:14.072969 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:58:14.072931 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:58:15.863733 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:15.863693 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:17.480156 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:17.480118 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:19.055601 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:19.055565 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:20.894549 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:20.894510 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:22.372487 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:22.372441 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:23.249131 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:23.248969 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:24.050268 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:24.050231 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:25.532199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:25.532172 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:25.975752 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:25.975714 6202 generic.go:356] "Generic (PLEG): container finished" podID="9720d030-7806-479f-840c-38401180d116" containerID="cf0bdbadc7890ddb7d870142156a3ced9ac0fbd159a375d3a80d5ff0a2e8529e" exitCode=0 Oct 30 16:58:25.975993 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:25.975803 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" event={"ID":"9720d030-7806-479f-840c-38401180d116","Type":"ContainerDied","Data":"cf0bdbadc7890ddb7d870142156a3ced9ac0fbd159a375d3a80d5ff0a2e8529e"} Oct 30 16:58:25.976375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:25.976356 6202 scope.go:122] "RemoveContainer" containerID="cf0bdbadc7890ddb7d870142156a3ced9ac0fbd159a375d3a80d5ff0a2e8529e" Oct 30 16:58:25.978468 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:25.978436 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:58:25.978586 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:25.978508 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:58:25.978586 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:25.978559 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:58:26.987830 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:26.987792 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" event={"ID":"9720d030-7806-479f-840c-38401180d116","Type":"ContainerStarted","Data":"2f0baa4e01f05c6df1e9d110478272b8f49779d7f8d84d7741db5676101d1014"} Oct 30 16:58:27.114211 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:27.114174 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:27.238983 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:27.235696 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:27.431522 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:27.431487 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:58:27.432911 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:27.432878 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 16:58:27.433071 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:27.432942 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" podUID="9720d030-7806-479f-840c-38401180d116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 16:58:28.431757 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:28.431715 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 16:58:28.432272 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:28.431821 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" podUID="9720d030-7806-479f-840c-38401180d116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 16:58:28.850794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:28.848437 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:29.432186 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:29.432141 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 16:58:29.432614 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:29.432216 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" podUID="9720d030-7806-479f-840c-38401180d116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 16:58:30.167791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:30.166878 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:58:30.342438 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:30.342401 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:30.431761 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:30.431672 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 16:58:30.431761 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:30.431733 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" podUID="9720d030-7806-479f-840c-38401180d116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 16:58:31.432036 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:31.431998 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 16:58:31.432481 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:31.432053 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" podUID="9720d030-7806-479f-840c-38401180d116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 16:58:31.918198 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:31.918167 6202 scope.go:122] "RemoveContainer" containerID="1f0f5ff7273daaef38573e8dba2c0dfd2381a7b2c203e0082418ae4a5dcf60be" Oct 30 16:58:31.933706 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:31.933687 6202 scope.go:122] "RemoveContainer" containerID="4938d26d5e49169ffd8473950da9b7cc4d114d9c3b43c80e726f0956633a76f3" Oct 30 16:58:31.972522 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:31.963370 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:32.432047 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:32.432008 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 16:58:32.432450 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:32.432087 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" podUID="9720d030-7806-479f-840c-38401180d116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 16:58:33.437231 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.437199 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:58:33.438461 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.438435 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 16:58:33.517204 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.517170 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:33.757028 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.756942 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf"] Oct 30 16:58:33.757934 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.757911 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3554e861-457c-4806-99ee-ad9e27786362" containerName="istio-proxy" Oct 30 16:58:33.757934 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.757933 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3554e861-457c-4806-99ee-ad9e27786362" containerName="istio-proxy" Oct 30 16:58:33.758133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.757962 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3554e861-457c-4806-99ee-ad9e27786362" containerName="istio-proxy" Oct 30 16:58:33.758133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.757970 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3554e861-457c-4806-99ee-ad9e27786362" containerName="istio-proxy" Oct 30 16:58:33.758133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.758005 6202 cpu_manager.go:414] "containerMap: removing container" podUID="f3638dac-29cc-4e5e-bc9b-3de82be10a96" containerName="revoke-keys" Oct 30 16:58:33.758133 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.758012 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3638dac-29cc-4e5e-bc9b-3de82be10a96" containerName="revoke-keys" Oct 30 16:58:33.758343 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.758195 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3554e861-457c-4806-99ee-ad9e27786362" containerName="istio-proxy" Oct 30 16:58:33.758343 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.758214 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3554e861-457c-4806-99ee-ad9e27786362" containerName="istio-proxy" Oct 30 16:58:33.758343 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.758229 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3638dac-29cc-4e5e-bc9b-3de82be10a96" containerName="revoke-keys" Oct 30 16:58:33.762458 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.762437 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf" Oct 30 16:58:33.782847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.780112 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:58:33.783146 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.783118 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:58:33.794155 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.794121 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf"] Oct 30 16:58:33.878260 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.878220 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc229aa-6e25-47b4-939d-23d34a39cd30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36323b68-5hvgf\" (UID: \"6bc229aa-6e25-47b4-939d-23d34a39cd30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf" Oct 30 16:58:33.878504 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.878477 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc229aa-6e25-47b4-939d-23d34a39cd30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36323b68-5hvgf\" (UID: \"6bc229aa-6e25-47b4-939d-23d34a39cd30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf" Oct 30 16:58:33.878614 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.878599 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv5sq\" (UniqueName: \"kubernetes.io/projected/6bc229aa-6e25-47b4-939d-23d34a39cd30-kube-api-access-nv5sq\") pod \"maas-api-revoke-sub-36323b68-5hvgf\" (UID: \"6bc229aa-6e25-47b4-939d-23d34a39cd30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf" Oct 30 16:58:33.980045 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.980007 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc229aa-6e25-47b4-939d-23d34a39cd30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36323b68-5hvgf\" (UID: \"6bc229aa-6e25-47b4-939d-23d34a39cd30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf" Oct 30 16:58:33.980236 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.980113 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv5sq\" (UniqueName: \"kubernetes.io/projected/6bc229aa-6e25-47b4-939d-23d34a39cd30-kube-api-access-nv5sq\") pod \"maas-api-revoke-sub-36323b68-5hvgf\" (UID: \"6bc229aa-6e25-47b4-939d-23d34a39cd30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf" Oct 30 16:58:33.980325 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.980293 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc229aa-6e25-47b4-939d-23d34a39cd30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36323b68-5hvgf\" (UID: \"6bc229aa-6e25-47b4-939d-23d34a39cd30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf" Oct 30 16:58:33.980820 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.980786 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc229aa-6e25-47b4-939d-23d34a39cd30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36323b68-5hvgf\" (UID: \"6bc229aa-6e25-47b4-939d-23d34a39cd30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf" Oct 30 16:58:33.983037 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:33.983015 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc229aa-6e25-47b4-939d-23d34a39cd30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36323b68-5hvgf\" (UID: \"6bc229aa-6e25-47b4-939d-23d34a39cd30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf" Oct 30 16:58:34.005309 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:34.005286 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv5sq\" (UniqueName: \"kubernetes.io/projected/6bc229aa-6e25-47b4-939d-23d34a39cd30-kube-api-access-nv5sq\") pod \"maas-api-revoke-sub-36323b68-5hvgf\" (UID: \"6bc229aa-6e25-47b4-939d-23d34a39cd30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf" Oct 30 16:58:34.103524 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:34.103502 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf" Oct 30 16:58:34.415154 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:34.415123 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf"] Oct 30 16:58:34.415154 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:58:34.415138 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc229aa_6e25_47b4_939d_23d34a39cd30.slice/crio-c1959744ab0fa28e37989b3d15e8d4caca323444f896789bc7d3270911df0c8e WatchSource:0}: Error finding container c1959744ab0fa28e37989b3d15e8d4caca323444f896789bc7d3270911df0c8e: Status 404 returned error can't find the container with id c1959744ab0fa28e37989b3d15e8d4caca323444f896789bc7d3270911df0c8e Oct 30 16:58:34.902207 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:34.902174 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25"] Oct 30 16:58:34.906627 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:34.906605 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:34.921315 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:34.921289 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f92caf-openshift-default-dockercfg-cxlmb\"" Oct 30 16:58:34.937661 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:34.937633 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25"] Oct 30 16:58:35.002342 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.002309 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-podinfo\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.002529 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.002402 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-workload-socket\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.002613 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.002551 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-workload-certs\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.002613 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.002597 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-data\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.002729 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.002670 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istiod-ca-cert\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.002826 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.002735 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqj7v\" (UniqueName: \"kubernetes.io/projected/26ca8cc6-0f79-47f4-807a-df2ac126efdc-kube-api-access-dqj7v\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.002907 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.002883 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-token\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.002968 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.002947 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-envoy\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.003031 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.003017 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-credential-socket\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.105667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.105579 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-workload-socket\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.105861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.105680 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-workload-certs\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.105861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.105714 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-data\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.105861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.105747 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istiod-ca-cert\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.105861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.105809 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqj7v\" (UniqueName: \"kubernetes.io/projected/26ca8cc6-0f79-47f4-807a-df2ac126efdc-kube-api-access-dqj7v\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.106100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.105939 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-token\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.106100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.105966 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-envoy\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.106100 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.106046 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-credential-socket\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.106267 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.106191 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-podinfo\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.107044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.106890 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-workload-socket\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.107044 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.106971 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-data\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.107270 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.107233 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-workload-certs\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.107385 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.107363 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istiod-ca-cert\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.107965 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.107919 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-credential-socket\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.108176 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.107970 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:35.109547 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.109515 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-envoy\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.109725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.109703 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-podinfo\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.130079 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.130042 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf" event={"ID":"6bc229aa-6e25-47b4-939d-23d34a39cd30","Type":"ContainerStarted","Data":"6b61dc69ce24c878600b3dd3f2abcbaa510b191381753a64bc54c930c70aa686"} Oct 30 16:58:35.130250 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.130107 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf" event={"ID":"6bc229aa-6e25-47b4-939d-23d34a39cd30","Type":"ContainerStarted","Data":"c1959744ab0fa28e37989b3d15e8d4caca323444f896789bc7d3270911df0c8e"} Oct 30 16:58:35.130323 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.130271 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-token\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.130663 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.130642 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqj7v\" (UniqueName: \"kubernetes.io/projected/26ca8cc6-0f79-47f4-807a-df2ac126efdc-kube-api-access-dqj7v\") pod \"e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.207791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.204326 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf" podStartSLOduration=2.204302052 podStartE2EDuration="2.204302052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:35.18018747 +0000 UTC m=+2228.890642215" watchObservedRunningTime="2026-10-30 16:58:35.204302052 +0000 UTC m=+2228.914756817" Oct 30 16:58:35.220194 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.220156 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:35.547001 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:58:35.546966 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ca8cc6_0f79_47f4_807a_df2ac126efdc.slice/crio-19020ecf74691ae284c007cffb334dd58cc00b3056ee5f555570b703b1f97714 WatchSource:0}: Error finding container 19020ecf74691ae284c007cffb334dd58cc00b3056ee5f555570b703b1f97714: Status 404 returned error can't find the container with id 19020ecf74691ae284c007cffb334dd58cc00b3056ee5f555570b703b1f97714 Oct 30 16:58:35.548095 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.548054 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25"] Oct 30 16:58:35.550918 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.550881 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:58:35.551051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.550955 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:58:35.551051 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:35.550986 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:58:36.176755 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:36.176719 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" event={"ID":"26ca8cc6-0f79-47f4-807a-df2ac126efdc","Type":"ContainerStarted","Data":"2b3c6776af50f271b30c268b58a0052bebb7116e2be7f854a27f869fdbc4aa7f"} Oct 30 16:58:36.177307 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:36.176828 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" event={"ID":"26ca8cc6-0f79-47f4-807a-df2ac126efdc","Type":"ContainerStarted","Data":"19020ecf74691ae284c007cffb334dd58cc00b3056ee5f555570b703b1f97714"} Oct 30 16:58:36.179303 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:36.179271 6202 generic.go:356] "Generic (PLEG): container finished" podID="6bc229aa-6e25-47b4-939d-23d34a39cd30" containerID="6b61dc69ce24c878600b3dd3f2abcbaa510b191381753a64bc54c930c70aa686" exitCode=0 Oct 30 16:58:36.179448 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:36.179433 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf" event={"ID":"6bc229aa-6e25-47b4-939d-23d34a39cd30","Type":"ContainerDied","Data":"6b61dc69ce24c878600b3dd3f2abcbaa510b191381753a64bc54c930c70aa686"} Oct 30 16:58:36.220928 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:36.220894 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:36.221866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:36.221808 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" podStartSLOduration=2.221760616 podStartE2EDuration="2.221760616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:36.220569266 +0000 UTC m=+2229.931024009" watchObservedRunningTime="2026-10-30 16:58:36.221760616 +0000 UTC m=+2229.932215361" Oct 30 16:58:36.222264 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:36.222235 6202 patch_prober.go:29] interesting pod/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 16:58:36.222393 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:36.222311 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" podUID="26ca8cc6-0f79-47f4-807a-df2ac126efdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 16:58:36.671790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:36.670870 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:37.221758 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:37.221712 6202 patch_prober.go:29] interesting pod/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 16:58:37.222185 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:37.221824 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" podUID="26ca8cc6-0f79-47f4-807a-df2ac126efdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 16:58:37.417954 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:37.417927 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf" Oct 30 16:58:37.573473 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:37.573435 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc229aa-6e25-47b4-939d-23d34a39cd30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc229aa-6e25-47b4-939d-23d34a39cd30-maas-api-openshift-ca\") pod \"6bc229aa-6e25-47b4-939d-23d34a39cd30\" (UID: \"6bc229aa-6e25-47b4-939d-23d34a39cd30\") " Oct 30 16:58:37.573668 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:37.573650 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bc229aa-6e25-47b4-939d-23d34a39cd30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc229aa-6e25-47b4-939d-23d34a39cd30-maas-api-xks-ca\") pod \"6bc229aa-6e25-47b4-939d-23d34a39cd30\" (UID: \"6bc229aa-6e25-47b4-939d-23d34a39cd30\") " Oct 30 16:58:37.573748 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:37.573737 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc229aa-6e25-47b4-939d-23d34a39cd30-kube-api-access-nv5sq\" (UniqueName: \"kubernetes.io/projected/6bc229aa-6e25-47b4-939d-23d34a39cd30-kube-api-access-nv5sq\") pod \"6bc229aa-6e25-47b4-939d-23d34a39cd30\" (UID: \"6bc229aa-6e25-47b4-939d-23d34a39cd30\") " Oct 30 16:58:37.573913 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:37.573883 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc229aa-6e25-47b4-939d-23d34a39cd30-maas-api-openshift-ca" pod "6bc229aa-6e25-47b4-939d-23d34a39cd30" (UID: "6bc229aa-6e25-47b4-939d-23d34a39cd30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:58:37.574408 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:37.574388 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc229aa-6e25-47b4-939d-23d34a39cd30-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:58:37.576088 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:37.576061 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bc229aa-6e25-47b4-939d-23d34a39cd30-maas-api-xks-ca" pod "6bc229aa-6e25-47b4-939d-23d34a39cd30" (UID: "6bc229aa-6e25-47b4-939d-23d34a39cd30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:58:37.576178 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:37.576158 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc229aa-6e25-47b4-939d-23d34a39cd30-kube-api-access-nv5sq" pod "6bc229aa-6e25-47b4-939d-23d34a39cd30" (UID: "6bc229aa-6e25-47b4-939d-23d34a39cd30"). InnerVolumeSpecName "kube-api-access-nv5sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:37.675657 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:37.675621 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc229aa-6e25-47b4-939d-23d34a39cd30-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:58:37.675870 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:37.675675 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv5sq\" (UniqueName: \"kubernetes.io/projected/6bc229aa-6e25-47b4-939d-23d34a39cd30-kube-api-access-nv5sq\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:58:38.197974 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:38.197937 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf" Oct 30 16:58:38.198165 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:38.197969 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf" event={"ID":"6bc229aa-6e25-47b4-939d-23d34a39cd30","Type":"ContainerDied","Data":"c1959744ab0fa28e37989b3d15e8d4caca323444f896789bc7d3270911df0c8e"} Oct 30 16:58:38.198165 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:38.198062 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1959744ab0fa28e37989b3d15e8d4caca323444f896789bc7d3270911df0c8e" Oct 30 16:58:38.220960 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:38.220931 6202 patch_prober.go:29] interesting pod/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 16:58:38.221123 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:38.220991 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" podUID="26ca8cc6-0f79-47f4-807a-df2ac126efdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 16:58:38.358792 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:38.356036 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:39.005052 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.005015 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k"] Oct 30 16:58:39.006053 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.005984 6202 cpu_manager.go:414] "containerMap: removing container" podUID="6bc229aa-6e25-47b4-939d-23d34a39cd30" containerName="revoke-keys" Oct 30 16:58:39.006053 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.006010 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc229aa-6e25-47b4-939d-23d34a39cd30" containerName="revoke-keys" Oct 30 16:58:39.006227 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.006214 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc229aa-6e25-47b4-939d-23d34a39cd30" containerName="revoke-keys" Oct 30 16:58:39.009404 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.009385 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k" Oct 30 16:58:39.024416 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.024390 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:58:39.073907 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.072477 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:58:39.089042 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.087890 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k"] Oct 30 16:58:39.109795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.109257 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5czpp\" (UniqueName: \"kubernetes.io/projected/2363fccd-b8eb-4896-a0e4-61af066ae754-kube-api-access-5czpp\") pod \"maas-api-revoke-sub-fbd1d14b-jdb2k\" (UID: \"2363fccd-b8eb-4896-a0e4-61af066ae754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k" Oct 30 16:58:39.109795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.109478 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2363fccd-b8eb-4896-a0e4-61af066ae754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbd1d14b-jdb2k\" (UID: \"2363fccd-b8eb-4896-a0e4-61af066ae754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k" Oct 30 16:58:39.109795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.109563 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2363fccd-b8eb-4896-a0e4-61af066ae754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbd1d14b-jdb2k\" (UID: \"2363fccd-b8eb-4896-a0e4-61af066ae754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k" Oct 30 16:58:39.210985 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.210955 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5czpp\" (UniqueName: \"kubernetes.io/projected/2363fccd-b8eb-4896-a0e4-61af066ae754-kube-api-access-5czpp\") pod \"maas-api-revoke-sub-fbd1d14b-jdb2k\" (UID: \"2363fccd-b8eb-4896-a0e4-61af066ae754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k" Oct 30 16:58:39.211344 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.211329 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2363fccd-b8eb-4896-a0e4-61af066ae754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbd1d14b-jdb2k\" (UID: \"2363fccd-b8eb-4896-a0e4-61af066ae754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k" Oct 30 16:58:39.211490 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.211477 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2363fccd-b8eb-4896-a0e4-61af066ae754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbd1d14b-jdb2k\" (UID: \"2363fccd-b8eb-4896-a0e4-61af066ae754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k" Oct 30 16:58:39.212157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.212128 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2363fccd-b8eb-4896-a0e4-61af066ae754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbd1d14b-jdb2k\" (UID: \"2363fccd-b8eb-4896-a0e4-61af066ae754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k" Oct 30 16:58:39.214112 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.214088 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2363fccd-b8eb-4896-a0e4-61af066ae754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbd1d14b-jdb2k\" (UID: \"2363fccd-b8eb-4896-a0e4-61af066ae754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k" Oct 30 16:58:39.221264 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.221243 6202 patch_prober.go:29] interesting pod/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 16:58:39.221349 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.221299 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" podUID="26ca8cc6-0f79-47f4-807a-df2ac126efdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 16:58:39.234425 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.234405 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5czpp\" (UniqueName: \"kubernetes.io/projected/2363fccd-b8eb-4896-a0e4-61af066ae754-kube-api-access-5czpp\") pod \"maas-api-revoke-sub-fbd1d14b-jdb2k\" (UID: \"2363fccd-b8eb-4896-a0e4-61af066ae754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k" Oct 30 16:58:39.324036 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.324007 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k" Oct 30 16:58:39.674241 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.674204 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k"] Oct 30 16:58:39.678031 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:58:39.677883 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2363fccd_b8eb_4896_a0e4_61af066ae754.slice/crio-ba6d704a3eb7d9c4e0d3865f63d2caba0946ff4590dba9b8a8649b3b6fa65e93 WatchSource:0}: Error finding container ba6d704a3eb7d9c4e0d3865f63d2caba0946ff4590dba9b8a8649b3b6fa65e93: Status 404 returned error can't find the container with id ba6d704a3eb7d9c4e0d3865f63d2caba0946ff4590dba9b8a8649b3b6fa65e93 Oct 30 16:58:39.951746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:39.951662 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:40.217500 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:40.217419 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k" event={"ID":"2363fccd-b8eb-4896-a0e4-61af066ae754","Type":"ContainerStarted","Data":"d92f23068a207b8e53014819a18447680664b70a89e32c415db6e8aac7b9c11d"} Oct 30 16:58:40.217500 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:40.217484 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k" event={"ID":"2363fccd-b8eb-4896-a0e4-61af066ae754","Type":"ContainerStarted","Data":"ba6d704a3eb7d9c4e0d3865f63d2caba0946ff4590dba9b8a8649b3b6fa65e93"} Oct 30 16:58:40.220951 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:40.220922 6202 patch_prober.go:29] interesting pod/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 16:58:40.221097 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:40.220986 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" podUID="26ca8cc6-0f79-47f4-807a-df2ac126efdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 16:58:40.265793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:40.260823 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k" podStartSLOduration=2.260802389 podStartE2EDuration="2.260802389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:40.260032735 +0000 UTC m=+2233.970487488" watchObservedRunningTime="2026-10-30 16:58:40.260802389 +0000 UTC m=+2233.971257134" Oct 30 16:58:40.536567 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:40.536460 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:41.221234 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:41.221146 6202 patch_prober.go:29] interesting pod/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 16:58:41.221234 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:41.221211 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" podUID="26ca8cc6-0f79-47f4-807a-df2ac126efdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 16:58:41.226954 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:41.226919 6202 generic.go:356] "Generic (PLEG): container finished" podID="2363fccd-b8eb-4896-a0e4-61af066ae754" containerID="d92f23068a207b8e53014819a18447680664b70a89e32c415db6e8aac7b9c11d" exitCode=0 Oct 30 16:58:41.227116 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:41.227010 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k" event={"ID":"2363fccd-b8eb-4896-a0e4-61af066ae754","Type":"ContainerDied","Data":"d92f23068a207b8e53014819a18447680664b70a89e32c415db6e8aac7b9c11d"} Oct 30 16:58:41.525315 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:41.525235 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:42.228892 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:42.228862 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:42.229408 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:42.229278 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:42.230043 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:42.230009 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 16:58:42.444371 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:42.444343 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k" Oct 30 16:58:42.603066 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:42.603034 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2363fccd-b8eb-4896-a0e4-61af066ae754-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2363fccd-b8eb-4896-a0e4-61af066ae754-maas-api-openshift-ca\") pod \"2363fccd-b8eb-4896-a0e4-61af066ae754\" (UID: \"2363fccd-b8eb-4896-a0e4-61af066ae754\") " Oct 30 16:58:42.603288 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:42.603203 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2363fccd-b8eb-4896-a0e4-61af066ae754-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2363fccd-b8eb-4896-a0e4-61af066ae754-maas-api-xks-ca\") pod \"2363fccd-b8eb-4896-a0e4-61af066ae754\" (UID: \"2363fccd-b8eb-4896-a0e4-61af066ae754\") " Oct 30 16:58:42.603378 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:42.603333 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2363fccd-b8eb-4896-a0e4-61af066ae754-kube-api-access-5czpp\" (UniqueName: \"kubernetes.io/projected/2363fccd-b8eb-4896-a0e4-61af066ae754-kube-api-access-5czpp\") pod \"2363fccd-b8eb-4896-a0e4-61af066ae754\" (UID: \"2363fccd-b8eb-4896-a0e4-61af066ae754\") " Oct 30 16:58:42.603450 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:42.603421 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2363fccd-b8eb-4896-a0e4-61af066ae754-maas-api-openshift-ca" pod "2363fccd-b8eb-4896-a0e4-61af066ae754" (UID: "2363fccd-b8eb-4896-a0e4-61af066ae754"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:58:42.604085 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:42.604059 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2363fccd-b8eb-4896-a0e4-61af066ae754-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:58:42.606265 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:42.606234 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2363fccd-b8eb-4896-a0e4-61af066ae754-maas-api-xks-ca" pod "2363fccd-b8eb-4896-a0e4-61af066ae754" (UID: "2363fccd-b8eb-4896-a0e4-61af066ae754"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:58:42.606369 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:42.606289 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2363fccd-b8eb-4896-a0e4-61af066ae754-kube-api-access-5czpp" pod "2363fccd-b8eb-4896-a0e4-61af066ae754" (UID: "2363fccd-b8eb-4896-a0e4-61af066ae754"). InnerVolumeSpecName "kube-api-access-5czpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:42.705638 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:42.705604 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2363fccd-b8eb-4896-a0e4-61af066ae754-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:58:42.705638 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:42.705635 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5czpp\" (UniqueName: \"kubernetes.io/projected/2363fccd-b8eb-4896-a0e4-61af066ae754-kube-api-access-5czpp\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:58:43.132349 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:43.132318 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:43.248861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:43.248826 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k" event={"ID":"2363fccd-b8eb-4896-a0e4-61af066ae754","Type":"ContainerDied","Data":"ba6d704a3eb7d9c4e0d3865f63d2caba0946ff4590dba9b8a8649b3b6fa65e93"} Oct 30 16:58:43.248861 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:43.248849 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k" Oct 30 16:58:43.249336 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:43.248888 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba6d704a3eb7d9c4e0d3865f63d2caba0946ff4590dba9b8a8649b3b6fa65e93" Oct 30 16:58:44.071500 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:58:44.071463 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:58:44.862793 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:44.858015 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:46.483941 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:46.483906 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:48.260842 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:48.260802 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:49.862752 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:49.862718 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:51.576963 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:51.576928 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:53.106294 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:53.106253 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:53.855990 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:53.855950 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:54.798978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:54.798815 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:54.951622 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:54.951572 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:55.406866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:55.406814 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:55.987318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:55.987274 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:56.649422 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:56.649367 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:56.919046 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:56.918960 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:58.457961 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:58.457921 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:59.224408 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.224367 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:59.371159 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.371123 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw"] Oct 30 16:58:59.372912 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.372879 6202 cpu_manager.go:414] "containerMap: removing container" podUID="2363fccd-b8eb-4896-a0e4-61af066ae754" containerName="revoke-keys" Oct 30 16:58:59.373118 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.373102 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2363fccd-b8eb-4896-a0e4-61af066ae754" containerName="revoke-keys" Oct 30 16:58:59.373523 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.373506 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="2363fccd-b8eb-4896-a0e4-61af066ae754" containerName="revoke-keys" Oct 30 16:58:59.389163 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.389138 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw" Oct 30 16:58:59.407001 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.406970 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:58:59.417488 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.417430 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:58:59.426661 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.426624 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw"] Oct 30 16:58:59.478276 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.478187 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-668047-z7fkw\" (UID: \"4db5f27d-57ed-48a4-8558-d2e8a51ffedc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw" Oct 30 16:58:59.478276 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.478247 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-668047-z7fkw\" (UID: \"4db5f27d-57ed-48a4-8558-d2e8a51ffedc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw" Oct 30 16:58:59.478836 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.478416 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gqrd\" (UniqueName: \"kubernetes.io/projected/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-kube-api-access-9gqrd\") pod \"maas-api-revoke-keys-e2e-inf-a-668047-z7fkw\" (UID: \"4db5f27d-57ed-48a4-8558-d2e8a51ffedc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw" Oct 30 16:58:59.580009 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.579726 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-668047-z7fkw\" (UID: \"4db5f27d-57ed-48a4-8558-d2e8a51ffedc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw" Oct 30 16:58:59.582571 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.582506 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-668047-z7fkw\" (UID: \"4db5f27d-57ed-48a4-8558-d2e8a51ffedc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw" Oct 30 16:58:59.583209 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.583186 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gqrd\" (UniqueName: \"kubernetes.io/projected/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-kube-api-access-9gqrd\") pod \"maas-api-revoke-keys-e2e-inf-a-668047-z7fkw\" (UID: \"4db5f27d-57ed-48a4-8558-d2e8a51ffedc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw" Oct 30 16:58:59.589024 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.588997 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-668047-z7fkw\" (UID: \"4db5f27d-57ed-48a4-8558-d2e8a51ffedc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw" Oct 30 16:58:59.593536 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.593503 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-668047-z7fkw\" (UID: \"4db5f27d-57ed-48a4-8558-d2e8a51ffedc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw" Oct 30 16:58:59.613252 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.613223 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gqrd\" (UniqueName: \"kubernetes.io/projected/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-kube-api-access-9gqrd\") pod \"maas-api-revoke-keys-e2e-inf-a-668047-z7fkw\" (UID: \"4db5f27d-57ed-48a4-8558-d2e8a51ffedc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw" Oct 30 16:58:59.670832 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.670796 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:59.777135 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.777045 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw" Oct 30 16:59:00.005594 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:58:59.993068 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:00.165628 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:59:00.165596 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db5f27d_57ed_48a4_8558_d2e8a51ffedc.slice/crio-78b09f009b05921558b3ef37b013412c906862d4162c6027a352ab7e24408fec WatchSource:0}: Error finding container 78b09f009b05921558b3ef37b013412c906862d4162c6027a352ab7e24408fec: Status 404 returned error can't find the container with id 78b09f009b05921558b3ef37b013412c906862d4162c6027a352ab7e24408fec Oct 30 16:59:00.167433 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:00.167402 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw"] Oct 30 16:59:00.169686 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:00.169643 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:00.217233 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:00.217203 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt"] Oct 30 16:59:00.223363 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:00.223339 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt" Oct 30 16:59:00.244503 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:00.244473 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt"] Oct 30 16:59:00.302972 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:00.302926 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250d2276-7b40-4aab-87dd-2a342eba32fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bf8bcf7-nwxrt\" (UID: \"250d2276-7b40-4aab-87dd-2a342eba32fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt" Oct 30 16:59:00.303181 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:00.302999 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm6lf\" (UniqueName: \"kubernetes.io/projected/250d2276-7b40-4aab-87dd-2a342eba32fc-kube-api-access-qm6lf\") pod \"maas-api-revoke-sub-0bf8bcf7-nwxrt\" (UID: \"250d2276-7b40-4aab-87dd-2a342eba32fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt" Oct 30 16:59:00.303263 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:00.303224 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250d2276-7b40-4aab-87dd-2a342eba32fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bf8bcf7-nwxrt\" (UID: \"250d2276-7b40-4aab-87dd-2a342eba32fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt" Oct 30 16:59:00.406382 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:00.406336 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250d2276-7b40-4aab-87dd-2a342eba32fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bf8bcf7-nwxrt\" (UID: \"250d2276-7b40-4aab-87dd-2a342eba32fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt" Oct 30 16:59:00.406594 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:00.406418 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm6lf\" (UniqueName: \"kubernetes.io/projected/250d2276-7b40-4aab-87dd-2a342eba32fc-kube-api-access-qm6lf\") pod \"maas-api-revoke-sub-0bf8bcf7-nwxrt\" (UID: \"250d2276-7b40-4aab-87dd-2a342eba32fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt" Oct 30 16:59:00.406594 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:00.406554 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250d2276-7b40-4aab-87dd-2a342eba32fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bf8bcf7-nwxrt\" (UID: \"250d2276-7b40-4aab-87dd-2a342eba32fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt" Oct 30 16:59:00.407247 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:00.407223 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250d2276-7b40-4aab-87dd-2a342eba32fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bf8bcf7-nwxrt\" (UID: \"250d2276-7b40-4aab-87dd-2a342eba32fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt" Oct 30 16:59:00.409222 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:00.409203 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250d2276-7b40-4aab-87dd-2a342eba32fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bf8bcf7-nwxrt\" (UID: \"250d2276-7b40-4aab-87dd-2a342eba32fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt" Oct 30 16:59:00.439018 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:00.438948 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm6lf\" (UniqueName: \"kubernetes.io/projected/250d2276-7b40-4aab-87dd-2a342eba32fc-kube-api-access-qm6lf\") pod \"maas-api-revoke-sub-0bf8bcf7-nwxrt\" (UID: \"250d2276-7b40-4aab-87dd-2a342eba32fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt" Oct 30 16:59:00.460709 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:00.460674 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw" event={"ID":"4db5f27d-57ed-48a4-8558-d2e8a51ffedc","Type":"ContainerStarted","Data":"78b09f009b05921558b3ef37b013412c906862d4162c6027a352ab7e24408fec"} Oct 30 16:59:00.538059 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:00.538024 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt" Oct 30 16:59:00.613790 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:00.611023 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:00.920478 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:59:00.920443 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250d2276_7b40_4aab_87dd_2a342eba32fc.slice/crio-221aa6645e9b4951643c0ba54a02e56143beefd2925de3a71f70ec4d7296275e WatchSource:0}: Error finding container 221aa6645e9b4951643c0ba54a02e56143beefd2925de3a71f70ec4d7296275e: Status 404 returned error can't find the container with id 221aa6645e9b4951643c0ba54a02e56143beefd2925de3a71f70ec4d7296275e Oct 30 16:59:00.925556 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:00.925497 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt"] Oct 30 16:59:01.475692 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:01.475656 6202 generic.go:356] "Generic (PLEG): container finished" podID="4db5f27d-57ed-48a4-8558-d2e8a51ffedc" containerID="4eea44602423055b25679bc409203dbaa23cf941d3977646674b22bc0619b61e" exitCode=0 Oct 30 16:59:01.475881 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:01.475749 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw" event={"ID":"4db5f27d-57ed-48a4-8558-d2e8a51ffedc","Type":"ContainerDied","Data":"4eea44602423055b25679bc409203dbaa23cf941d3977646674b22bc0619b61e"} Oct 30 16:59:01.477682 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:01.477655 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt" event={"ID":"250d2276-7b40-4aab-87dd-2a342eba32fc","Type":"ContainerStarted","Data":"2f02ac66ba72a04c57015d0dc0e8432683f6c37ff2dd5ace9ccd9c6c8b50a9d9"} Oct 30 16:59:01.477841 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:01.477708 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt" event={"ID":"250d2276-7b40-4aab-87dd-2a342eba32fc","Type":"ContainerStarted","Data":"221aa6645e9b4951643c0ba54a02e56143beefd2925de3a71f70ec4d7296275e"} Oct 30 16:59:01.564331 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:01.564228 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt" podStartSLOduration=1.564209517 podStartE2EDuration="1.564209517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:01.558057426 +0000 UTC m=+2255.268512171" watchObservedRunningTime="2026-10-30 16:59:01.564209517 +0000 UTC m=+2255.274664262" Oct 30 16:59:02.124370 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:02.124328 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:02.267870 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:02.267830 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:02.678062 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:02.678023 6202 generic.go:356] "Generic (PLEG): container finished" podID="250d2276-7b40-4aab-87dd-2a342eba32fc" containerID="2f02ac66ba72a04c57015d0dc0e8432683f6c37ff2dd5ace9ccd9c6c8b50a9d9" exitCode=0 Oct 30 16:59:02.678601 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:02.678108 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt" event={"ID":"250d2276-7b40-4aab-87dd-2a342eba32fc","Type":"ContainerDied","Data":"2f02ac66ba72a04c57015d0dc0e8432683f6c37ff2dd5ace9ccd9c6c8b50a9d9"} Oct 30 16:59:02.885177 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:02.885147 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw" Oct 30 16:59:02.917557 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:02.917528 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-maas-api-xks-ca\") pod \"4db5f27d-57ed-48a4-8558-d2e8a51ffedc\" (UID: \"4db5f27d-57ed-48a4-8558-d2e8a51ffedc\") " Oct 30 16:59:02.917728 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:02.917650 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-maas-api-openshift-ca\") pod \"4db5f27d-57ed-48a4-8558-d2e8a51ffedc\" (UID: \"4db5f27d-57ed-48a4-8558-d2e8a51ffedc\") " Oct 30 16:59:02.917864 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:02.917820 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-kube-api-access-9gqrd\" (UniqueName: \"kubernetes.io/projected/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-kube-api-access-9gqrd\") pod \"4db5f27d-57ed-48a4-8558-d2e8a51ffedc\" (UID: \"4db5f27d-57ed-48a4-8558-d2e8a51ffedc\") " Oct 30 16:59:02.918074 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:02.918049 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-maas-api-openshift-ca" pod "4db5f27d-57ed-48a4-8558-d2e8a51ffedc" (UID: "4db5f27d-57ed-48a4-8558-d2e8a51ffedc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:02.918596 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:02.918574 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:02.920037 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:02.920008 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-maas-api-xks-ca" pod "4db5f27d-57ed-48a4-8558-d2e8a51ffedc" (UID: "4db5f27d-57ed-48a4-8558-d2e8a51ffedc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:59:02.920653 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:02.920619 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-kube-api-access-9gqrd" pod "4db5f27d-57ed-48a4-8558-d2e8a51ffedc" (UID: "4db5f27d-57ed-48a4-8558-d2e8a51ffedc"). InnerVolumeSpecName "kube-api-access-9gqrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:03.020551 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:03.020517 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gqrd\" (UniqueName: \"kubernetes.io/projected/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-kube-api-access-9gqrd\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:03.020551 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:03.020549 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db5f27d-57ed-48a4-8558-d2e8a51ffedc-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:03.689518 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:03.689476 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw" Oct 30 16:59:03.689518 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:03.689509 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw" event={"ID":"4db5f27d-57ed-48a4-8558-d2e8a51ffedc","Type":"ContainerDied","Data":"78b09f009b05921558b3ef37b013412c906862d4162c6027a352ab7e24408fec"} Oct 30 16:59:03.690159 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:03.689565 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78b09f009b05921558b3ef37b013412c906862d4162c6027a352ab7e24408fec" Oct 30 16:59:03.876971 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:03.876944 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt" Oct 30 16:59:03.934556 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:03.934521 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/250d2276-7b40-4aab-87dd-2a342eba32fc-kube-api-access-qm6lf\" (UniqueName: \"kubernetes.io/projected/250d2276-7b40-4aab-87dd-2a342eba32fc-kube-api-access-qm6lf\") pod \"250d2276-7b40-4aab-87dd-2a342eba32fc\" (UID: \"250d2276-7b40-4aab-87dd-2a342eba32fc\") " Oct 30 16:59:03.934944 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:03.934721 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/250d2276-7b40-4aab-87dd-2a342eba32fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250d2276-7b40-4aab-87dd-2a342eba32fc-maas-api-openshift-ca\") pod \"250d2276-7b40-4aab-87dd-2a342eba32fc\" (UID: \"250d2276-7b40-4aab-87dd-2a342eba32fc\") " Oct 30 16:59:03.934944 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:03.934928 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/250d2276-7b40-4aab-87dd-2a342eba32fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250d2276-7b40-4aab-87dd-2a342eba32fc-maas-api-xks-ca\") pod \"250d2276-7b40-4aab-87dd-2a342eba32fc\" (UID: \"250d2276-7b40-4aab-87dd-2a342eba32fc\") " Oct 30 16:59:03.935350 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:03.935288 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250d2276-7b40-4aab-87dd-2a342eba32fc-maas-api-openshift-ca" pod "250d2276-7b40-4aab-87dd-2a342eba32fc" (UID: "250d2276-7b40-4aab-87dd-2a342eba32fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:03.936014 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:03.935983 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250d2276-7b40-4aab-87dd-2a342eba32fc-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:03.937378 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:03.937349 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250d2276-7b40-4aab-87dd-2a342eba32fc-kube-api-access-qm6lf" pod "250d2276-7b40-4aab-87dd-2a342eba32fc" (UID: "250d2276-7b40-4aab-87dd-2a342eba32fc"). InnerVolumeSpecName "kube-api-access-qm6lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:03.937465 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:03.937359 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/250d2276-7b40-4aab-87dd-2a342eba32fc-maas-api-xks-ca" pod "250d2276-7b40-4aab-87dd-2a342eba32fc" (UID: "250d2276-7b40-4aab-87dd-2a342eba32fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:59:04.037752 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:04.037710 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm6lf\" (UniqueName: \"kubernetes.io/projected/250d2276-7b40-4aab-87dd-2a342eba32fc-kube-api-access-qm6lf\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:04.037752 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:04.037751 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250d2276-7b40-4aab-87dd-2a342eba32fc-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:04.099634 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:04.099599 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:04.702143 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:04.702063 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt" Oct 30 16:59:04.702143 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:04.702091 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt" event={"ID":"250d2276-7b40-4aab-87dd-2a342eba32fc","Type":"ContainerDied","Data":"221aa6645e9b4951643c0ba54a02e56143beefd2925de3a71f70ec4d7296275e"} Oct 30 16:59:04.702143 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:04.702141 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="221aa6645e9b4951643c0ba54a02e56143beefd2925de3a71f70ec4d7296275e" Oct 30 16:59:05.636397 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:05.636319 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:07.275134 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:07.275089 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:08.674090 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:08.674052 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:08.805131 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:08.805092 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:09.075848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:09.072189 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:10.315007 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:10.314381 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j"] Oct 30 16:59:10.401853 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:10.401818 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf"] Oct 30 16:59:10.514847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:10.513631 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:10.636746 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:10.636705 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:11.520330 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:11.520286 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:13.240529 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:13.240483 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:13.667318 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:13.667277 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:14.083691 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:59:14.083646 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:59:14.087069 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.087038 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:14.298364 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.298310 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8"] Oct 30 16:59:14.300209 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.300177 6202 cpu_manager.go:414] "containerMap: removing container" podUID="250d2276-7b40-4aab-87dd-2a342eba32fc" containerName="revoke-keys" Oct 30 16:59:14.300369 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.300357 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="250d2276-7b40-4aab-87dd-2a342eba32fc" containerName="revoke-keys" Oct 30 16:59:14.300538 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.300527 6202 cpu_manager.go:414] "containerMap: removing container" podUID="4db5f27d-57ed-48a4-8558-d2e8a51ffedc" containerName="revoke-keys" Oct 30 16:59:14.300620 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.300611 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db5f27d-57ed-48a4-8558-d2e8a51ffedc" containerName="revoke-keys" Oct 30 16:59:14.301037 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.301016 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db5f27d-57ed-48a4-8558-d2e8a51ffedc" containerName="revoke-keys" Oct 30 16:59:14.301144 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.301063 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="250d2276-7b40-4aab-87dd-2a342eba32fc" containerName="revoke-keys" Oct 30 16:59:14.306439 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.306416 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8" Oct 30 16:59:14.330680 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.330646 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8"] Oct 30 16:59:14.336066 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.336005 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:59:14.336238 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.336022 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:59:14.381422 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.381385 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d73c8ad-03ed-48bc-94ef-e312042a1265-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8\" (UID: \"4d73c8ad-03ed-48bc-94ef-e312042a1265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8" Oct 30 16:59:14.381866 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.381840 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d73c8ad-03ed-48bc-94ef-e312042a1265-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8\" (UID: \"4d73c8ad-03ed-48bc-94ef-e312042a1265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8" Oct 30 16:59:14.382046 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.382027 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2c7h\" (UniqueName: \"kubernetes.io/projected/4d73c8ad-03ed-48bc-94ef-e312042a1265-kube-api-access-r2c7h\") pod \"maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8\" (UID: \"4d73c8ad-03ed-48bc-94ef-e312042a1265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8" Oct 30 16:59:14.484261 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.484218 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d73c8ad-03ed-48bc-94ef-e312042a1265-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8\" (UID: \"4d73c8ad-03ed-48bc-94ef-e312042a1265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8" Oct 30 16:59:14.484587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.484566 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d73c8ad-03ed-48bc-94ef-e312042a1265-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8\" (UID: \"4d73c8ad-03ed-48bc-94ef-e312042a1265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8" Oct 30 16:59:14.484705 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.484643 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2c7h\" (UniqueName: \"kubernetes.io/projected/4d73c8ad-03ed-48bc-94ef-e312042a1265-kube-api-access-r2c7h\") pod \"maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8\" (UID: \"4d73c8ad-03ed-48bc-94ef-e312042a1265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8" Oct 30 16:59:14.485032 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.485010 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d73c8ad-03ed-48bc-94ef-e312042a1265-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8\" (UID: \"4d73c8ad-03ed-48bc-94ef-e312042a1265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8" Oct 30 16:59:14.487130 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.487106 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d73c8ad-03ed-48bc-94ef-e312042a1265-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8\" (UID: \"4d73c8ad-03ed-48bc-94ef-e312042a1265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8" Oct 30 16:59:14.509611 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.509581 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2c7h\" (UniqueName: \"kubernetes.io/projected/4d73c8ad-03ed-48bc-94ef-e312042a1265-kube-api-access-r2c7h\") pod \"maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8\" (UID: \"4d73c8ad-03ed-48bc-94ef-e312042a1265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8" Oct 30 16:59:14.625832 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.625718 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8" Oct 30 16:59:14.976785 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.976738 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8"] Oct 30 16:59:14.977051 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:59:14.976833 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d73c8ad_03ed_48bc_94ef_e312042a1265.slice/crio-ccf279d54cbdf2630b4414f4495229bad0dd8f0598f1463652a98c12ba72054c WatchSource:0}: Error finding container ccf279d54cbdf2630b4414f4495229bad0dd8f0598f1463652a98c12ba72054c: Status 404 returned error can't find the container with id ccf279d54cbdf2630b4414f4495229bad0dd8f0598f1463652a98c12ba72054c Oct 30 16:59:14.978389 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:14.978335 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:15.315209 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:15.315154 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" podUID="d00bc805-8ee0-4851-aee7-cfc2856ee609" containerName="payload-processing" containerID="cri-o://1896cec0ede03bac79ec5261c3b6cf3f2720755c0e36208594ac2418276c8fc1" gracePeriod=25 Oct 30 16:59:15.402606 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:15.402554 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" podUID="e3847b39-1962-4f73-abeb-b57ea2a2f8b1" containerName="payload-pre-processing" containerID="cri-o://de0e361b09c4e4a7840bf4547dac2fc0632d3b75be73f7064010557681dfe7bd" gracePeriod=25 Oct 30 16:59:15.905118 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:15.905080 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8" event={"ID":"4d73c8ad-03ed-48bc-94ef-e312042a1265","Type":"ContainerStarted","Data":"d338bb26291bb6467fa95f4da28630c67148c5225f466ab85976bb59615ca15e"} Oct 30 16:59:15.905346 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:15.905137 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8" event={"ID":"4d73c8ad-03ed-48bc-94ef-e312042a1265","Type":"ContainerStarted","Data":"ccf279d54cbdf2630b4414f4495229bad0dd8f0598f1463652a98c12ba72054c"} Oct 30 16:59:15.951469 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:15.951401 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8" podStartSLOduration=1.951381578 podStartE2EDuration="1.951381578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:15.948454391 +0000 UTC m=+2269.658909136" watchObservedRunningTime="2026-10-30 16:59:15.951381578 +0000 UTC m=+2269.661836321" Oct 30 16:59:16.800498 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:16.800438 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:16.917357 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:16.917313 6202 generic.go:356] "Generic (PLEG): container finished" podID="4d73c8ad-03ed-48bc-94ef-e312042a1265" containerID="d338bb26291bb6467fa95f4da28630c67148c5225f466ab85976bb59615ca15e" exitCode=0 Oct 30 16:59:16.917559 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:16.917397 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8" event={"ID":"4d73c8ad-03ed-48bc-94ef-e312042a1265","Type":"ContainerDied","Data":"d338bb26291bb6467fa95f4da28630c67148c5225f466ab85976bb59615ca15e"} Oct 30 16:59:18.126922 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:18.126891 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8" Oct 30 16:59:18.254097 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:18.254010 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d73c8ad-03ed-48bc-94ef-e312042a1265-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d73c8ad-03ed-48bc-94ef-e312042a1265-maas-api-xks-ca\") pod \"4d73c8ad-03ed-48bc-94ef-e312042a1265\" (UID: \"4d73c8ad-03ed-48bc-94ef-e312042a1265\") " Oct 30 16:59:18.254286 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:18.254190 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d73c8ad-03ed-48bc-94ef-e312042a1265-kube-api-access-r2c7h\" (UniqueName: \"kubernetes.io/projected/4d73c8ad-03ed-48bc-94ef-e312042a1265-kube-api-access-r2c7h\") pod \"4d73c8ad-03ed-48bc-94ef-e312042a1265\" (UID: \"4d73c8ad-03ed-48bc-94ef-e312042a1265\") " Oct 30 16:59:18.254365 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:18.254304 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d73c8ad-03ed-48bc-94ef-e312042a1265-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d73c8ad-03ed-48bc-94ef-e312042a1265-maas-api-openshift-ca\") pod \"4d73c8ad-03ed-48bc-94ef-e312042a1265\" (UID: \"4d73c8ad-03ed-48bc-94ef-e312042a1265\") " Oct 30 16:59:18.254725 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:18.254697 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d73c8ad-03ed-48bc-94ef-e312042a1265-maas-api-openshift-ca" pod "4d73c8ad-03ed-48bc-94ef-e312042a1265" (UID: "4d73c8ad-03ed-48bc-94ef-e312042a1265"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:18.255161 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:18.255139 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d73c8ad-03ed-48bc-94ef-e312042a1265-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:18.256434 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:18.256404 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d73c8ad-03ed-48bc-94ef-e312042a1265-maas-api-xks-ca" pod "4d73c8ad-03ed-48bc-94ef-e312042a1265" (UID: "4d73c8ad-03ed-48bc-94ef-e312042a1265"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:59:18.256586 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:18.256568 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d73c8ad-03ed-48bc-94ef-e312042a1265-kube-api-access-r2c7h" pod "4d73c8ad-03ed-48bc-94ef-e312042a1265" (UID: "4d73c8ad-03ed-48bc-94ef-e312042a1265"). InnerVolumeSpecName "kube-api-access-r2c7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:18.356290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:18.356252 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d73c8ad-03ed-48bc-94ef-e312042a1265-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:18.356290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:18.356285 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2c7h\" (UniqueName: \"kubernetes.io/projected/4d73c8ad-03ed-48bc-94ef-e312042a1265-kube-api-access-r2c7h\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:18.513684 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:18.513575 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:18.936579 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:18.936547 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8" Oct 30 16:59:18.936806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:18.936554 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8" event={"ID":"4d73c8ad-03ed-48bc-94ef-e312042a1265","Type":"ContainerDied","Data":"ccf279d54cbdf2630b4414f4495229bad0dd8f0598f1463652a98c12ba72054c"} Oct 30 16:59:18.936806 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:18.936668 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccf279d54cbdf2630b4414f4495229bad0dd8f0598f1463652a98c12ba72054c" Oct 30 16:59:20.119138 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:20.119103 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:21.814812 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:21.814744 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:23.392530 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:23.392487 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:24.022818 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:24.022754 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" podUID="d00bc805-8ee0-4851-aee7-cfc2856ee609" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:59:24.023298 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:24.023272 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" podUID="e3847b39-1962-4f73-abeb-b57ea2a2f8b1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:59:24.695976 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:24.695943 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm"] Oct 30 16:59:24.727083 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:24.727054 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f"] Oct 30 16:59:24.852042 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:24.851968 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:24.991230 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:24.991095 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:26.048162 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:26.048123 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:26.681425 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:26.681391 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:27.082626 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:27.082595 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:27.235420 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:27.235389 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:28.201479 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:28.201432 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:29.697359 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:29.697319 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" podUID="940b403f-6a30-4c02-bae8-4215fc9cd643" containerName="payload-processing" containerID="cri-o://191efb322cff1203681d730ec13bb2e295da5c7e66ffffaf16b15cc9ac2efdba" gracePeriod=25 Oct 30 16:59:29.727719 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:29.727678 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" podUID="b0da4a2d-617f-4b93-8e0a-19db319bee2f" containerName="payload-pre-processing" containerID="cri-o://fc26a141889022af3f0bac3cad9e62fe26986fd8e1e9693da1830949809c2eee" gracePeriod=25 Oct 30 16:59:29.801675 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:29.801612 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:31.372244 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:31.372209 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:33.027539 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:33.027505 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:34.022551 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:34.022508 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" podUID="d00bc805-8ee0-4851-aee7-cfc2856ee609" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:59:34.022890 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:34.022578 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" podUID="e3847b39-1962-4f73-abeb-b57ea2a2f8b1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:59:34.175585 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:34.175550 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" podUID="b0da4a2d-617f-4b93-8e0a-19db319bee2f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:59:34.670182 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:34.670149 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:34.979172 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:34.979099 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx"] Oct 30 16:59:34.980033 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:34.980017 6202 cpu_manager.go:414] "containerMap: removing container" podUID="4d73c8ad-03ed-48bc-94ef-e312042a1265" containerName="revoke-keys" Oct 30 16:59:34.980105 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:34.980034 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d73c8ad-03ed-48bc-94ef-e312042a1265" containerName="revoke-keys" Oct 30 16:59:34.980157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:34.980148 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d73c8ad-03ed-48bc-94ef-e312042a1265" containerName="revoke-keys" Oct 30 16:59:34.984505 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:34.984488 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx" Oct 30 16:59:35.004594 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.004569 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx"] Oct 30 16:59:35.010970 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.010952 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:59:35.011104 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.011052 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:59:35.044904 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.044876 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab7d97a-1e00-4022-8828-25abdd86d43b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc0fd799-xcvfx\" (UID: \"3ab7d97a-1e00-4022-8828-25abdd86d43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx" Oct 30 16:59:35.045090 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.045067 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab7d97a-1e00-4022-8828-25abdd86d43b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc0fd799-xcvfx\" (UID: \"3ab7d97a-1e00-4022-8828-25abdd86d43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx" Oct 30 16:59:35.045260 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.045244 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnqzx\" (UniqueName: \"kubernetes.io/projected/3ab7d97a-1e00-4022-8828-25abdd86d43b-kube-api-access-rnqzx\") pod \"maas-api-revoke-sub-bc0fd799-xcvfx\" (UID: \"3ab7d97a-1e00-4022-8828-25abdd86d43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx" Oct 30 16:59:35.146824 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.146791 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab7d97a-1e00-4022-8828-25abdd86d43b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc0fd799-xcvfx\" (UID: \"3ab7d97a-1e00-4022-8828-25abdd86d43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx" Oct 30 16:59:35.147038 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.146916 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab7d97a-1e00-4022-8828-25abdd86d43b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc0fd799-xcvfx\" (UID: \"3ab7d97a-1e00-4022-8828-25abdd86d43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx" Oct 30 16:59:35.147111 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.147056 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnqzx\" (UniqueName: \"kubernetes.io/projected/3ab7d97a-1e00-4022-8828-25abdd86d43b-kube-api-access-rnqzx\") pod \"maas-api-revoke-sub-bc0fd799-xcvfx\" (UID: \"3ab7d97a-1e00-4022-8828-25abdd86d43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx" Oct 30 16:59:35.147592 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.147562 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab7d97a-1e00-4022-8828-25abdd86d43b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc0fd799-xcvfx\" (UID: \"3ab7d97a-1e00-4022-8828-25abdd86d43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx" Oct 30 16:59:35.149444 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.149421 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab7d97a-1e00-4022-8828-25abdd86d43b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc0fd799-xcvfx\" (UID: \"3ab7d97a-1e00-4022-8828-25abdd86d43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx" Oct 30 16:59:35.171085 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.171059 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnqzx\" (UniqueName: \"kubernetes.io/projected/3ab7d97a-1e00-4022-8828-25abdd86d43b-kube-api-access-rnqzx\") pod \"maas-api-revoke-sub-bc0fd799-xcvfx\" (UID: \"3ab7d97a-1e00-4022-8828-25abdd86d43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx" Oct 30 16:59:35.185747 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.185717 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" podUID="940b403f-6a30-4c02-bae8-4215fc9cd643" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:59:35.303813 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.303704 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx" Oct 30 16:59:35.622684 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.622657 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx"] Oct 30 16:59:35.624986 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:59:35.624955 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab7d97a_1e00_4022_8828_25abdd86d43b.slice/crio-1c95e3a3b3e961e29d79dcca2c77f998aae4d24c3526e95ef64c27e07d648111 WatchSource:0}: Error finding container 1c95e3a3b3e961e29d79dcca2c77f998aae4d24c3526e95ef64c27e07d648111: Status 404 returned error can't find the container with id 1c95e3a3b3e961e29d79dcca2c77f998aae4d24c3526e95ef64c27e07d648111 Oct 30 16:59:35.681529 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.681492 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" Oct 30 16:59:35.758692 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.758650 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d00bc805-8ee0-4851-aee7-cfc2856ee609-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00bc805-8ee0-4851-aee7-cfc2856ee609-plugins-config-volume\") pod \"d00bc805-8ee0-4851-aee7-cfc2856ee609\" (UID: \"d00bc805-8ee0-4851-aee7-cfc2856ee609\") " Oct 30 16:59:35.759015 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.758990 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00bc805-8ee0-4851-aee7-cfc2856ee609-plugins-config-volume" pod "d00bc805-8ee0-4851-aee7-cfc2856ee609" (UID: "d00bc805-8ee0-4851-aee7-cfc2856ee609"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:35.759208 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.759180 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00bc805-8ee0-4851-aee7-cfc2856ee609-kube-api-access-9dgj9\" (UniqueName: \"kubernetes.io/projected/d00bc805-8ee0-4851-aee7-cfc2856ee609-kube-api-access-9dgj9\") pod \"d00bc805-8ee0-4851-aee7-cfc2856ee609\" (UID: \"d00bc805-8ee0-4851-aee7-cfc2856ee609\") " Oct 30 16:59:35.760101 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.760075 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00bc805-8ee0-4851-aee7-cfc2856ee609-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:35.839293 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.839210 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00bc805-8ee0-4851-aee7-cfc2856ee609-kube-api-access-9dgj9" pod "d00bc805-8ee0-4851-aee7-cfc2856ee609" (UID: "d00bc805-8ee0-4851-aee7-cfc2856ee609"). InnerVolumeSpecName "kube-api-access-9dgj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:35.850494 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.850462 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" Oct 30 16:59:35.861473 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.861447 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dgj9\" (UniqueName: \"kubernetes.io/projected/d00bc805-8ee0-4851-aee7-cfc2856ee609-kube-api-access-9dgj9\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:35.962535 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.962503 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3847b39-1962-4f73-abeb-b57ea2a2f8b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3847b39-1962-4f73-abeb-b57ea2a2f8b1-plugins-config-volume\") pod \"e3847b39-1962-4f73-abeb-b57ea2a2f8b1\" (UID: \"e3847b39-1962-4f73-abeb-b57ea2a2f8b1\") " Oct 30 16:59:35.962708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.962676 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3847b39-1962-4f73-abeb-b57ea2a2f8b1-kube-api-access-kxmb9\" (UniqueName: \"kubernetes.io/projected/e3847b39-1962-4f73-abeb-b57ea2a2f8b1-kube-api-access-kxmb9\") pod \"e3847b39-1962-4f73-abeb-b57ea2a2f8b1\" (UID: \"e3847b39-1962-4f73-abeb-b57ea2a2f8b1\") " Oct 30 16:59:35.962928 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.962894 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3847b39-1962-4f73-abeb-b57ea2a2f8b1-plugins-config-volume" pod "e3847b39-1962-4f73-abeb-b57ea2a2f8b1" (UID: "e3847b39-1962-4f73-abeb-b57ea2a2f8b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:35.963556 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.963534 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3847b39-1962-4f73-abeb-b57ea2a2f8b1-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:35.965476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:35.965453 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3847b39-1962-4f73-abeb-b57ea2a2f8b1-kube-api-access-kxmb9" pod "e3847b39-1962-4f73-abeb-b57ea2a2f8b1" (UID: "e3847b39-1962-4f73-abeb-b57ea2a2f8b1"). InnerVolumeSpecName "kube-api-access-kxmb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:36.064824 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.064790 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxmb9\" (UniqueName: \"kubernetes.io/projected/e3847b39-1962-4f73-abeb-b57ea2a2f8b1-kube-api-access-kxmb9\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:36.144787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.141152 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx" event={"ID":"3ab7d97a-1e00-4022-8828-25abdd86d43b","Type":"ContainerStarted","Data":"1c95e3a3b3e961e29d79dcca2c77f998aae4d24c3526e95ef64c27e07d648111"} Oct 30 16:59:36.144787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.142759 6202 generic.go:356] "Generic (PLEG): container finished" podID="d00bc805-8ee0-4851-aee7-cfc2856ee609" containerID="1896cec0ede03bac79ec5261c3b6cf3f2720755c0e36208594ac2418276c8fc1" exitCode=0 Oct 30 16:59:36.144787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.142854 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" Oct 30 16:59:36.144787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.142860 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" event={"ID":"d00bc805-8ee0-4851-aee7-cfc2856ee609","Type":"ContainerDied","Data":"1896cec0ede03bac79ec5261c3b6cf3f2720755c0e36208594ac2418276c8fc1"} Oct 30 16:59:36.144787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.142923 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j" event={"ID":"d00bc805-8ee0-4851-aee7-cfc2856ee609","Type":"ContainerDied","Data":"c42559e15d053d10a73d9f1b8eab6d7616d828e9c1375b6282be41050354dab1"} Oct 30 16:59:36.144787 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.142946 6202 scope.go:122] "RemoveContainer" containerID="1896cec0ede03bac79ec5261c3b6cf3f2720755c0e36208594ac2418276c8fc1" Oct 30 16:59:36.145301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.144896 6202 generic.go:356] "Generic (PLEG): container finished" podID="e3847b39-1962-4f73-abeb-b57ea2a2f8b1" containerID="de0e361b09c4e4a7840bf4547dac2fc0632d3b75be73f7064010557681dfe7bd" exitCode=0 Oct 30 16:59:36.145301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.144955 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" Oct 30 16:59:36.145301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.144954 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" event={"ID":"e3847b39-1962-4f73-abeb-b57ea2a2f8b1","Type":"ContainerDied","Data":"de0e361b09c4e4a7840bf4547dac2fc0632d3b75be73f7064010557681dfe7bd"} Oct 30 16:59:36.145301 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.145064 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf" event={"ID":"e3847b39-1962-4f73-abeb-b57ea2a2f8b1","Type":"ContainerDied","Data":"89967007387b48dd12c6876bd5f357bd12358179c9cf45f42282a8d5474ae3ec"} Oct 30 16:59:36.158632 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.158612 6202 scope.go:122] "RemoveContainer" containerID="1896cec0ede03bac79ec5261c3b6cf3f2720755c0e36208594ac2418276c8fc1" Oct 30 16:59:36.158998 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:59:36.158979 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1896cec0ede03bac79ec5261c3b6cf3f2720755c0e36208594ac2418276c8fc1\": container with ID starting with 1896cec0ede03bac79ec5261c3b6cf3f2720755c0e36208594ac2418276c8fc1 not found: ID does not exist" containerID="1896cec0ede03bac79ec5261c3b6cf3f2720755c0e36208594ac2418276c8fc1" Oct 30 16:59:36.159078 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.159005 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1896cec0ede03bac79ec5261c3b6cf3f2720755c0e36208594ac2418276c8fc1"} err="failed to get container status \"1896cec0ede03bac79ec5261c3b6cf3f2720755c0e36208594ac2418276c8fc1\": rpc error: code = NotFound desc = could not find container \"1896cec0ede03bac79ec5261c3b6cf3f2720755c0e36208594ac2418276c8fc1\": container with ID starting with 1896cec0ede03bac79ec5261c3b6cf3f2720755c0e36208594ac2418276c8fc1 not found: ID does not exist" Oct 30 16:59:36.159078 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.159023 6202 scope.go:122] "RemoveContainer" containerID="de0e361b09c4e4a7840bf4547dac2fc0632d3b75be73f7064010557681dfe7bd" Oct 30 16:59:36.189029 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.188996 6202 scope.go:122] "RemoveContainer" containerID="de0e361b09c4e4a7840bf4547dac2fc0632d3b75be73f7064010557681dfe7bd" Oct 30 16:59:36.189350 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:59:36.189326 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de0e361b09c4e4a7840bf4547dac2fc0632d3b75be73f7064010557681dfe7bd\": container with ID starting with de0e361b09c4e4a7840bf4547dac2fc0632d3b75be73f7064010557681dfe7bd not found: ID does not exist" containerID="de0e361b09c4e4a7840bf4547dac2fc0632d3b75be73f7064010557681dfe7bd" Oct 30 16:59:36.189394 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.189362 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0e361b09c4e4a7840bf4547dac2fc0632d3b75be73f7064010557681dfe7bd"} err="failed to get container status \"de0e361b09c4e4a7840bf4547dac2fc0632d3b75be73f7064010557681dfe7bd\": rpc error: code = NotFound desc = could not find container \"de0e361b09c4e4a7840bf4547dac2fc0632d3b75be73f7064010557681dfe7bd\": container with ID starting with de0e361b09c4e4a7840bf4547dac2fc0632d3b75be73f7064010557681dfe7bd not found: ID does not exist" Oct 30 16:59:36.227013 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.226975 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf"] Oct 30 16:59:36.231258 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.231231 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-668047-784475c859-j84gf"] Oct 30 16:59:36.262874 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.262835 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j"] Oct 30 16:59:36.270720 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.270689 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-668047-66fcdf4b7d-56z9j"] Oct 30 16:59:36.375134 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.375102 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:36.452484 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.452391 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d00bc805-8ee0-4851-aee7-cfc2856ee609" path="/var/lib/kubelet/pods/d00bc805-8ee0-4851-aee7-cfc2856ee609/volumes" Oct 30 16:59:36.452990 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:36.452968 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3847b39-1962-4f73-abeb-b57ea2a2f8b1" path="/var/lib/kubelet/pods/e3847b39-1962-4f73-abeb-b57ea2a2f8b1/volumes" Oct 30 16:59:37.157243 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:37.157152 6202 generic.go:356] "Generic (PLEG): container finished" podID="3ab7d97a-1e00-4022-8828-25abdd86d43b" containerID="bd56178b1d2c7a15be791712901fc0c208cad25642774662b7554149fb4c99cc" exitCode=0 Oct 30 16:59:37.157414 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:37.157244 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx" event={"ID":"3ab7d97a-1e00-4022-8828-25abdd86d43b","Type":"ContainerDied","Data":"bd56178b1d2c7a15be791712901fc0c208cad25642774662b7554149fb4c99cc"} Oct 30 16:59:37.983084 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:37.983047 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:38.039523 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:38.037786 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd"] Oct 30 16:59:38.048675 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:38.048647 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e571fb5-wpwpd"] Oct 30 16:59:38.350919 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:38.350892 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx" Oct 30 16:59:38.404892 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:38.404843 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ab7d97a-1e00-4022-8828-25abdd86d43b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab7d97a-1e00-4022-8828-25abdd86d43b-maas-api-openshift-ca\") pod \"3ab7d97a-1e00-4022-8828-25abdd86d43b\" (UID: \"3ab7d97a-1e00-4022-8828-25abdd86d43b\") " Oct 30 16:59:38.405102 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:38.404901 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ab7d97a-1e00-4022-8828-25abdd86d43b-kube-api-access-rnqzx\" (UniqueName: \"kubernetes.io/projected/3ab7d97a-1e00-4022-8828-25abdd86d43b-kube-api-access-rnqzx\") pod \"3ab7d97a-1e00-4022-8828-25abdd86d43b\" (UID: \"3ab7d97a-1e00-4022-8828-25abdd86d43b\") " Oct 30 16:59:38.405102 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:38.404940 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ab7d97a-1e00-4022-8828-25abdd86d43b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab7d97a-1e00-4022-8828-25abdd86d43b-maas-api-xks-ca\") pod \"3ab7d97a-1e00-4022-8828-25abdd86d43b\" (UID: \"3ab7d97a-1e00-4022-8828-25abdd86d43b\") " Oct 30 16:59:38.405754 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:38.405724 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ab7d97a-1e00-4022-8828-25abdd86d43b-maas-api-openshift-ca" pod "3ab7d97a-1e00-4022-8828-25abdd86d43b" (UID: "3ab7d97a-1e00-4022-8828-25abdd86d43b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:38.413967 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:38.413938 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab7d97a-1e00-4022-8828-25abdd86d43b-kube-api-access-rnqzx" pod "3ab7d97a-1e00-4022-8828-25abdd86d43b" (UID: "3ab7d97a-1e00-4022-8828-25abdd86d43b"). InnerVolumeSpecName "kube-api-access-rnqzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:38.418496 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:38.417868 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ab7d97a-1e00-4022-8828-25abdd86d43b-maas-api-xks-ca" pod "3ab7d97a-1e00-4022-8828-25abdd86d43b" (UID: "3ab7d97a-1e00-4022-8828-25abdd86d43b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:59:38.423413 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:38.423379 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70e2da8d-fe73-4d43-9db4-2a7acd7d9706" path="/var/lib/kubelet/pods/70e2da8d-fe73-4d43-9db4-2a7acd7d9706/volumes" Oct 30 16:59:38.507657 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:38.507616 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab7d97a-1e00-4022-8828-25abdd86d43b-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:38.507657 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:38.507643 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnqzx\" (UniqueName: \"kubernetes.io/projected/3ab7d97a-1e00-4022-8828-25abdd86d43b-kube-api-access-rnqzx\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:38.507657 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:38.507654 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab7d97a-1e00-4022-8828-25abdd86d43b-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:39.179098 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:39.179065 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx" Oct 30 16:59:39.179476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:39.179100 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx" event={"ID":"3ab7d97a-1e00-4022-8828-25abdd86d43b","Type":"ContainerDied","Data":"1c95e3a3b3e961e29d79dcca2c77f998aae4d24c3526e95ef64c27e07d648111"} Oct 30 16:59:39.179476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:39.179147 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c95e3a3b3e961e29d79dcca2c77f998aae4d24c3526e95ef64c27e07d648111" Oct 30 16:59:39.790817 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:39.790748 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:40.665145 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:40.665100 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:41.233371 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:41.233337 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:42.804575 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:42.804532 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:44.079981 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:59:44.079936 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 16:59:44.124143 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:44.124109 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:44.175897 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:44.175847 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" podUID="b0da4a2d-617f-4b93-8e0a-19db319bee2f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:59:44.730080 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:44.730045 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:44.884555 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:44.884524 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:45.001409 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:45.001329 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8"] Oct 30 16:59:45.013394 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:45.012360 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a59502-n4fq8"] Oct 30 16:59:45.018290 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:45.018244 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:45.172198 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:45.172156 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:45.185893 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:45.185850 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" podUID="940b403f-6a30-4c02-bae8-4215fc9cd643" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:59:45.297340 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:45.297239 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:45.639749 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:45.639674 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:45.988379 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:45.988247 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:46.415808 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.415749 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d73c8ad-03ed-48bc-94ef-e312042a1265" path="/var/lib/kubelet/pods/4d73c8ad-03ed-48bc-94ef-e312042a1265/volumes" Oct 30 16:59:46.566453 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.566137 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh"] Oct 30 16:59:46.575789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.574642 6202 cpu_manager.go:414] "containerMap: removing container" podUID="e3847b39-1962-4f73-abeb-b57ea2a2f8b1" containerName="payload-pre-processing" Oct 30 16:59:46.575789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.574681 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3847b39-1962-4f73-abeb-b57ea2a2f8b1" containerName="payload-pre-processing" Oct 30 16:59:46.575789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.574728 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3ab7d97a-1e00-4022-8828-25abdd86d43b" containerName="revoke-keys" Oct 30 16:59:46.575789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.574736 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ab7d97a-1e00-4022-8828-25abdd86d43b" containerName="revoke-keys" Oct 30 16:59:46.575789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.574809 6202 cpu_manager.go:414] "containerMap: removing container" podUID="d00bc805-8ee0-4851-aee7-cfc2856ee609" containerName="payload-processing" Oct 30 16:59:46.575789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.574817 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d00bc805-8ee0-4851-aee7-cfc2856ee609" containerName="payload-processing" Oct 30 16:59:46.575789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.575008 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3847b39-1962-4f73-abeb-b57ea2a2f8b1" containerName="payload-pre-processing" Oct 30 16:59:46.575789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.575026 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ab7d97a-1e00-4022-8828-25abdd86d43b" containerName="revoke-keys" Oct 30 16:59:46.575789 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.575047 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="d00bc805-8ee0-4851-aee7-cfc2856ee609" containerName="payload-processing" Oct 30 16:59:46.666925 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.663970 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh"] Oct 30 16:59:46.666925 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.664124 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh" Oct 30 16:59:46.682218 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.682183 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:59:46.682892 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.682872 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:59:46.821523 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.821479 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxnvr\" (UniqueName: \"kubernetes.io/projected/e87fa350-f777-4857-8d44-7d19efbbd689-kube-api-access-qxnvr\") pod \"maas-api-revoke-sub-c19327a7-xwbwh\" (UID: \"e87fa350-f777-4857-8d44-7d19efbbd689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh" Oct 30 16:59:46.821869 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.821843 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87fa350-f777-4857-8d44-7d19efbbd689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c19327a7-xwbwh\" (UID: \"e87fa350-f777-4857-8d44-7d19efbbd689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh" Oct 30 16:59:46.822114 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.822096 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87fa350-f777-4857-8d44-7d19efbbd689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c19327a7-xwbwh\" (UID: \"e87fa350-f777-4857-8d44-7d19efbbd689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh" Oct 30 16:59:46.846675 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.846639 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:46.924333 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.924243 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87fa350-f777-4857-8d44-7d19efbbd689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c19327a7-xwbwh\" (UID: \"e87fa350-f777-4857-8d44-7d19efbbd689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh" Oct 30 16:59:46.924512 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.924497 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxnvr\" (UniqueName: \"kubernetes.io/projected/e87fa350-f777-4857-8d44-7d19efbbd689-kube-api-access-qxnvr\") pod \"maas-api-revoke-sub-c19327a7-xwbwh\" (UID: \"e87fa350-f777-4857-8d44-7d19efbbd689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh" Oct 30 16:59:46.924553 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.924527 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87fa350-f777-4857-8d44-7d19efbbd689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c19327a7-xwbwh\" (UID: \"e87fa350-f777-4857-8d44-7d19efbbd689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh" Oct 30 16:59:46.925240 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.925216 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87fa350-f777-4857-8d44-7d19efbbd689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c19327a7-xwbwh\" (UID: \"e87fa350-f777-4857-8d44-7d19efbbd689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh" Oct 30 16:59:46.926794 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.926752 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87fa350-f777-4857-8d44-7d19efbbd689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c19327a7-xwbwh\" (UID: \"e87fa350-f777-4857-8d44-7d19efbbd689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh" Oct 30 16:59:46.949157 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:46.949121 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxnvr\" (UniqueName: \"kubernetes.io/projected/e87fa350-f777-4857-8d44-7d19efbbd689-kube-api-access-qxnvr\") pod \"maas-api-revoke-sub-c19327a7-xwbwh\" (UID: \"e87fa350-f777-4857-8d44-7d19efbbd689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh" Oct 30 16:59:47.013129 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:47.013089 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh" Oct 30 16:59:47.577415 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:47.577365 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:47.713032 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:47.712978 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:47.720593 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:47.720553 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw"] Oct 30 16:59:47.732176 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:47.732142 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668047-z7fkw"] Oct 30 16:59:47.778826 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:59:47.778792 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87fa350_f777_4857_8d44_7d19efbbd689.slice/crio-8820f397f43ab705b4346bbc5891c2927971676cc994ff76cca18f874ae7b571 WatchSource:0}: Error finding container 8820f397f43ab705b4346bbc5891c2927971676cc994ff76cca18f874ae7b571: Status 404 returned error can't find the container with id 8820f397f43ab705b4346bbc5891c2927971676cc994ff76cca18f874ae7b571 Oct 30 16:59:47.781960 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:47.781927 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh"] Oct 30 16:59:47.840994 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:47.838416 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:48.220914 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:48.220875 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:48.375439 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:48.375404 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh" event={"ID":"e87fa350-f777-4857-8d44-7d19efbbd689","Type":"ContainerStarted","Data":"61a93ba69069153b22cf72ccc2f01d01190adff8458ad3bad88a9b3ee91a6957"} Oct 30 16:59:48.375615 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:48.375459 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh" event={"ID":"e87fa350-f777-4857-8d44-7d19efbbd689","Type":"ContainerStarted","Data":"8820f397f43ab705b4346bbc5891c2927971676cc994ff76cca18f874ae7b571"} Oct 30 16:59:48.413394 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:48.413359 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db5f27d-57ed-48a4-8558-d2e8a51ffedc" path="/var/lib/kubelet/pods/4db5f27d-57ed-48a4-8558-d2e8a51ffedc/volumes" Oct 30 16:59:48.420199 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:48.420142 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh" podStartSLOduration=2.420125373 podStartE2EDuration="2.420125373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:48.417052516 +0000 UTC m=+2302.127507260" watchObservedRunningTime="2026-10-30 16:59:48.420125373 +0000 UTC m=+2302.130580119" Oct 30 16:59:49.387590 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:49.387551 6202 generic.go:356] "Generic (PLEG): container finished" podID="e87fa350-f777-4857-8d44-7d19efbbd689" containerID="61a93ba69069153b22cf72ccc2f01d01190adff8458ad3bad88a9b3ee91a6957" exitCode=0 Oct 30 16:59:49.388104 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:49.387609 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh" event={"ID":"e87fa350-f777-4857-8d44-7d19efbbd689","Type":"ContainerDied","Data":"61a93ba69069153b22cf72ccc2f01d01190adff8458ad3bad88a9b3ee91a6957"} Oct 30 16:59:49.648535 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:49.648440 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:49.982472 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:49.982439 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" Oct 30 16:59:50.050736 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.050700 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf"] Oct 30 16:59:50.054380 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.054348 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edc52165-k4kdf"] Oct 30 16:59:50.092229 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.092195 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" Oct 30 16:59:50.162791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.159973 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940b403f-6a30-4c02-bae8-4215fc9cd643-kube-api-access-s66q5\" (UniqueName: \"kubernetes.io/projected/940b403f-6a30-4c02-bae8-4215fc9cd643-kube-api-access-s66q5\") pod \"940b403f-6a30-4c02-bae8-4215fc9cd643\" (UID: \"940b403f-6a30-4c02-bae8-4215fc9cd643\") " Oct 30 16:59:50.162791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.160673 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/940b403f-6a30-4c02-bae8-4215fc9cd643-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940b403f-6a30-4c02-bae8-4215fc9cd643-plugins-config-volume\") pod \"940b403f-6a30-4c02-bae8-4215fc9cd643\" (UID: \"940b403f-6a30-4c02-bae8-4215fc9cd643\") " Oct 30 16:59:50.162791 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.162127 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940b403f-6a30-4c02-bae8-4215fc9cd643-plugins-config-volume" pod "940b403f-6a30-4c02-bae8-4215fc9cd643" (UID: "940b403f-6a30-4c02-bae8-4215fc9cd643"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:50.167849 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.163389 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940b403f-6a30-4c02-bae8-4215fc9cd643-kube-api-access-s66q5" pod "940b403f-6a30-4c02-bae8-4215fc9cd643" (UID: "940b403f-6a30-4c02-bae8-4215fc9cd643"). InnerVolumeSpecName "kube-api-access-s66q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:50.263231 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.263188 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0da4a2d-617f-4b93-8e0a-19db319bee2f-kube-api-access-4fbdh\" (UniqueName: \"kubernetes.io/projected/b0da4a2d-617f-4b93-8e0a-19db319bee2f-kube-api-access-4fbdh\") pod \"b0da4a2d-617f-4b93-8e0a-19db319bee2f\" (UID: \"b0da4a2d-617f-4b93-8e0a-19db319bee2f\") " Oct 30 16:59:50.263471 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.263457 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0da4a2d-617f-4b93-8e0a-19db319bee2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0da4a2d-617f-4b93-8e0a-19db319bee2f-plugins-config-volume\") pod \"b0da4a2d-617f-4b93-8e0a-19db319bee2f\" (UID: \"b0da4a2d-617f-4b93-8e0a-19db319bee2f\") " Oct 30 16:59:50.263993 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.263913 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0da4a2d-617f-4b93-8e0a-19db319bee2f-plugins-config-volume" pod "b0da4a2d-617f-4b93-8e0a-19db319bee2f" (UID: "b0da4a2d-617f-4b93-8e0a-19db319bee2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:50.264230 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.264213 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940b403f-6a30-4c02-bae8-4215fc9cd643-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:50.264314 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.264233 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0da4a2d-617f-4b93-8e0a-19db319bee2f-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:50.264314 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.264242 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s66q5\" (UniqueName: \"kubernetes.io/projected/940b403f-6a30-4c02-bae8-4215fc9cd643-kube-api-access-s66q5\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:50.265698 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.265650 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0da4a2d-617f-4b93-8e0a-19db319bee2f-kube-api-access-4fbdh" pod "b0da4a2d-617f-4b93-8e0a-19db319bee2f" (UID: "b0da4a2d-617f-4b93-8e0a-19db319bee2f"). InnerVolumeSpecName "kube-api-access-4fbdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:50.365951 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.365890 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fbdh\" (UniqueName: \"kubernetes.io/projected/b0da4a2d-617f-4b93-8e0a-19db319bee2f-kube-api-access-4fbdh\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:50.397656 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.397617 6202 generic.go:356] "Generic (PLEG): container finished" podID="940b403f-6a30-4c02-bae8-4215fc9cd643" containerID="191efb322cff1203681d730ec13bb2e295da5c7e66ffffaf16b15cc9ac2efdba" exitCode=0 Oct 30 16:59:50.398184 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.397668 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" Oct 30 16:59:50.398184 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.397697 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" event={"ID":"940b403f-6a30-4c02-bae8-4215fc9cd643","Type":"ContainerDied","Data":"191efb322cff1203681d730ec13bb2e295da5c7e66ffffaf16b15cc9ac2efdba"} Oct 30 16:59:50.398184 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.397750 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm" event={"ID":"940b403f-6a30-4c02-bae8-4215fc9cd643","Type":"ContainerDied","Data":"10886bea11482ad2e0d62a1aac240bccd3e7006031f78b5417d011d086a15da7"} Oct 30 16:59:50.398184 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.397803 6202 scope.go:122] "RemoveContainer" containerID="191efb322cff1203681d730ec13bb2e295da5c7e66ffffaf16b15cc9ac2efdba" Oct 30 16:59:50.399667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.399597 6202 generic.go:356] "Generic (PLEG): container finished" podID="b0da4a2d-617f-4b93-8e0a-19db319bee2f" containerID="fc26a141889022af3f0bac3cad9e62fe26986fd8e1e9693da1830949809c2eee" exitCode=0 Oct 30 16:59:50.399667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.399631 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" event={"ID":"b0da4a2d-617f-4b93-8e0a-19db319bee2f","Type":"ContainerDied","Data":"fc26a141889022af3f0bac3cad9e62fe26986fd8e1e9693da1830949809c2eee"} Oct 30 16:59:50.399667 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.399688 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" event={"ID":"b0da4a2d-617f-4b93-8e0a-19db319bee2f","Type":"ContainerDied","Data":"e22e873425ca29c32447610faaba84edd83ca098f20e917a3f2d480c11f81468"} Oct 30 16:59:50.400075 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.399641 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f" Oct 30 16:59:50.411911 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.411887 6202 scope.go:122] "RemoveContainer" containerID="191efb322cff1203681d730ec13bb2e295da5c7e66ffffaf16b15cc9ac2efdba" Oct 30 16:59:50.412655 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:59:50.412498 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191efb322cff1203681d730ec13bb2e295da5c7e66ffffaf16b15cc9ac2efdba\": container with ID starting with 191efb322cff1203681d730ec13bb2e295da5c7e66ffffaf16b15cc9ac2efdba not found: ID does not exist" containerID="191efb322cff1203681d730ec13bb2e295da5c7e66ffffaf16b15cc9ac2efdba" Oct 30 16:59:50.412655 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.412539 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191efb322cff1203681d730ec13bb2e295da5c7e66ffffaf16b15cc9ac2efdba"} err="failed to get container status \"191efb322cff1203681d730ec13bb2e295da5c7e66ffffaf16b15cc9ac2efdba\": rpc error: code = NotFound desc = could not find container \"191efb322cff1203681d730ec13bb2e295da5c7e66ffffaf16b15cc9ac2efdba\": container with ID starting with 191efb322cff1203681d730ec13bb2e295da5c7e66ffffaf16b15cc9ac2efdba not found: ID does not exist" Oct 30 16:59:50.412655 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.412564 6202 scope.go:122] "RemoveContainer" containerID="fc26a141889022af3f0bac3cad9e62fe26986fd8e1e9693da1830949809c2eee" Oct 30 16:59:50.418002 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.417927 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23364ade-95d2-4603-bc01-aec45b1c4c56" path="/var/lib/kubelet/pods/23364ade-95d2-4603-bc01-aec45b1c4c56/volumes" Oct 30 16:59:50.426503 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.426478 6202 scope.go:122] "RemoveContainer" containerID="fc26a141889022af3f0bac3cad9e62fe26986fd8e1e9693da1830949809c2eee" Oct 30 16:59:50.426841 ip-10-0-129-65 kubenswrapper[6202]: E1030 16:59:50.426817 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc26a141889022af3f0bac3cad9e62fe26986fd8e1e9693da1830949809c2eee\": container with ID starting with fc26a141889022af3f0bac3cad9e62fe26986fd8e1e9693da1830949809c2eee not found: ID does not exist" containerID="fc26a141889022af3f0bac3cad9e62fe26986fd8e1e9693da1830949809c2eee" Oct 30 16:59:50.426960 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.426845 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc26a141889022af3f0bac3cad9e62fe26986fd8e1e9693da1830949809c2eee"} err="failed to get container status \"fc26a141889022af3f0bac3cad9e62fe26986fd8e1e9693da1830949809c2eee\": rpc error: code = NotFound desc = could not find container \"fc26a141889022af3f0bac3cad9e62fe26986fd8e1e9693da1830949809c2eee\": container with ID starting with fc26a141889022af3f0bac3cad9e62fe26986fd8e1e9693da1830949809c2eee not found: ID does not exist" Oct 30 16:59:50.456690 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.456652 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm"] Oct 30 16:59:50.466243 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.466212 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a59502-5894bf954d-qr5nm"] Oct 30 16:59:50.497189 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.497147 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f"] Oct 30 16:59:50.511037 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.510965 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a59502-69989b968c-vmg2f"] Oct 30 16:59:50.585658 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.585619 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh" Oct 30 16:59:50.773120 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.773064 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e87fa350-f777-4857-8d44-7d19efbbd689-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87fa350-f777-4857-8d44-7d19efbbd689-maas-api-openshift-ca\") pod \"e87fa350-f777-4857-8d44-7d19efbbd689\" (UID: \"e87fa350-f777-4857-8d44-7d19efbbd689\") " Oct 30 16:59:50.773399 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.773150 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e87fa350-f777-4857-8d44-7d19efbbd689-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87fa350-f777-4857-8d44-7d19efbbd689-maas-api-xks-ca\") pod \"e87fa350-f777-4857-8d44-7d19efbbd689\" (UID: \"e87fa350-f777-4857-8d44-7d19efbbd689\") " Oct 30 16:59:50.773399 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.773261 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e87fa350-f777-4857-8d44-7d19efbbd689-kube-api-access-qxnvr\" (UniqueName: \"kubernetes.io/projected/e87fa350-f777-4857-8d44-7d19efbbd689-kube-api-access-qxnvr\") pod \"e87fa350-f777-4857-8d44-7d19efbbd689\" (UID: \"e87fa350-f777-4857-8d44-7d19efbbd689\") " Oct 30 16:59:50.773548 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.773525 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87fa350-f777-4857-8d44-7d19efbbd689-maas-api-openshift-ca" pod "e87fa350-f777-4857-8d44-7d19efbbd689" (UID: "e87fa350-f777-4857-8d44-7d19efbbd689"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:50.773953 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.773927 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87fa350-f777-4857-8d44-7d19efbbd689-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:50.775674 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.775638 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87fa350-f777-4857-8d44-7d19efbbd689-kube-api-access-qxnvr" pod "e87fa350-f777-4857-8d44-7d19efbbd689" (UID: "e87fa350-f777-4857-8d44-7d19efbbd689"). InnerVolumeSpecName "kube-api-access-qxnvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:50.775896 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.775862 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e87fa350-f777-4857-8d44-7d19efbbd689-maas-api-xks-ca" pod "e87fa350-f777-4857-8d44-7d19efbbd689" (UID: "e87fa350-f777-4857-8d44-7d19efbbd689"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:59:50.876228 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.876187 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxnvr\" (UniqueName: \"kubernetes.io/projected/e87fa350-f777-4857-8d44-7d19efbbd689-kube-api-access-qxnvr\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:50.876228 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:50.876222 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87fa350-f777-4857-8d44-7d19efbbd689-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:51.374098 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:51.374060 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:51.438621 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:51.438552 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh" event={"ID":"e87fa350-f777-4857-8d44-7d19efbbd689","Type":"ContainerDied","Data":"8820f397f43ab705b4346bbc5891c2927971676cc994ff76cca18f874ae7b571"} Oct 30 16:59:51.439246 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:51.438676 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8820f397f43ab705b4346bbc5891c2927971676cc994ff76cca18f874ae7b571" Oct 30 16:59:51.439246 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:51.438854 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh" Oct 30 16:59:52.312947 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:52.312861 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:52.413235 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:52.413205 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="940b403f-6a30-4c02-bae8-4215fc9cd643" path="/var/lib/kubelet/pods/940b403f-6a30-4c02-bae8-4215fc9cd643/volumes" Oct 30 16:59:52.413633 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:52.413619 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0da4a2d-617f-4b93-8e0a-19db319bee2f" path="/var/lib/kubelet/pods/b0da4a2d-617f-4b93-8e0a-19db319bee2f/volumes" Oct 30 16:59:52.445110 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:52.445071 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:52.662795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:52.648046 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj"] Oct 30 16:59:52.662795 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:52.648300 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" podUID="4775a051-8cf1-4771-ab83-0b31f6f9f40e" containerName="istio-proxy" containerID="cri-o://59fdb85be3a3aac56d9ef48c7fa6adea7eb8f7ae4f30b1099a15723f2a94f38d" gracePeriod=30 Oct 30 16:59:52.677573 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:52.677539 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:53.235553 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.235516 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf"] Oct 30 16:59:53.236389 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.236371 6202 cpu_manager.go:414] "containerMap: removing container" podUID="e87fa350-f777-4857-8d44-7d19efbbd689" containerName="revoke-keys" Oct 30 16:59:53.236478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.236390 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87fa350-f777-4857-8d44-7d19efbbd689" containerName="revoke-keys" Oct 30 16:59:53.236478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.236408 6202 cpu_manager.go:414] "containerMap: removing container" podUID="b0da4a2d-617f-4b93-8e0a-19db319bee2f" containerName="payload-pre-processing" Oct 30 16:59:53.236478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.236413 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0da4a2d-617f-4b93-8e0a-19db319bee2f" containerName="payload-pre-processing" Oct 30 16:59:53.236478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.236438 6202 cpu_manager.go:414] "containerMap: removing container" podUID="940b403f-6a30-4c02-bae8-4215fc9cd643" containerName="payload-processing" Oct 30 16:59:53.236478 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.236444 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940b403f-6a30-4c02-bae8-4215fc9cd643" containerName="payload-processing" Oct 30 16:59:53.236634 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.236577 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="940b403f-6a30-4c02-bae8-4215fc9cd643" containerName="payload-processing" Oct 30 16:59:53.236634 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.236594 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="e87fa350-f777-4857-8d44-7d19efbbd689" containerName="revoke-keys" Oct 30 16:59:53.236634 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.236603 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0da4a2d-617f-4b93-8e0a-19db319bee2f" containerName="payload-pre-processing" Oct 30 16:59:53.241868 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.241844 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf" Oct 30 16:59:53.264194 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.264084 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:59:53.264375 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.264316 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf"] Oct 30 16:59:53.269735 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.269695 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:59:53.292305 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.292272 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:53.360123 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.349101 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf\" (UID: \"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf" Oct 30 16:59:53.360123 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.349352 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64449\" (UniqueName: \"kubernetes.io/projected/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-kube-api-access-64449\") pod \"maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf\" (UID: \"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf" Oct 30 16:59:53.360123 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.349544 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf\" (UID: \"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf" Oct 30 16:59:53.454644 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.454606 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf\" (UID: \"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf" Oct 30 16:59:53.455206 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.454679 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64449\" (UniqueName: \"kubernetes.io/projected/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-kube-api-access-64449\") pod \"maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf\" (UID: \"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf" Oct 30 16:59:53.455206 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.454863 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf\" (UID: \"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf" Oct 30 16:59:53.455657 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.455627 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf\" (UID: \"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf" Oct 30 16:59:53.457575 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.457551 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf\" (UID: \"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf" Oct 30 16:59:53.481847 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.481817 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64449\" (UniqueName: \"kubernetes.io/projected/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-kube-api-access-64449\") pod \"maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf\" (UID: \"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf" Oct 30 16:59:53.559620 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.559529 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf" Oct 30 16:59:53.880559 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:53.880527 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf"] Oct 30 16:59:53.882277 ip-10-0-129-65 kubenswrapper[6202]: W1030 16:59:53.882247 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2f1a10_33b9_45db_bf19_ddf7b26c3ce5.slice/crio-3ae30d5c12d56951ba52f112aceac60fc41e69eb75b4f7b8e57bcc43eeb3020c WatchSource:0}: Error finding container 3ae30d5c12d56951ba52f112aceac60fc41e69eb75b4f7b8e57bcc43eeb3020c: Status 404 returned error can't find the container with id 3ae30d5c12d56951ba52f112aceac60fc41e69eb75b4f7b8e57bcc43eeb3020c Oct 30 16:59:54.484061 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:54.484026 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf" event={"ID":"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5","Type":"ContainerStarted","Data":"35ee7d286ededb0d47a37b8e4da5d5e4cf82abaa882e7435dfd1fe44b39a1094"} Oct 30 16:59:54.484476 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:54.484075 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf" event={"ID":"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5","Type":"ContainerStarted","Data":"3ae30d5c12d56951ba52f112aceac60fc41e69eb75b4f7b8e57bcc43eeb3020c"} Oct 30 16:59:54.529001 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:54.528880 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf" podStartSLOduration=1.5288620640000001 podStartE2EDuration="1.528862064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:54.525289835 +0000 UTC m=+2308.235744578" watchObservedRunningTime="2026-10-30 16:59:54.528862064 +0000 UTC m=+2308.239316807" Oct 30 16:59:55.131289 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:55.131255 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:55.496054 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:55.495967 6202 generic.go:356] "Generic (PLEG): container finished" podID="bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5" containerID="35ee7d286ededb0d47a37b8e4da5d5e4cf82abaa882e7435dfd1fe44b39a1094" exitCode=0 Oct 30 16:59:55.496054 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:55.496012 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf" event={"ID":"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5","Type":"ContainerDied","Data":"35ee7d286ededb0d47a37b8e4da5d5e4cf82abaa882e7435dfd1fe44b39a1094"} Oct 30 16:59:55.601607 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:55.601568 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:55.824351 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:55.824311 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:55.976437 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:55.976397 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh"] Oct 30 16:59:55.976698 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:55.976662 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" podUID="927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" containerName="istio-proxy" containerID="cri-o://9b88b039b2e82009b766fe282e75aa79c4426a72e970bf35e3edf5bea53d2646" gracePeriod=30 Oct 30 16:59:56.003756 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.003718 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:56.548169 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.548131 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g"] Oct 30 16:59:56.553525 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.553465 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g" Oct 30 16:59:56.576590 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.576555 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g"] Oct 30 16:59:56.650300 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.650260 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181bd0c6-3a62-4ad2-8244-9987e0ac0606-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g\" (UID: \"181bd0c6-3a62-4ad2-8244-9987e0ac0606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g" Oct 30 16:59:56.650782 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.650736 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-876xd\" (UniqueName: \"kubernetes.io/projected/181bd0c6-3a62-4ad2-8244-9987e0ac0606-kube-api-access-876xd\") pod \"maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g\" (UID: \"181bd0c6-3a62-4ad2-8244-9987e0ac0606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g" Oct 30 16:59:56.650933 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.650800 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181bd0c6-3a62-4ad2-8244-9987e0ac0606-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g\" (UID: \"181bd0c6-3a62-4ad2-8244-9987e0ac0606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g" Oct 30 16:59:56.739978 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.739944 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf" Oct 30 16:59:56.752250 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.752219 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-maas-api-openshift-ca\") pod \"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5\" (UID: \"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5\") " Oct 30 16:59:56.752461 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.752400 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-maas-api-xks-ca\") pod \"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5\" (UID: \"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5\") " Oct 30 16:59:56.752578 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.752559 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-maas-api-openshift-ca" pod "bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5" (UID: "bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:56.752644 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.752610 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-kube-api-access-64449\" (UniqueName: \"kubernetes.io/projected/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-kube-api-access-64449\") pod \"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5\" (UID: \"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5\") " Oct 30 16:59:56.752868 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.752846 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-876xd\" (UniqueName: \"kubernetes.io/projected/181bd0c6-3a62-4ad2-8244-9987e0ac0606-kube-api-access-876xd\") pod \"maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g\" (UID: \"181bd0c6-3a62-4ad2-8244-9987e0ac0606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g" Oct 30 16:59:56.752989 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.752881 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181bd0c6-3a62-4ad2-8244-9987e0ac0606-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g\" (UID: \"181bd0c6-3a62-4ad2-8244-9987e0ac0606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g" Oct 30 16:59:56.753098 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.753078 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181bd0c6-3a62-4ad2-8244-9987e0ac0606-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g\" (UID: \"181bd0c6-3a62-4ad2-8244-9987e0ac0606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g" Oct 30 16:59:56.753380 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.753360 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:56.754453 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.754420 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181bd0c6-3a62-4ad2-8244-9987e0ac0606-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g\" (UID: \"181bd0c6-3a62-4ad2-8244-9987e0ac0606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g" Oct 30 16:59:56.755068 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.755039 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-maas-api-xks-ca" pod "bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5" (UID: "bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:59:56.755197 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.755179 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-kube-api-access-64449" pod "bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5" (UID: "bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5"). InnerVolumeSpecName "kube-api-access-64449". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:56.755539 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.755516 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181bd0c6-3a62-4ad2-8244-9987e0ac0606-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g\" (UID: \"181bd0c6-3a62-4ad2-8244-9987e0ac0606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g" Oct 30 16:59:56.782610 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.782572 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-876xd\" (UniqueName: \"kubernetes.io/projected/181bd0c6-3a62-4ad2-8244-9987e0ac0606-kube-api-access-876xd\") pod \"maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g\" (UID: \"181bd0c6-3a62-4ad2-8244-9987e0ac0606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g" Oct 30 16:59:56.854917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.854882 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:56.854917 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.854915 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64449\" (UniqueName: \"kubernetes.io/projected/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5-kube-api-access-64449\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:56.875587 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:56.875547 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g" Oct 30 16:59:57.311840 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:57.311072 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g"] Oct 30 16:59:57.327850 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:57.327177 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:57.517751 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:57.517709 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g" event={"ID":"181bd0c6-3a62-4ad2-8244-9987e0ac0606","Type":"ContainerStarted","Data":"fa680fc0df29b959edaf71fb77666c1ca9b1e7ce14c27f5d80623b04e1cb3cc4"} Oct 30 16:59:57.519366 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:57.519339 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf" event={"ID":"bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5","Type":"ContainerDied","Data":"3ae30d5c12d56951ba52f112aceac60fc41e69eb75b4f7b8e57bcc43eeb3020c"} Oct 30 16:59:57.519366 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:57.519362 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf" Oct 30 16:59:57.519580 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:57.519389 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ae30d5c12d56951ba52f112aceac60fc41e69eb75b4f7b8e57bcc43eeb3020c" Oct 30 16:59:58.592029 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.591116 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g" event={"ID":"181bd0c6-3a62-4ad2-8244-9987e0ac0606","Type":"ContainerStarted","Data":"f8745303692e66388c7ad2329f4eafbf0b57926a713dfcee8622693e29d22d3d"} Oct 30 16:59:58.600443 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.600399 6202 generic.go:356] "Generic (PLEG): container finished" podID="4775a051-8cf1-4771-ab83-0b31f6f9f40e" containerID="59fdb85be3a3aac56d9ef48c7fa6adea7eb8f7ae4f30b1099a15723f2a94f38d" exitCode=0 Oct 30 16:59:58.600740 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.600710 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" event={"ID":"4775a051-8cf1-4771-ab83-0b31f6f9f40e","Type":"ContainerDied","Data":"59fdb85be3a3aac56d9ef48c7fa6adea7eb8f7ae4f30b1099a15723f2a94f38d"} Oct 30 16:59:58.646848 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.646172 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g" podStartSLOduration=2.646149881 podStartE2EDuration="2.646149881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:58.644280931 +0000 UTC m=+2312.354735675" watchObservedRunningTime="2026-10-30 16:59:58.646149881 +0000 UTC m=+2312.356604626" Oct 30 16:59:58.687384 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.675881 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:59:58.789708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.784564 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-credential-socket\") pod \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " Oct 30 16:59:58.789708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.784676 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-workload-certs\") pod \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " Oct 30 16:59:58.789708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.784716 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-podinfo\") pod \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " Oct 30 16:59:58.789708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.784823 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-token\" (UniqueName: \"kubernetes.io/projected/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-token\") pod \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " Oct 30 16:59:58.789708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.784918 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-envoy\") pod \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " Oct 30 16:59:58.789708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.784957 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4775a051-8cf1-4771-ab83-0b31f6f9f40e-kube-api-access-vmltb\" (UniqueName: \"kubernetes.io/projected/4775a051-8cf1-4771-ab83-0b31f6f9f40e-kube-api-access-vmltb\") pod \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " Oct 30 16:59:58.789708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.785096 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-workload-socket\") pod \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " Oct 30 16:59:58.789708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.785142 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-workload-certs" pod "4775a051-8cf1-4771-ab83-0b31f6f9f40e" (UID: "4775a051-8cf1-4771-ab83-0b31f6f9f40e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:58.789708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.785261 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istiod-ca-cert\") pod \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " Oct 30 16:59:58.789708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.785338 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-data\") pod \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\" (UID: \"4775a051-8cf1-4771-ab83-0b31f6f9f40e\") " Oct 30 16:59:58.789708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.785379 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-credential-socket" pod "4775a051-8cf1-4771-ab83-0b31f6f9f40e" (UID: "4775a051-8cf1-4771-ab83-0b31f6f9f40e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:58.789708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.786086 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:58.789708 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.786109 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:58.832377 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.832228 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-workload-socket" pod "4775a051-8cf1-4771-ab83-0b31f6f9f40e" (UID: "4775a051-8cf1-4771-ab83-0b31f6f9f40e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:58.832638 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.832349 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istiod-ca-cert" pod "4775a051-8cf1-4771-ab83-0b31f6f9f40e" (UID: "4775a051-8cf1-4771-ab83-0b31f6f9f40e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:58.832860 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.832358 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-podinfo" pod "4775a051-8cf1-4771-ab83-0b31f6f9f40e" (UID: "4775a051-8cf1-4771-ab83-0b31f6f9f40e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:59:58.833531 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.833492 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-envoy" pod "4775a051-8cf1-4771-ab83-0b31f6f9f40e" (UID: "4775a051-8cf1-4771-ab83-0b31f6f9f40e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:58.834570 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.834536 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-data" pod "4775a051-8cf1-4771-ab83-0b31f6f9f40e" (UID: "4775a051-8cf1-4771-ab83-0b31f6f9f40e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:58.834926 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.834900 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4775a051-8cf1-4771-ab83-0b31f6f9f40e-kube-api-access-vmltb" pod "4775a051-8cf1-4771-ab83-0b31f6f9f40e" (UID: "4775a051-8cf1-4771-ab83-0b31f6f9f40e"). InnerVolumeSpecName "kube-api-access-vmltb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:58.835308 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.835283 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-token" pod "4775a051-8cf1-4771-ab83-0b31f6f9f40e" (UID: "4775a051-8cf1-4771-ab83-0b31f6f9f40e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:58.895670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.895536 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:58.895670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.895576 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:58.895670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.895592 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:58.895670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.895604 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:58.895670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.895616 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:58.895670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.895628 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4775a051-8cf1-4771-ab83-0b31f6f9f40e-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:58.895670 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:58.895640 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmltb\" (UniqueName: \"kubernetes.io/projected/4775a051-8cf1-4771-ab83-0b31f6f9f40e-kube-api-access-vmltb\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 16:59:59.610986 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:59.610957 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" Oct 30 16:59:59.611481 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:59.610987 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj" event={"ID":"4775a051-8cf1-4771-ab83-0b31f6f9f40e","Type":"ContainerDied","Data":"93cdca363853d97caa04ef9a23f6d96655098ac3b6e2e1f22fc4428af2d358eb"} Oct 30 16:59:59.611481 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:59.611049 6202 scope.go:122] "RemoveContainer" containerID="59fdb85be3a3aac56d9ef48c7fa6adea7eb8f7ae4f30b1099a15723f2a94f38d" Oct 30 16:59:59.613303 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:59.613272 6202 generic.go:356] "Generic (PLEG): container finished" podID="181bd0c6-3a62-4ad2-8244-9987e0ac0606" containerID="f8745303692e66388c7ad2329f4eafbf0b57926a713dfcee8622693e29d22d3d" exitCode=0 Oct 30 16:59:59.613448 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:59.613357 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g" event={"ID":"181bd0c6-3a62-4ad2-8244-9987e0ac0606","Type":"ContainerDied","Data":"f8745303692e66388c7ad2329f4eafbf0b57926a713dfcee8622693e29d22d3d"} Oct 30 16:59:59.679936 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:59.679890 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj"] Oct 30 16:59:59.691607 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:59.691573 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a59502-openshift-default-74889b7c5f-gntmj"] Oct 30 16:59:59.790283 ip-10-0-129-65 kubenswrapper[6202]: I1030 16:59:59.790248 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:00.177829 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.177785 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj"] Oct 30 17:00:00.178717 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.178696 6202 cpu_manager.go:414] "containerMap: removing container" podUID="4775a051-8cf1-4771-ab83-0b31f6f9f40e" containerName="istio-proxy" Oct 30 17:00:00.178717 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.178717 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4775a051-8cf1-4771-ab83-0b31f6f9f40e" containerName="istio-proxy" Oct 30 17:00:00.178917 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.178744 6202 cpu_manager.go:414] "containerMap: removing container" podUID="bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5" containerName="revoke-keys" Oct 30 17:00:00.178917 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.178750 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5" containerName="revoke-keys" Oct 30 17:00:00.179028 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.178949 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="4775a051-8cf1-4771-ab83-0b31f6f9f40e" containerName="istio-proxy" Oct 30 17:00:00.179028 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.178966 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5" containerName="revoke-keys" Oct 30 17:00:00.184119 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.184095 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47"] Oct 30 17:00:00.184241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.184225 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" Oct 30 17:00:00.189709 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.189685 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k"] Oct 30 17:00:00.189871 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.189851 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" Oct 30 17:00:00.195869 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.195308 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4"] Oct 30 17:00:00.195869 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.195508 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" Oct 30 17:00:00.200202 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.200181 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v"] Oct 30 17:00:00.200320 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.200301 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" Oct 30 17:00:00.204626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.204606 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" Oct 30 17:00:00.215429 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.215408 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj"] Oct 30 17:00:00.222027 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.221897 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47"] Oct 30 17:00:00.229684 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.229514 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r"] Oct 30 17:00:00.235673 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.235650 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc"] Oct 30 17:00:00.240905 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.240653 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" Oct 30 17:00:00.241159 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.241142 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4"] Oct 30 17:00:00.241388 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.241373 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" Oct 30 17:00:00.248550 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.247096 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp"] Oct 30 17:00:00.254587 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.254565 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh"] Oct 30 17:00:00.254843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.254827 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp" Oct 30 17:00:00.313926 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.313893 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v"] Oct 30 17:00:00.314099 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.314030 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" Oct 30 17:00:00.334366 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.334171 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6957d4c0-8197-4aab-a89a-ad17414cd011-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj\" (UID: \"6957d4c0-8197-4aab-a89a-ad17414cd011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" Oct 30 17:00:00.334366 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.334226 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d338cc2-8d69-4b14-99d1-370710f8cb99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47\" (UID: \"3d338cc2-8d69-4b14-99d1-370710f8cb99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" Oct 30 17:00:00.334988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.334439 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660c4258-30bc-4a2f-b130-766dcbbd88f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4\" (UID: \"660c4258-30bc-4a2f-b130-766dcbbd88f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" Oct 30 17:00:00.334988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.334471 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17f9002-4b15-4e46-a720-7a298c9c3f68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k\" (UID: \"c17f9002-4b15-4e46-a720-7a298c9c3f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" Oct 30 17:00:00.334988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.334559 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94v9q\" (UniqueName: \"kubernetes.io/projected/c17f9002-4b15-4e46-a720-7a298c9c3f68-kube-api-access-94v9q\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k\" (UID: \"c17f9002-4b15-4e46-a720-7a298c9c3f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" Oct 30 17:00:00.334988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.334580 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz7t2\" (UniqueName: \"kubernetes.io/projected/660c4258-30bc-4a2f-b130-766dcbbd88f4-kube-api-access-lz7t2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4\" (UID: \"660c4258-30bc-4a2f-b130-766dcbbd88f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" Oct 30 17:00:00.334988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.334598 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmbcs\" (UniqueName: \"kubernetes.io/projected/6957d4c0-8197-4aab-a89a-ad17414cd011-kube-api-access-bmbcs\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj\" (UID: \"6957d4c0-8197-4aab-a89a-ad17414cd011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" Oct 30 17:00:00.334988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.334641 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba09798-de28-4d76-8fea-839bbd8ea7e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889660-l5wqc\" (UID: \"5ba09798-de28-4d76-8fea-839bbd8ea7e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" Oct 30 17:00:00.334988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.334662 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c799e0b-668f-4c55-a254-69651dd5b005-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r\" (UID: \"8c799e0b-668f-4c55-a254-69651dd5b005\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" Oct 30 17:00:00.334988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.334699 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d8bm\" (UniqueName: \"kubernetes.io/projected/3d338cc2-8d69-4b14-99d1-370710f8cb99-kube-api-access-2d8bm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47\" (UID: \"3d338cc2-8d69-4b14-99d1-370710f8cb99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" Oct 30 17:00:00.334988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.334872 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxbw7\" (UniqueName: \"kubernetes.io/projected/7aadca90-6dbe-4fe5-9dee-10431e9210a0-kube-api-access-kxbw7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v\" (UID: \"7aadca90-6dbe-4fe5-9dee-10431e9210a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" Oct 30 17:00:00.334988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.334917 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aadca90-6dbe-4fe5-9dee-10431e9210a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v\" (UID: \"7aadca90-6dbe-4fe5-9dee-10431e9210a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" Oct 30 17:00:00.334988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.334949 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pwvq\" (UniqueName: \"kubernetes.io/projected/8c799e0b-668f-4c55-a254-69651dd5b005-kube-api-access-7pwvq\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r\" (UID: \"8c799e0b-668f-4c55-a254-69651dd5b005\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" Oct 30 17:00:00.335565 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.335082 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84r6t\" (UniqueName: \"kubernetes.io/projected/5ba09798-de28-4d76-8fea-839bbd8ea7e4-kube-api-access-84r6t\") pod \"maas-api-key-cleanup-29889660-l5wqc\" (UID: \"5ba09798-de28-4d76-8fea-839bbd8ea7e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" Oct 30 17:00:00.356789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.347280 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k"] Oct 30 17:00:00.360508 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.357098 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r"] Oct 30 17:00:00.369964 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.369831 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp"] Oct 30 17:00:00.379304 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.379077 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc"] Oct 30 17:00:00.381821 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.380796 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh"] Oct 30 17:00:00.388804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.387597 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm"] Oct 30 17:00:00.395993 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.394901 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n"] Oct 30 17:00:00.396589 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.396441 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" Oct 30 17:00:00.415681 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.415549 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t"] Oct 30 17:00:00.418194 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.418161 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" Oct 30 17:00:00.423954 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.423928 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm"] Oct 30 17:00:00.424061 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.424032 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n"] Oct 30 17:00:00.426795 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.425456 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.437185 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxbw7\" (UniqueName: \"kubernetes.io/projected/7aadca90-6dbe-4fe5-9dee-10431e9210a0-kube-api-access-kxbw7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v\" (UID: \"7aadca90-6dbe-4fe5-9dee-10431e9210a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.437227 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb4nr\" (UniqueName: \"kubernetes.io/projected/f9872bb2-dae7-43b4-9a03-6d0c9804cb79-kube-api-access-wb4nr\") pod \"maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp\" (UID: \"f9872bb2-dae7-43b4-9a03-6d0c9804cb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.437254 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aadca90-6dbe-4fe5-9dee-10431e9210a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v\" (UID: \"7aadca90-6dbe-4fe5-9dee-10431e9210a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.437276 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwvq\" (UniqueName: \"kubernetes.io/projected/8c799e0b-668f-4c55-a254-69651dd5b005-kube-api-access-7pwvq\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r\" (UID: \"8c799e0b-668f-4c55-a254-69651dd5b005\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.437305 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znkpz\" (UniqueName: \"kubernetes.io/projected/5c75b279-d01e-43e8-9a3e-8dff3dae29c7-kube-api-access-znkpz\") pod \"maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh\" (UID: \"5c75b279-d01e-43e8-9a3e-8dff3dae29c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.437369 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84r6t\" (UniqueName: \"kubernetes.io/projected/5ba09798-de28-4d76-8fea-839bbd8ea7e4-kube-api-access-84r6t\") pod \"maas-api-key-cleanup-29889660-l5wqc\" (UID: \"5ba09798-de28-4d76-8fea-839bbd8ea7e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.437398 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6957d4c0-8197-4aab-a89a-ad17414cd011-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj\" (UID: \"6957d4c0-8197-4aab-a89a-ad17414cd011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.437421 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d338cc2-8d69-4b14-99d1-370710f8cb99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47\" (UID: \"3d338cc2-8d69-4b14-99d1-370710f8cb99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.437492 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660c4258-30bc-4a2f-b130-766dcbbd88f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4\" (UID: \"660c4258-30bc-4a2f-b130-766dcbbd88f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.437508 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9872bb2-dae7-43b4-9a03-6d0c9804cb79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp\" (UID: \"f9872bb2-dae7-43b4-9a03-6d0c9804cb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.437528 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17f9002-4b15-4e46-a720-7a298c9c3f68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k\" (UID: \"c17f9002-4b15-4e46-a720-7a298c9c3f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.437571 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94v9q\" (UniqueName: \"kubernetes.io/projected/c17f9002-4b15-4e46-a720-7a298c9c3f68-kube-api-access-94v9q\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k\" (UID: \"c17f9002-4b15-4e46-a720-7a298c9c3f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.437588 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz7t2\" (UniqueName: \"kubernetes.io/projected/660c4258-30bc-4a2f-b130-766dcbbd88f4-kube-api-access-lz7t2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4\" (UID: \"660c4258-30bc-4a2f-b130-766dcbbd88f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.437604 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmbcs\" (UniqueName: \"kubernetes.io/projected/6957d4c0-8197-4aab-a89a-ad17414cd011-kube-api-access-bmbcs\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj\" (UID: \"6957d4c0-8197-4aab-a89a-ad17414cd011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.437646 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba09798-de28-4d76-8fea-839bbd8ea7e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889660-l5wqc\" (UID: \"5ba09798-de28-4d76-8fea-839bbd8ea7e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.437662 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c75b279-d01e-43e8-9a3e-8dff3dae29c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh\" (UID: \"5c75b279-d01e-43e8-9a3e-8dff3dae29c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.437680 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c799e0b-668f-4c55-a254-69651dd5b005-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r\" (UID: \"8c799e0b-668f-4c55-a254-69651dd5b005\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.437711 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d8bm\" (UniqueName: \"kubernetes.io/projected/3d338cc2-8d69-4b14-99d1-370710f8cb99-kube-api-access-2d8bm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47\" (UID: \"3d338cc2-8d69-4b14-99d1-370710f8cb99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" Oct 30 17:00:00.438805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.438728 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aadca90-6dbe-4fe5-9dee-10431e9210a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v\" (UID: \"7aadca90-6dbe-4fe5-9dee-10431e9210a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" Oct 30 17:00:00.448878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.440030 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6957d4c0-8197-4aab-a89a-ad17414cd011-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj\" (UID: \"6957d4c0-8197-4aab-a89a-ad17414cd011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" Oct 30 17:00:00.448878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.440620 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d338cc2-8d69-4b14-99d1-370710f8cb99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47\" (UID: \"3d338cc2-8d69-4b14-99d1-370710f8cb99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" Oct 30 17:00:00.448878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.442113 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c799e0b-668f-4c55-a254-69651dd5b005-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r\" (UID: \"8c799e0b-668f-4c55-a254-69651dd5b005\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" Oct 30 17:00:00.448878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.442696 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660c4258-30bc-4a2f-b130-766dcbbd88f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4\" (UID: \"660c4258-30bc-4a2f-b130-766dcbbd88f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" Oct 30 17:00:00.448878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.443301 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17f9002-4b15-4e46-a720-7a298c9c3f68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k\" (UID: \"c17f9002-4b15-4e46-a720-7a298c9c3f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" Oct 30 17:00:00.448878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.444421 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba09798-de28-4d76-8fea-839bbd8ea7e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889660-l5wqc\" (UID: \"5ba09798-de28-4d76-8fea-839bbd8ea7e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" Oct 30 17:00:00.448878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.445146 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4775a051-8cf1-4771-ab83-0b31f6f9f40e" path="/var/lib/kubelet/pods/4775a051-8cf1-4771-ab83-0b31f6f9f40e/volumes" Oct 30 17:00:00.448878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.445630 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t"] Oct 30 17:00:00.448878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.445680 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh"] Oct 30 17:00:00.452408 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.452386 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr"] Oct 30 17:00:00.453219 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.452637 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" Oct 30 17:00:00.458924 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.458902 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh"] Oct 30 17:00:00.459039 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.458946 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2"] Oct 30 17:00:00.460110 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.460090 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" Oct 30 17:00:00.466337 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.466315 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84r6t\" (UniqueName: \"kubernetes.io/projected/5ba09798-de28-4d76-8fea-839bbd8ea7e4-kube-api-access-84r6t\") pod \"maas-api-key-cleanup-29889660-l5wqc\" (UID: \"5ba09798-de28-4d76-8fea-839bbd8ea7e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" Oct 30 17:00:00.468813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.468791 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr"] Oct 30 17:00:00.468909 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.468892 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" Oct 30 17:00:00.469499 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.469435 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxbw7\" (UniqueName: \"kubernetes.io/projected/7aadca90-6dbe-4fe5-9dee-10431e9210a0-kube-api-access-kxbw7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v\" (UID: \"7aadca90-6dbe-4fe5-9dee-10431e9210a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" Oct 30 17:00:00.471974 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.470563 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94v9q\" (UniqueName: \"kubernetes.io/projected/c17f9002-4b15-4e46-a720-7a298c9c3f68-kube-api-access-94v9q\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k\" (UID: \"c17f9002-4b15-4e46-a720-7a298c9c3f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" Oct 30 17:00:00.471974 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.471334 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pwvq\" (UniqueName: \"kubernetes.io/projected/8c799e0b-668f-4c55-a254-69651dd5b005-kube-api-access-7pwvq\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r\" (UID: \"8c799e0b-668f-4c55-a254-69651dd5b005\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" Oct 30 17:00:00.473839 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.473067 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz7t2\" (UniqueName: \"kubernetes.io/projected/660c4258-30bc-4a2f-b130-766dcbbd88f4-kube-api-access-lz7t2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4\" (UID: \"660c4258-30bc-4a2f-b130-766dcbbd88f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" Oct 30 17:00:00.473839 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.473342 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz"] Oct 30 17:00:00.474556 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.474427 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d8bm\" (UniqueName: \"kubernetes.io/projected/3d338cc2-8d69-4b14-99d1-370710f8cb99-kube-api-access-2d8bm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47\" (UID: \"3d338cc2-8d69-4b14-99d1-370710f8cb99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" Oct 30 17:00:00.474556 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.474483 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmbcs\" (UniqueName: \"kubernetes.io/projected/6957d4c0-8197-4aab-a89a-ad17414cd011-kube-api-access-bmbcs\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj\" (UID: \"6957d4c0-8197-4aab-a89a-ad17414cd011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" Oct 30 17:00:00.481994 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.481967 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" Oct 30 17:00:00.484110 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.484084 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2"] Oct 30 17:00:00.487878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.487844 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz"] Oct 30 17:00:00.505904 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.505882 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" Oct 30 17:00:00.524620 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.524591 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" Oct 30 17:00:00.538964 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.538931 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9872bb2-dae7-43b4-9a03-6d0c9804cb79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp\" (UID: \"f9872bb2-dae7-43b4-9a03-6d0c9804cb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp" Oct 30 17:00:00.539097 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.539082 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c75b279-d01e-43e8-9a3e-8dff3dae29c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh\" (UID: \"5c75b279-d01e-43e8-9a3e-8dff3dae29c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" Oct 30 17:00:00.539171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.539147 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0015d61c-9d93-4652-9414-3b817ab96276-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm\" (UID: \"0015d61c-9d93-4652-9414-3b817ab96276\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" Oct 30 17:00:00.539259 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.539190 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b2402f-79e8-4586-9f36-2b86f307f1cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t\" (UID: \"c0b2402f-79e8-4586-9f36-2b86f307f1cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" Oct 30 17:00:00.539259 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.539222 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s45pg\" (UniqueName: \"kubernetes.io/projected/0015d61c-9d93-4652-9414-3b817ab96276-kube-api-access-s45pg\") pod \"maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm\" (UID: \"0015d61c-9d93-4652-9414-3b817ab96276\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" Oct 30 17:00:00.539514 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.539489 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb4nr\" (UniqueName: \"kubernetes.io/projected/f9872bb2-dae7-43b4-9a03-6d0c9804cb79-kube-api-access-wb4nr\") pod \"maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp\" (UID: \"f9872bb2-dae7-43b4-9a03-6d0c9804cb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp" Oct 30 17:00:00.539595 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.539574 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znkpz\" (UniqueName: \"kubernetes.io/projected/5c75b279-d01e-43e8-9a3e-8dff3dae29c7-kube-api-access-znkpz\") pod \"maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh\" (UID: \"5c75b279-d01e-43e8-9a3e-8dff3dae29c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" Oct 30 17:00:00.539647 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.539630 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfgvs\" (UniqueName: \"kubernetes.io/projected/c0b2402f-79e8-4586-9f36-2b86f307f1cb-kube-api-access-bfgvs\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t\" (UID: \"c0b2402f-79e8-4586-9f36-2b86f307f1cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" Oct 30 17:00:00.539732 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.539713 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09a458f-a69b-441a-b91a-11854c180a4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n\" (UID: \"b09a458f-a69b-441a-b91a-11854c180a4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" Oct 30 17:00:00.539867 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.539846 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffcx4\" (UniqueName: \"kubernetes.io/projected/b09a458f-a69b-441a-b91a-11854c180a4e-kube-api-access-ffcx4\") pod \"maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n\" (UID: \"b09a458f-a69b-441a-b91a-11854c180a4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" Oct 30 17:00:00.540491 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.540347 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c75b279-d01e-43e8-9a3e-8dff3dae29c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh\" (UID: \"5c75b279-d01e-43e8-9a3e-8dff3dae29c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" Oct 30 17:00:00.541051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.541029 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9872bb2-dae7-43b4-9a03-6d0c9804cb79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp\" (UID: \"f9872bb2-dae7-43b4-9a03-6d0c9804cb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp" Oct 30 17:00:00.553451 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.551095 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" Oct 30 17:00:00.556656 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.555053 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" Oct 30 17:00:00.591957 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.584881 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znkpz\" (UniqueName: \"kubernetes.io/projected/5c75b279-d01e-43e8-9a3e-8dff3dae29c7-kube-api-access-znkpz\") pod \"maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh\" (UID: \"5c75b279-d01e-43e8-9a3e-8dff3dae29c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" Oct 30 17:00:00.591957 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.588536 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb4nr\" (UniqueName: \"kubernetes.io/projected/f9872bb2-dae7-43b4-9a03-6d0c9804cb79-kube-api-access-wb4nr\") pod \"maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp\" (UID: \"f9872bb2-dae7-43b4-9a03-6d0c9804cb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp" Oct 30 17:00:00.621018 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.619962 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" Oct 30 17:00:00.633873 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.633591 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" Oct 30 17:00:00.645173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.641372 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5px6\" (UniqueName: \"kubernetes.io/projected/ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9-kube-api-access-k5px6\") pod \"maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2\" (UID: \"ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" Oct 30 17:00:00.645173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.641437 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz\" (UID: \"bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" Oct 30 17:00:00.645173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.641522 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7d7d\" (UniqueName: \"kubernetes.io/projected/bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5-kube-api-access-z7d7d\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz\" (UID: \"bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" Oct 30 17:00:00.645173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.641556 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0015d61c-9d93-4652-9414-3b817ab96276-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm\" (UID: \"0015d61c-9d93-4652-9414-3b817ab96276\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" Oct 30 17:00:00.645173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.641588 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b2402f-79e8-4586-9f36-2b86f307f1cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t\" (UID: \"c0b2402f-79e8-4586-9f36-2b86f307f1cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" Oct 30 17:00:00.645173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.641615 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s45pg\" (UniqueName: \"kubernetes.io/projected/0015d61c-9d93-4652-9414-3b817ab96276-kube-api-access-s45pg\") pod \"maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm\" (UID: \"0015d61c-9d93-4652-9414-3b817ab96276\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" Oct 30 17:00:00.645173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.641697 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7550e3d-3bc1-4ff5-a641-0d68b4bf6127-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr\" (UID: \"c7550e3d-3bc1-4ff5-a641-0d68b4bf6127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" Oct 30 17:00:00.645173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.641861 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgvs\" (UniqueName: \"kubernetes.io/projected/c0b2402f-79e8-4586-9f36-2b86f307f1cb-kube-api-access-bfgvs\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t\" (UID: \"c0b2402f-79e8-4586-9f36-2b86f307f1cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" Oct 30 17:00:00.645173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.641894 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p78m5\" (UniqueName: \"kubernetes.io/projected/c7550e3d-3bc1-4ff5-a641-0d68b4bf6127-kube-api-access-p78m5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr\" (UID: \"c7550e3d-3bc1-4ff5-a641-0d68b4bf6127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" Oct 30 17:00:00.645173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.641952 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09a458f-a69b-441a-b91a-11854c180a4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n\" (UID: \"b09a458f-a69b-441a-b91a-11854c180a4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" Oct 30 17:00:00.645173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.642012 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2\" (UID: \"ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" Oct 30 17:00:00.645173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.642044 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffcx4\" (UniqueName: \"kubernetes.io/projected/b09a458f-a69b-441a-b91a-11854c180a4e-kube-api-access-ffcx4\") pod \"maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n\" (UID: \"b09a458f-a69b-441a-b91a-11854c180a4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" Oct 30 17:00:00.645173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.642084 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75af19e6-9fae-4786-86d2-3ef7f4433f08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh\" (UID: \"75af19e6-9fae-4786-86d2-3ef7f4433f08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" Oct 30 17:00:00.645173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.642165 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whb9s\" (UniqueName: \"kubernetes.io/projected/75af19e6-9fae-4786-86d2-3ef7f4433f08-kube-api-access-whb9s\") pod \"maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh\" (UID: \"75af19e6-9fae-4786-86d2-3ef7f4433f08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" Oct 30 17:00:00.645173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.644564 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0015d61c-9d93-4652-9414-3b817ab96276-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm\" (UID: \"0015d61c-9d93-4652-9414-3b817ab96276\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" Oct 30 17:00:00.645173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.644732 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b2402f-79e8-4586-9f36-2b86f307f1cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t\" (UID: \"c0b2402f-79e8-4586-9f36-2b86f307f1cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" Oct 30 17:00:00.645726 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.645274 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09a458f-a69b-441a-b91a-11854c180a4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n\" (UID: \"b09a458f-a69b-441a-b91a-11854c180a4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" Oct 30 17:00:00.666217 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.664563 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" Oct 30 17:00:00.675102 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.675015 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffcx4\" (UniqueName: \"kubernetes.io/projected/b09a458f-a69b-441a-b91a-11854c180a4e-kube-api-access-ffcx4\") pod \"maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n\" (UID: \"b09a458f-a69b-441a-b91a-11854c180a4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" Oct 30 17:00:00.681584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.677592 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s45pg\" (UniqueName: \"kubernetes.io/projected/0015d61c-9d93-4652-9414-3b817ab96276-kube-api-access-s45pg\") pod \"maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm\" (UID: \"0015d61c-9d93-4652-9414-3b817ab96276\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" Oct 30 17:00:00.681584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.681505 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgvs\" (UniqueName: \"kubernetes.io/projected/c0b2402f-79e8-4586-9f36-2b86f307f1cb-kube-api-access-bfgvs\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t\" (UID: \"c0b2402f-79e8-4586-9f36-2b86f307f1cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" Oct 30 17:00:00.696372 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.696337 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp" Oct 30 17:00:00.707055 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.706137 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" Oct 30 17:00:00.727546 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.725730 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" Oct 30 17:00:00.789121 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.787662 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" Oct 30 17:00:00.789878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.789288 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p78m5\" (UniqueName: \"kubernetes.io/projected/c7550e3d-3bc1-4ff5-a641-0d68b4bf6127-kube-api-access-p78m5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr\" (UID: \"c7550e3d-3bc1-4ff5-a641-0d68b4bf6127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" Oct 30 17:00:00.789878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.789423 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2\" (UID: \"ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" Oct 30 17:00:00.789878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.789472 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75af19e6-9fae-4786-86d2-3ef7f4433f08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh\" (UID: \"75af19e6-9fae-4786-86d2-3ef7f4433f08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" Oct 30 17:00:00.789878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.789550 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whb9s\" (UniqueName: \"kubernetes.io/projected/75af19e6-9fae-4786-86d2-3ef7f4433f08-kube-api-access-whb9s\") pod \"maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh\" (UID: \"75af19e6-9fae-4786-86d2-3ef7f4433f08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" Oct 30 17:00:00.789878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.789607 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5px6\" (UniqueName: \"kubernetes.io/projected/ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9-kube-api-access-k5px6\") pod \"maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2\" (UID: \"ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" Oct 30 17:00:00.789878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.789656 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz\" (UID: \"bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" Oct 30 17:00:00.789878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.789749 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7d7d\" (UniqueName: \"kubernetes.io/projected/bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5-kube-api-access-z7d7d\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz\" (UID: \"bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" Oct 30 17:00:00.792836 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.790442 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7550e3d-3bc1-4ff5-a641-0d68b4bf6127-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr\" (UID: \"c7550e3d-3bc1-4ff5-a641-0d68b4bf6127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" Oct 30 17:00:00.792836 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.791147 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7550e3d-3bc1-4ff5-a641-0d68b4bf6127-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr\" (UID: \"c7550e3d-3bc1-4ff5-a641-0d68b4bf6127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" Oct 30 17:00:00.792836 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.791822 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2\" (UID: \"ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" Oct 30 17:00:00.792836 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.792322 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75af19e6-9fae-4786-86d2-3ef7f4433f08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh\" (UID: \"75af19e6-9fae-4786-86d2-3ef7f4433f08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" Oct 30 17:00:00.793416 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.793337 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz\" (UID: \"bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" Oct 30 17:00:00.800827 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.799738 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" Oct 30 17:00:00.838027 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.837834 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7d7d\" (UniqueName: \"kubernetes.io/projected/bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5-kube-api-access-z7d7d\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz\" (UID: \"bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" Oct 30 17:00:00.838692 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.838609 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5px6\" (UniqueName: \"kubernetes.io/projected/ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9-kube-api-access-k5px6\") pod \"maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2\" (UID: \"ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" Oct 30 17:00:00.846602 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.846530 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p78m5\" (UniqueName: \"kubernetes.io/projected/c7550e3d-3bc1-4ff5-a641-0d68b4bf6127-kube-api-access-p78m5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr\" (UID: \"c7550e3d-3bc1-4ff5-a641-0d68b4bf6127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" Oct 30 17:00:00.846844 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.846736 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whb9s\" (UniqueName: \"kubernetes.io/projected/75af19e6-9fae-4786-86d2-3ef7f4433f08-kube-api-access-whb9s\") pod \"maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh\" (UID: \"75af19e6-9fae-4786-86d2-3ef7f4433f08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" Oct 30 17:00:00.919005 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.918807 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" Oct 30 17:00:00.936283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:00.936089 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" Oct 30 17:00:01.042608 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.042561 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj"] Oct 30 17:00:01.066051 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:00:01.066010 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6957d4c0_8197_4aab_a89a_ad17414cd011.slice/crio-a66fb2e86c910c9d0af2512fdf19e082553694abd8ff69c7027b5ba4a1df3e88 WatchSource:0}: Error finding container a66fb2e86c910c9d0af2512fdf19e082553694abd8ff69c7027b5ba4a1df3e88: Status 404 returned error can't find the container with id a66fb2e86c910c9d0af2512fdf19e082553694abd8ff69c7027b5ba4a1df3e88 Oct 30 17:00:01.087561 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.087524 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47"] Oct 30 17:00:01.131394 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.131354 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" Oct 30 17:00:01.132782 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.131758 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" Oct 30 17:00:01.199116 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.199085 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4"] Oct 30 17:00:01.230996 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.226076 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g" Oct 30 17:00:01.366057 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.365460 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181bd0c6-3a62-4ad2-8244-9987e0ac0606-kube-api-access-876xd\" (UniqueName: \"kubernetes.io/projected/181bd0c6-3a62-4ad2-8244-9987e0ac0606-kube-api-access-876xd\") pod \"181bd0c6-3a62-4ad2-8244-9987e0ac0606\" (UID: \"181bd0c6-3a62-4ad2-8244-9987e0ac0606\") " Oct 30 17:00:01.366057 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.365553 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181bd0c6-3a62-4ad2-8244-9987e0ac0606-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181bd0c6-3a62-4ad2-8244-9987e0ac0606-maas-api-openshift-ca\") pod \"181bd0c6-3a62-4ad2-8244-9987e0ac0606\" (UID: \"181bd0c6-3a62-4ad2-8244-9987e0ac0606\") " Oct 30 17:00:01.366057 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.365730 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181bd0c6-3a62-4ad2-8244-9987e0ac0606-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181bd0c6-3a62-4ad2-8244-9987e0ac0606-maas-api-xks-ca\") pod \"181bd0c6-3a62-4ad2-8244-9987e0ac0606\" (UID: \"181bd0c6-3a62-4ad2-8244-9987e0ac0606\") " Oct 30 17:00:01.366896 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.366588 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181bd0c6-3a62-4ad2-8244-9987e0ac0606-maas-api-openshift-ca" pod "181bd0c6-3a62-4ad2-8244-9987e0ac0606" (UID: "181bd0c6-3a62-4ad2-8244-9987e0ac0606"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:01.367059 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.367030 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181bd0c6-3a62-4ad2-8244-9987e0ac0606-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:01.400516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.400482 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k"] Oct 30 17:00:01.443536 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.442881 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181bd0c6-3a62-4ad2-8244-9987e0ac0606-maas-api-xks-ca" pod "181bd0c6-3a62-4ad2-8244-9987e0ac0606" (UID: "181bd0c6-3a62-4ad2-8244-9987e0ac0606"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:01.445732 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.445671 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181bd0c6-3a62-4ad2-8244-9987e0ac0606-kube-api-access-876xd" pod "181bd0c6-3a62-4ad2-8244-9987e0ac0606" (UID: "181bd0c6-3a62-4ad2-8244-9987e0ac0606"). InnerVolumeSpecName "kube-api-access-876xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:01.477096 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.469270 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-876xd\" (UniqueName: \"kubernetes.io/projected/181bd0c6-3a62-4ad2-8244-9987e0ac0606-kube-api-access-876xd\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:01.477096 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.469308 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181bd0c6-3a62-4ad2-8244-9987e0ac0606-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:01.609209 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.609145 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp"] Oct 30 17:00:01.617073 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:00:01.617038 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9872bb2_dae7_43b4_9a03_6d0c9804cb79.slice/crio-211eb779680b463ddef24e357fbbfbc647f0e1d6ea4b7f124bc3c45ef3ca7331 WatchSource:0}: Error finding container 211eb779680b463ddef24e357fbbfbc647f0e1d6ea4b7f124bc3c45ef3ca7331: Status 404 returned error can't find the container with id 211eb779680b463ddef24e357fbbfbc647f0e1d6ea4b7f124bc3c45ef3ca7331 Oct 30 17:00:01.627203 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.627109 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n"] Oct 30 17:00:01.646341 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.644300 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm"] Oct 30 17:00:01.657972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.657943 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v"] Oct 30 17:00:01.667372 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.667166 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t"] Oct 30 17:00:01.675218 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.675185 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc"] Oct 30 17:00:01.682171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.681715 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 17:00:01.689900 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.689819 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" event={"ID":"c17f9002-4b15-4e46-a720-7a298c9c3f68","Type":"ContainerStarted","Data":"2ad9d641a213a1ff614f4d5764ff1c3d7fa2706902fc049cd95d2419cd748205"} Oct 30 17:00:01.691861 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.691825 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" event={"ID":"3d338cc2-8d69-4b14-99d1-370710f8cb99","Type":"ContainerStarted","Data":"ebd0c2785abe43b3867bdc641517a8543655965cfd165bbb97797e9f7f6a5a1e"} Oct 30 17:00:01.693869 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.693814 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g" Oct 30 17:00:01.693869 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.693826 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g" event={"ID":"181bd0c6-3a62-4ad2-8244-9987e0ac0606","Type":"ContainerDied","Data":"fa680fc0df29b959edaf71fb77666c1ca9b1e7ce14c27f5d80623b04e1cb3cc4"} Oct 30 17:00:01.703116 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.693871 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa680fc0df29b959edaf71fb77666c1ca9b1e7ce14c27f5d80623b04e1cb3cc4" Oct 30 17:00:01.703116 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.696730 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" event={"ID":"0015d61c-9d93-4652-9414-3b817ab96276","Type":"ContainerStarted","Data":"c7bc739f4037063c450d01b6478295b8973e9eee0f75b142dae83a9dbc89689a"} Oct 30 17:00:01.703116 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.700908 6202 generic.go:356] "Generic (PLEG): container finished" podID="927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" containerID="9b88b039b2e82009b766fe282e75aa79c4426a72e970bf35e3edf5bea53d2646" exitCode=0 Oct 30 17:00:01.703116 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.701091 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" event={"ID":"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5","Type":"ContainerDied","Data":"9b88b039b2e82009b766fe282e75aa79c4426a72e970bf35e3edf5bea53d2646"} Oct 30 17:00:01.703116 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.701137 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" event={"ID":"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5","Type":"ContainerDied","Data":"46d7b357e7c26bcab7253dbc33f1735a851f89c70abe52b5d6c66b0eca0c9d54"} Oct 30 17:00:01.703116 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.701167 6202 scope.go:122] "RemoveContainer" containerID="9b88b039b2e82009b766fe282e75aa79c4426a72e970bf35e3edf5bea53d2646" Oct 30 17:00:01.704493 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.704406 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" event={"ID":"660c4258-30bc-4a2f-b130-766dcbbd88f4","Type":"ContainerStarted","Data":"117fad5af6d36c80ddddaf7b8e2360b02ca86276ff9824181a5cb8174a1fbf49"} Oct 30 17:00:01.747087 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.743225 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r"] Oct 30 17:00:01.747087 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.744397 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:01.747087 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.744698 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh" Oct 30 17:00:01.751485 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.749902 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp" event={"ID":"f9872bb2-dae7-43b4-9a03-6d0c9804cb79","Type":"ContainerStarted","Data":"211eb779680b463ddef24e357fbbfbc647f0e1d6ea4b7f124bc3c45ef3ca7331"} Oct 30 17:00:01.771040 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:00:01.770947 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aadca90_6dbe_4fe5_9dee_10431e9210a0.slice/crio-55fe104d78099de8d226ce1572bf55e0aec8169a9dffe29931206b866fcfe2f1 WatchSource:0}: Error finding container 55fe104d78099de8d226ce1572bf55e0aec8169a9dffe29931206b866fcfe2f1: Status 404 returned error can't find the container with id 55fe104d78099de8d226ce1572bf55e0aec8169a9dffe29931206b866fcfe2f1 Oct 30 17:00:01.772612 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.772382 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" event={"ID":"b09a458f-a69b-441a-b91a-11854c180a4e","Type":"ContainerStarted","Data":"7ff08e8f6ef2584961eda65c68b17a3054e9bcd6eb8344cfcc99ebeaea9be883"} Oct 30 17:00:01.786922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.774503 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-data\") pod \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " Oct 30 17:00:01.786922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.774547 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istiod-ca-cert\") pod \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " Oct 30 17:00:01.786922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.774724 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-workload-socket\") pod \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " Oct 30 17:00:01.786922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.774801 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-kube-api-access-vkrjw\" (UniqueName: \"kubernetes.io/projected/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-kube-api-access-vkrjw\") pod \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " Oct 30 17:00:01.786922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.774892 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-token\" (UniqueName: \"kubernetes.io/projected/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-token\") pod \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " Oct 30 17:00:01.786922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.774965 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-envoy\") pod \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " Oct 30 17:00:01.786922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.774996 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-podinfo\") pod \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " Oct 30 17:00:01.786922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.775143 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-workload-certs\") pod \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " Oct 30 17:00:01.786922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.775184 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-credential-socket\") pod \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\" (UID: \"927cb01d-5133-4ecd-abb7-fa13f1fbe8c5\") " Oct 30 17:00:01.786922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.776101 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-credential-socket" pod "927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" (UID: "927cb01d-5133-4ecd-abb7-fa13f1fbe8c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:01.786922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.776338 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-data" pod "927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" (UID: "927cb01d-5133-4ecd-abb7-fa13f1fbe8c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:01.786922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.776985 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istiod-ca-cert" pod "927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" (UID: "927cb01d-5133-4ecd-abb7-fa13f1fbe8c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:01.786922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.778064 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-workload-socket" pod "927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" (UID: "927cb01d-5133-4ecd-abb7-fa13f1fbe8c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:01.786922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.779868 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-workload-certs" pod "927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" (UID: "927cb01d-5133-4ecd-abb7-fa13f1fbe8c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:01.786922 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:00:01.785310 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba09798_de28_4d76_8fea_839bbd8ea7e4.slice/crio-94835a422ba61196d754951c9120cd9a54db07e5d428aab137c2206388f92325 WatchSource:0}: Error finding container 94835a422ba61196d754951c9120cd9a54db07e5d428aab137c2206388f92325: Status 404 returned error can't find the container with id 94835a422ba61196d754951c9120cd9a54db07e5d428aab137c2206388f92325 Oct 30 17:00:01.794902 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.787417 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" event={"ID":"6957d4c0-8197-4aab-a89a-ad17414cd011","Type":"ContainerStarted","Data":"a66fb2e86c910c9d0af2512fdf19e082553694abd8ff69c7027b5ba4a1df3e88"} Oct 30 17:00:01.794902 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.788253 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-token" pod "927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" (UID: "927cb01d-5133-4ecd-abb7-fa13f1fbe8c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:01.794902 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.788977 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-envoy" pod "927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" (UID: "927cb01d-5133-4ecd-abb7-fa13f1fbe8c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:01.794902 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.789013 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-kube-api-access-vkrjw" pod "927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" (UID: "927cb01d-5133-4ecd-abb7-fa13f1fbe8c5"). InnerVolumeSpecName "kube-api-access-vkrjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:01.794902 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:00:01.792080 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c799e0b_668f_4c55_a254_69651dd5b005.slice/crio-4f401a56aa157f20e31cc67511c3cf96f02d022a4a08e89373971fce9dfb0a58 WatchSource:0}: Error finding container 4f401a56aa157f20e31cc67511c3cf96f02d022a4a08e89373971fce9dfb0a58: Status 404 returned error can't find the container with id 4f401a56aa157f20e31cc67511c3cf96f02d022a4a08e89373971fce9dfb0a58 Oct 30 17:00:01.794902 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.792450 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-podinfo" pod "927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" (UID: "927cb01d-5133-4ecd-abb7-fa13f1fbe8c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:00:01.843349 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:00:01.820592 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b2402f_79e8_4586_9f36_2b86f307f1cb.slice/crio-152546631ab13644f4e99445bc0c90d4246d7df181b39e1289b41804ee236b14 WatchSource:0}: Error finding container 152546631ab13644f4e99445bc0c90d4246d7df181b39e1289b41804ee236b14: Status 404 returned error can't find the container with id 152546631ab13644f4e99445bc0c90d4246d7df181b39e1289b41804ee236b14 Oct 30 17:00:01.843349 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.840741 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz"] Oct 30 17:00:01.846596 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.846566 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2"] Oct 30 17:00:01.852831 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.852801 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr"] Oct 30 17:00:01.854912 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.854886 6202 scope.go:122] "RemoveContainer" containerID="c8c0a858d4907d0a55f065d3af3b8eeef108dd178fe3f6e47f43304fbfbdef15" Oct 30 17:00:01.858701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.858549 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh"] Oct 30 17:00:01.877709 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.877667 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:01.877709 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.877701 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkrjw\" (UniqueName: \"kubernetes.io/projected/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-kube-api-access-vkrjw\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:01.877709 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.877719 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:01.878046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.877728 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:01.878046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.877736 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:01.878046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.877745 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:01.878046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.877752 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:01.878046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.877778 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:01.878046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.877790 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:01.898644 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:01.898610 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh"] Oct 30 17:00:01.942217 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:00:01.942183 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c75b279_d01e_43e8_9a3e_8dff3dae29c7.slice/crio-ed29b314303a5fb6305f4bbdd68b36bb896aeafa43c346666d2736808beb907e WatchSource:0}: Error finding container ed29b314303a5fb6305f4bbdd68b36bb896aeafa43c346666d2736808beb907e: Status 404 returned error can't find the container with id ed29b314303a5fb6305f4bbdd68b36bb896aeafa43c346666d2736808beb907e Oct 30 17:00:02.044847 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:00:02.043203 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad6a2e21_ad1d_4f11_9613_0e5c5fa536e9.slice/crio-9ea9932c9100021fe5a7eafac7733e84c7e534e1e4c94c22af9a0d2ba91f364e WatchSource:0}: Error finding container 9ea9932c9100021fe5a7eafac7733e84c7e534e1e4c94c22af9a0d2ba91f364e: Status 404 returned error can't find the container with id 9ea9932c9100021fe5a7eafac7733e84c7e534e1e4c94c22af9a0d2ba91f364e Oct 30 17:00:02.058837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.058807 6202 scope.go:122] "RemoveContainer" containerID="9b88b039b2e82009b766fe282e75aa79c4426a72e970bf35e3edf5bea53d2646" Oct 30 17:00:02.059325 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:02.059286 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b88b039b2e82009b766fe282e75aa79c4426a72e970bf35e3edf5bea53d2646\": container with ID starting with 9b88b039b2e82009b766fe282e75aa79c4426a72e970bf35e3edf5bea53d2646 not found: ID does not exist" containerID="9b88b039b2e82009b766fe282e75aa79c4426a72e970bf35e3edf5bea53d2646" Oct 30 17:00:02.059437 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.059336 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b88b039b2e82009b766fe282e75aa79c4426a72e970bf35e3edf5bea53d2646"} err="failed to get container status \"9b88b039b2e82009b766fe282e75aa79c4426a72e970bf35e3edf5bea53d2646\": rpc error: code = NotFound desc = could not find container \"9b88b039b2e82009b766fe282e75aa79c4426a72e970bf35e3edf5bea53d2646\": container with ID starting with 9b88b039b2e82009b766fe282e75aa79c4426a72e970bf35e3edf5bea53d2646 not found: ID does not exist" Oct 30 17:00:02.059437 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.059367 6202 scope.go:122] "RemoveContainer" containerID="c8c0a858d4907d0a55f065d3af3b8eeef108dd178fe3f6e47f43304fbfbdef15" Oct 30 17:00:02.060104 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:02.060078 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8c0a858d4907d0a55f065d3af3b8eeef108dd178fe3f6e47f43304fbfbdef15\": container with ID starting with c8c0a858d4907d0a55f065d3af3b8eeef108dd178fe3f6e47f43304fbfbdef15 not found: ID does not exist" containerID="c8c0a858d4907d0a55f065d3af3b8eeef108dd178fe3f6e47f43304fbfbdef15" Oct 30 17:00:02.060222 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.060107 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c0a858d4907d0a55f065d3af3b8eeef108dd178fe3f6e47f43304fbfbdef15"} err="failed to get container status \"c8c0a858d4907d0a55f065d3af3b8eeef108dd178fe3f6e47f43304fbfbdef15\": rpc error: code = NotFound desc = could not find container \"c8c0a858d4907d0a55f065d3af3b8eeef108dd178fe3f6e47f43304fbfbdef15\": container with ID starting with c8c0a858d4907d0a55f065d3af3b8eeef108dd178fe3f6e47f43304fbfbdef15 not found: ID does not exist" Oct 30 17:00:02.069456 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:00:02.069429 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75af19e6_9fae_4786_86d2_3ef7f4433f08.slice/crio-c746797433518a53eb208665d8f16c411d366b2177f5d17cd9a28ae383df1498 WatchSource:0}: Error finding container c746797433518a53eb208665d8f16c411d366b2177f5d17cd9a28ae383df1498: Status 404 returned error can't find the container with id c746797433518a53eb208665d8f16c411d366b2177f5d17cd9a28ae383df1498 Oct 30 17:00:02.131593 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.131515 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh"] Oct 30 17:00:02.136714 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.136677 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-668047-openshift-default-c49c74d65-p65gh"] Oct 30 17:00:02.281534 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.281495 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl"] Oct 30 17:00:02.282649 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.282626 6202 cpu_manager.go:414] "containerMap: removing container" podUID="181bd0c6-3a62-4ad2-8244-9987e0ac0606" containerName="revoke-keys" Oct 30 17:00:02.282649 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.282650 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181bd0c6-3a62-4ad2-8244-9987e0ac0606" containerName="revoke-keys" Oct 30 17:00:02.282866 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.282696 6202 cpu_manager.go:414] "containerMap: removing container" podUID="927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" containerName="istio-proxy" Oct 30 17:00:02.282866 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.282701 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" containerName="istio-proxy" Oct 30 17:00:02.282968 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.282876 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" containerName="istio-proxy" Oct 30 17:00:02.282968 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.282891 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" containerName="istio-proxy" Oct 30 17:00:02.282968 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.282897 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="181bd0c6-3a62-4ad2-8244-9987e0ac0606" containerName="revoke-keys" Oct 30 17:00:02.287135 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.287112 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl" Oct 30 17:00:02.303682 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.303653 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:00:02.308799 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.308723 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl"] Oct 30 17:00:02.386204 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.386163 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f0ec459-kxwhl\" (UID: \"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl" Oct 30 17:00:02.386361 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.386315 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn9lc\" (UniqueName: \"kubernetes.io/projected/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-kube-api-access-hn9lc\") pod \"maas-api-revoke-sub-6f0ec459-kxwhl\" (UID: \"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl" Oct 30 17:00:02.386624 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.386600 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f0ec459-kxwhl\" (UID: \"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl" Oct 30 17:00:02.417048 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.417004 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" path="/var/lib/kubelet/pods/927cb01d-5133-4ecd-abb7-fa13f1fbe8c5/volumes" Oct 30 17:00:02.488066 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.488029 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f0ec459-kxwhl\" (UID: \"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl" Oct 30 17:00:02.488242 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.488140 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn9lc\" (UniqueName: \"kubernetes.io/projected/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-kube-api-access-hn9lc\") pod \"maas-api-revoke-sub-6f0ec459-kxwhl\" (UID: \"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl" Oct 30 17:00:02.488835 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.488810 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f0ec459-kxwhl\" (UID: \"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl" Oct 30 17:00:02.488951 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.488918 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f0ec459-kxwhl\" (UID: \"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl" Oct 30 17:00:02.491907 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.491869 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f0ec459-kxwhl\" (UID: \"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl" Oct 30 17:00:02.514532 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.514492 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn9lc\" (UniqueName: \"kubernetes.io/projected/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-kube-api-access-hn9lc\") pod \"maas-api-revoke-sub-6f0ec459-kxwhl\" (UID: \"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl" Oct 30 17:00:02.605370 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.605329 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl" Oct 30 17:00:02.809190 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.809147 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" event={"ID":"5ba09798-de28-4d76-8fea-839bbd8ea7e4","Type":"ContainerStarted","Data":"d4a20258d07150a34f3e4c858d6680caef504b06e78081f11c55cf62b2925332"} Oct 30 17:00:02.809699 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.809223 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" event={"ID":"5ba09798-de28-4d76-8fea-839bbd8ea7e4","Type":"ContainerStarted","Data":"94835a422ba61196d754951c9120cd9a54db07e5d428aab137c2206388f92325"} Oct 30 17:00:02.811695 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.811664 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" event={"ID":"6957d4c0-8197-4aab-a89a-ad17414cd011","Type":"ContainerDied","Data":"9b8556fd75623fc8ada6f051c0b9f8ca43601110ea19fdeb81cb50c0878817e9"} Oct 30 17:00:02.812165 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.812148 6202 scope.go:122] "RemoveContainer" containerID="9b8556fd75623fc8ada6f051c0b9f8ca43601110ea19fdeb81cb50c0878817e9" Oct 30 17:00:02.812532 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.811543 6202 generic.go:356] "Generic (PLEG): container finished" podID="6957d4c0-8197-4aab-a89a-ad17414cd011" containerID="9b8556fd75623fc8ada6f051c0b9f8ca43601110ea19fdeb81cb50c0878817e9" exitCode=60 Oct 30 17:00:02.815519 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.815494 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" event={"ID":"c17f9002-4b15-4e46-a720-7a298c9c3f68","Type":"ContainerStarted","Data":"2e8582d4955d41f39706e842a7428d1b1a9a62d3373476a5907f3307e7b97d87"} Oct 30 17:00:02.817842 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.817816 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" event={"ID":"ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9","Type":"ContainerStarted","Data":"77c95c7442109687546f15316c58e77e3bb154b76974785fe20d3ff46536a9e9"} Oct 30 17:00:02.818004 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.817979 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" event={"ID":"ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9","Type":"ContainerStarted","Data":"9ea9932c9100021fe5a7eafac7733e84c7e534e1e4c94c22af9a0d2ba91f364e"} Oct 30 17:00:02.819660 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.819637 6202 generic.go:356] "Generic (PLEG): container finished" podID="3d338cc2-8d69-4b14-99d1-370710f8cb99" containerID="357d3053de235095e30968b0ea65013ae6df2106a81728360b6d67f6933af928" exitCode=60 Oct 30 17:00:02.819831 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.819670 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" event={"ID":"3d338cc2-8d69-4b14-99d1-370710f8cb99","Type":"ContainerDied","Data":"357d3053de235095e30968b0ea65013ae6df2106a81728360b6d67f6933af928"} Oct 30 17:00:02.820070 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.820048 6202 scope.go:122] "RemoveContainer" containerID="357d3053de235095e30968b0ea65013ae6df2106a81728360b6d67f6933af928" Oct 30 17:00:02.821843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.821823 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" event={"ID":"0015d61c-9d93-4652-9414-3b817ab96276","Type":"ContainerStarted","Data":"64e6c91817cb6c6095717a21ca934d9be7453a9eca9ef6ef18620125f15ba947"} Oct 30 17:00:02.823277 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.823255 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" event={"ID":"c7550e3d-3bc1-4ff5-a641-0d68b4bf6127","Type":"ContainerStarted","Data":"4f15ddb65677933cc012f9cc903c878b495bf44ab900b890f48d4d72d70fdc73"} Oct 30 17:00:02.824825 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.824756 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" event={"ID":"c0b2402f-79e8-4586-9f36-2b86f307f1cb","Type":"ContainerStarted","Data":"152546631ab13644f4e99445bc0c90d4246d7df181b39e1289b41804ee236b14"} Oct 30 17:00:02.827384 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.827356 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" event={"ID":"bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5","Type":"ContainerStarted","Data":"a4a701f831c04185b540e4465eb181693e4f47581d4435eb3c422810a4b1a5cb"} Oct 30 17:00:02.829107 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.829076 6202 generic.go:356] "Generic (PLEG): container finished" podID="660c4258-30bc-4a2f-b130-766dcbbd88f4" containerID="55480398a2c1d96f41cb3d6c194bb84beb401e526d23618669e898662662a69a" exitCode=60 Oct 30 17:00:02.829206 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.829119 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" event={"ID":"660c4258-30bc-4a2f-b130-766dcbbd88f4","Type":"ContainerDied","Data":"55480398a2c1d96f41cb3d6c194bb84beb401e526d23618669e898662662a69a"} Oct 30 17:00:02.829447 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.829428 6202 scope.go:122] "RemoveContainer" containerID="55480398a2c1d96f41cb3d6c194bb84beb401e526d23618669e898662662a69a" Oct 30 17:00:02.830975 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.830954 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" event={"ID":"7aadca90-6dbe-4fe5-9dee-10431e9210a0","Type":"ContainerStarted","Data":"02130268047bafb47dab524b2d25b5267cf6b05ec6b8b5b617f3ec9229e51bc1"} Oct 30 17:00:02.831061 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.830996 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" event={"ID":"7aadca90-6dbe-4fe5-9dee-10431e9210a0","Type":"ContainerStarted","Data":"55fe104d78099de8d226ce1572bf55e0aec8169a9dffe29931206b866fcfe2f1"} Oct 30 17:00:02.832948 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.832926 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" event={"ID":"5c75b279-d01e-43e8-9a3e-8dff3dae29c7","Type":"ContainerStarted","Data":"ed29b314303a5fb6305f4bbdd68b36bb896aeafa43c346666d2736808beb907e"} Oct 30 17:00:02.834565 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.834543 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" event={"ID":"8c799e0b-668f-4c55-a254-69651dd5b005","Type":"ContainerStarted","Data":"4f401a56aa157f20e31cc67511c3cf96f02d022a4a08e89373971fce9dfb0a58"} Oct 30 17:00:02.836345 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.836318 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" event={"ID":"75af19e6-9fae-4786-86d2-3ef7f4433f08","Type":"ContainerStarted","Data":"600c4aeff11ce7372075c01a2cff5a881c3246a1c9ca24832748bd1afeb4ca33"} Oct 30 17:00:02.836446 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.836373 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" event={"ID":"75af19e6-9fae-4786-86d2-3ef7f4433f08","Type":"ContainerStarted","Data":"c746797433518a53eb208665d8f16c411d366b2177f5d17cd9a28ae383df1498"} Oct 30 17:00:02.872266 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.871289 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" podStartSLOduration=2.871268384 podStartE2EDuration="2.871268384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.848308441 +0000 UTC m=+2316.558763185" watchObservedRunningTime="2026-10-30 17:00:02.871268384 +0000 UTC m=+2316.581723134" Oct 30 17:00:02.904356 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.902944 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" podStartSLOduration=2.902923692 podStartE2EDuration="2.902923692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.899507414 +0000 UTC m=+2316.609962159" watchObservedRunningTime="2026-10-30 17:00:02.902923692 +0000 UTC m=+2316.613378450" Oct 30 17:00:02.947198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.946477 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" podStartSLOduration=2.946457049 podStartE2EDuration="2.946457049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.94452547 +0000 UTC m=+2316.654980215" watchObservedRunningTime="2026-10-30 17:00:02.946457049 +0000 UTC m=+2316.656911796" Oct 30 17:00:02.985879 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:02.985818 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" podStartSLOduration=2.9857844 podStartE2EDuration="2.9857844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.984358643 +0000 UTC m=+2316.694813387" watchObservedRunningTime="2026-10-30 17:00:02.9857844 +0000 UTC m=+2316.696239145" Oct 30 17:00:03.088810 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.088732 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" podStartSLOduration=3.088713405 podStartE2EDuration="3.088713405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:03.086251559 +0000 UTC m=+2316.796706304" watchObservedRunningTime="2026-10-30 17:00:03.088713405 +0000 UTC m=+2316.799168150" Oct 30 17:00:03.119946 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.119910 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl"] Oct 30 17:00:03.128292 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.128239 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" podStartSLOduration=3.1282233919999998 podStartE2EDuration="3.128223392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:03.126573183 +0000 UTC m=+2316.837027929" watchObservedRunningTime="2026-10-30 17:00:03.128223392 +0000 UTC m=+2316.838678127" Oct 30 17:00:03.586284 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.586245 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:03.817308 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.817281 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5"] Oct 30 17:00:03.848612 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.848562 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" event={"ID":"660c4258-30bc-4a2f-b130-766dcbbd88f4","Type":"ContainerStarted","Data":"abdd5c043c18619f8be1cb8b70e5e927882242ca6bd9190737984ba7a471ea8d"} Oct 30 17:00:03.850488 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.850461 6202 generic.go:356] "Generic (PLEG): container finished" podID="7aadca90-6dbe-4fe5-9dee-10431e9210a0" containerID="02130268047bafb47dab524b2d25b5267cf6b05ec6b8b5b617f3ec9229e51bc1" exitCode=60 Oct 30 17:00:03.850643 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.850549 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" event={"ID":"7aadca90-6dbe-4fe5-9dee-10431e9210a0","Type":"ContainerDied","Data":"02130268047bafb47dab524b2d25b5267cf6b05ec6b8b5b617f3ec9229e51bc1"} Oct 30 17:00:03.850901 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.850883 6202 scope.go:122] "RemoveContainer" containerID="02130268047bafb47dab524b2d25b5267cf6b05ec6b8b5b617f3ec9229e51bc1" Oct 30 17:00:03.852553 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.852524 6202 generic.go:356] "Generic (PLEG): container finished" podID="f9872bb2-dae7-43b4-9a03-6d0c9804cb79" containerID="e045cc150f79af9dde89740c42463a28f90a8f9c3b61082b27aa84f1010017ad" exitCode=60 Oct 30 17:00:03.852649 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.852605 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp" event={"ID":"f9872bb2-dae7-43b4-9a03-6d0c9804cb79","Type":"ContainerDied","Data":"e045cc150f79af9dde89740c42463a28f90a8f9c3b61082b27aa84f1010017ad"} Oct 30 17:00:03.852954 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.852925 6202 scope.go:122] "RemoveContainer" containerID="e045cc150f79af9dde89740c42463a28f90a8f9c3b61082b27aa84f1010017ad" Oct 30 17:00:03.855614 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.855587 6202 generic.go:356] "Generic (PLEG): container finished" podID="5c75b279-d01e-43e8-9a3e-8dff3dae29c7" containerID="9ee66921bc67e156541f3454a99f5e38d9b0a0319ac670be4436cadc5208e23d" exitCode=60 Oct 30 17:00:03.855723 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.855671 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" event={"ID":"5c75b279-d01e-43e8-9a3e-8dff3dae29c7","Type":"ContainerDied","Data":"9ee66921bc67e156541f3454a99f5e38d9b0a0319ac670be4436cadc5208e23d"} Oct 30 17:00:03.855972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.855941 6202 scope.go:122] "RemoveContainer" containerID="9ee66921bc67e156541f3454a99f5e38d9b0a0319ac670be4436cadc5208e23d" Oct 30 17:00:03.871346 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.871210 6202 generic.go:356] "Generic (PLEG): container finished" podID="8c799e0b-668f-4c55-a254-69651dd5b005" containerID="41ae8cc2af1a49d34d00c46f406725bf89074caebc041895d42da53a55ed9f9f" exitCode=60 Oct 30 17:00:03.873412 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.872710 6202 scope.go:122] "RemoveContainer" containerID="41ae8cc2af1a49d34d00c46f406725bf89074caebc041895d42da53a55ed9f9f" Oct 30 17:00:03.873412 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.873337 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" event={"ID":"8c799e0b-668f-4c55-a254-69651dd5b005","Type":"ContainerDied","Data":"41ae8cc2af1a49d34d00c46f406725bf89074caebc041895d42da53a55ed9f9f"} Oct 30 17:00:03.875804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.875753 6202 generic.go:356] "Generic (PLEG): container finished" podID="75af19e6-9fae-4786-86d2-3ef7f4433f08" containerID="600c4aeff11ce7372075c01a2cff5a881c3246a1c9ca24832748bd1afeb4ca33" exitCode=60 Oct 30 17:00:03.875912 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.875892 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" event={"ID":"75af19e6-9fae-4786-86d2-3ef7f4433f08","Type":"ContainerDied","Data":"600c4aeff11ce7372075c01a2cff5a881c3246a1c9ca24832748bd1afeb4ca33"} Oct 30 17:00:03.876348 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.876325 6202 scope.go:122] "RemoveContainer" containerID="600c4aeff11ce7372075c01a2cff5a881c3246a1c9ca24832748bd1afeb4ca33" Oct 30 17:00:03.898735 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.898700 6202 generic.go:356] "Generic (PLEG): container finished" podID="b09a458f-a69b-441a-b91a-11854c180a4e" containerID="84cfa8e617668f761d091ea2b830f9f9a58074a9ea045ede540c464800010cfe" exitCode=60 Oct 30 17:00:03.900316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.900289 6202 scope.go:122] "RemoveContainer" containerID="84cfa8e617668f761d091ea2b830f9f9a58074a9ea045ede540c464800010cfe" Oct 30 17:00:03.900501 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.900480 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" event={"ID":"b09a458f-a69b-441a-b91a-11854c180a4e","Type":"ContainerDied","Data":"84cfa8e617668f761d091ea2b830f9f9a58074a9ea045ede540c464800010cfe"} Oct 30 17:00:03.917105 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.915666 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl" event={"ID":"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9","Type":"ContainerStarted","Data":"c21297396157778082fbaddbf2d17a469ca355957467b01721e8750c59c9940f"} Oct 30 17:00:03.930611 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.930126 6202 generic.go:356] "Generic (PLEG): container finished" podID="5ba09798-de28-4d76-8fea-839bbd8ea7e4" containerID="d4a20258d07150a34f3e4c858d6680caef504b06e78081f11c55cf62b2925332" exitCode=60 Oct 30 17:00:03.930611 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.930300 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" event={"ID":"5ba09798-de28-4d76-8fea-839bbd8ea7e4","Type":"ContainerDied","Data":"d4a20258d07150a34f3e4c858d6680caef504b06e78081f11c55cf62b2925332"} Oct 30 17:00:03.930885 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.930840 6202 scope.go:122] "RemoveContainer" containerID="d4a20258d07150a34f3e4c858d6680caef504b06e78081f11c55cf62b2925332" Oct 30 17:00:03.940258 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.940216 6202 generic.go:356] "Generic (PLEG): container finished" podID="c17f9002-4b15-4e46-a720-7a298c9c3f68" containerID="2e8582d4955d41f39706e842a7428d1b1a9a62d3373476a5907f3307e7b97d87" exitCode=60 Oct 30 17:00:03.940413 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.940323 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" event={"ID":"c17f9002-4b15-4e46-a720-7a298c9c3f68","Type":"ContainerDied","Data":"2e8582d4955d41f39706e842a7428d1b1a9a62d3373476a5907f3307e7b97d87"} Oct 30 17:00:03.940872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.940847 6202 scope.go:122] "RemoveContainer" containerID="2e8582d4955d41f39706e842a7428d1b1a9a62d3373476a5907f3307e7b97d87" Oct 30 17:00:03.943845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.943817 6202 generic.go:356] "Generic (PLEG): container finished" podID="ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9" containerID="77c95c7442109687546f15316c58e77e3bb154b76974785fe20d3ff46536a9e9" exitCode=60 Oct 30 17:00:03.944076 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.944013 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" event={"ID":"ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9","Type":"ContainerDied","Data":"77c95c7442109687546f15316c58e77e3bb154b76974785fe20d3ff46536a9e9"} Oct 30 17:00:03.944624 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.944507 6202 scope.go:122] "RemoveContainer" containerID="77c95c7442109687546f15316c58e77e3bb154b76974785fe20d3ff46536a9e9" Oct 30 17:00:03.947691 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.947669 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:03.980017 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.979989 6202 generic.go:356] "Generic (PLEG): container finished" podID="0015d61c-9d93-4652-9414-3b817ab96276" containerID="64e6c91817cb6c6095717a21ca934d9be7453a9eca9ef6ef18620125f15ba947" exitCode=60 Oct 30 17:00:03.980884 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.980839 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" event={"ID":"0015d61c-9d93-4652-9414-3b817ab96276","Type":"ContainerDied","Data":"64e6c91817cb6c6095717a21ca934d9be7453a9eca9ef6ef18620125f15ba947"} Oct 30 17:00:03.981757 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.981732 6202 scope.go:122] "RemoveContainer" containerID="64e6c91817cb6c6095717a21ca934d9be7453a9eca9ef6ef18620125f15ba947" Oct 30 17:00:03.984506 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.984480 6202 generic.go:356] "Generic (PLEG): container finished" podID="c7550e3d-3bc1-4ff5-a641-0d68b4bf6127" containerID="6f437e27c06f1a51b4ea24d766f6d4f6a7e22ee4b76a11d6ad945d09c59de929" exitCode=60 Oct 30 17:00:03.984712 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.984686 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" event={"ID":"c7550e3d-3bc1-4ff5-a641-0d68b4bf6127","Type":"ContainerDied","Data":"6f437e27c06f1a51b4ea24d766f6d4f6a7e22ee4b76a11d6ad945d09c59de929"} Oct 30 17:00:03.985337 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.985303 6202 scope.go:122] "RemoveContainer" containerID="6f437e27c06f1a51b4ea24d766f6d4f6a7e22ee4b76a11d6ad945d09c59de929" Oct 30 17:00:03.991585 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.991560 6202 generic.go:356] "Generic (PLEG): container finished" podID="c0b2402f-79e8-4586-9f36-2b86f307f1cb" containerID="74dcfcfb9de4d947a3838c599fb9a730a26382ecd0c6222fbe894804043956fe" exitCode=60 Oct 30 17:00:03.991851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.991828 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" event={"ID":"c0b2402f-79e8-4586-9f36-2b86f307f1cb","Type":"ContainerDied","Data":"74dcfcfb9de4d947a3838c599fb9a730a26382ecd0c6222fbe894804043956fe"} Oct 30 17:00:03.992124 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.992106 6202 scope.go:122] "RemoveContainer" containerID="74dcfcfb9de4d947a3838c599fb9a730a26382ecd0c6222fbe894804043956fe" Oct 30 17:00:03.993832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.993758 6202 generic.go:356] "Generic (PLEG): container finished" podID="bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5" containerID="92a2d78b38953e80f839f7a5ad0a2bb6178600058510da010814cbcc1d558d1b" exitCode=60 Oct 30 17:00:03.993935 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.993918 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" event={"ID":"bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5","Type":"ContainerDied","Data":"92a2d78b38953e80f839f7a5ad0a2bb6178600058510da010814cbcc1d558d1b"} Oct 30 17:00:03.994083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:03.994069 6202 scope.go:122] "RemoveContainer" containerID="92a2d78b38953e80f839f7a5ad0a2bb6178600058510da010814cbcc1d558d1b" Oct 30 17:00:04.019745 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:04.019691 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" podStartSLOduration=4.01967062 podStartE2EDuration="4.01967062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:04.013226569 +0000 UTC m=+2317.723681317" watchObservedRunningTime="2026-10-30 17:00:04.01967062 +0000 UTC m=+2317.730125365" Oct 30 17:00:04.091503 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:04.091415 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:05.004926 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.004815 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" event={"ID":"c0b2402f-79e8-4586-9f36-2b86f307f1cb","Type":"ContainerStarted","Data":"24bf9e58d54ea4ed0abf0b7e57ef5beb8951ebb54ad07413972d8314be4acfc9"} Oct 30 17:00:05.007010 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.006978 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" event={"ID":"bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5","Type":"ContainerStarted","Data":"4fbfc7d452355aee1a469bade80d249260ef0961687867612677eb4750b5584f"} Oct 30 17:00:05.009144 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.009121 6202 generic.go:356] "Generic (PLEG): container finished" podID="660c4258-30bc-4a2f-b130-766dcbbd88f4" containerID="abdd5c043c18619f8be1cb8b70e5e927882242ca6bd9190737984ba7a471ea8d" exitCode=60 Oct 30 17:00:05.009256 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.009205 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" event={"ID":"660c4258-30bc-4a2f-b130-766dcbbd88f4","Type":"ContainerDied","Data":"abdd5c043c18619f8be1cb8b70e5e927882242ca6bd9190737984ba7a471ea8d"} Oct 30 17:00:05.009319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.009268 6202 scope.go:122] "RemoveContainer" containerID="55480398a2c1d96f41cb3d6c194bb84beb401e526d23618669e898662662a69a" Oct 30 17:00:05.009554 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.009517 6202 scope.go:122] "RemoveContainer" containerID="abdd5c043c18619f8be1cb8b70e5e927882242ca6bd9190737984ba7a471ea8d" Oct 30 17:00:05.009981 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:05.009951 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4_odh-ai-gateway-infra(660c4258-30bc-4a2f-b130-766dcbbd88f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" podUID="660c4258-30bc-4a2f-b130-766dcbbd88f4" Oct 30 17:00:05.011476 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.011447 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" event={"ID":"7aadca90-6dbe-4fe5-9dee-10431e9210a0","Type":"ContainerStarted","Data":"4c3473df9be92d62380d4ed6bf05c7aa17c78370916ae619ed19e2b4f34ddc83"} Oct 30 17:00:05.013498 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.013474 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" event={"ID":"b09a458f-a69b-441a-b91a-11854c180a4e","Type":"ContainerStarted","Data":"d3f58a5a36aed980ceb99e50148481fc32db504eac8cb7feb4cdfe371177486c"} Oct 30 17:00:05.015217 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.015190 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl" event={"ID":"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9","Type":"ContainerStarted","Data":"c6c85e7b4f2fe1aa5ebbb97c071987c8f1fcf96f18bb6d92a1d3b223c64268b1"} Oct 30 17:00:05.017187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.017163 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" event={"ID":"5ba09798-de28-4d76-8fea-839bbd8ea7e4","Type":"ContainerStarted","Data":"9f0340876d419cacb0be50e95574582fdcee83cc6ce5786efc7ad18984a7832e"} Oct 30 17:00:05.031271 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.031231 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" event={"ID":"6957d4c0-8197-4aab-a89a-ad17414cd011","Type":"ContainerStarted","Data":"4eeee6b375a206c0296015b431aa464978d137a35e4bf7e2b707492aa9da3405"} Oct 30 17:00:05.058840 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.053040 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" event={"ID":"ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9","Type":"ContainerStarted","Data":"7fd04118ae9210a0885d8015555ed0263dd03a04c0b1bfa4abda639b0ffb6a03"} Oct 30 17:00:05.059013 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.058869 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" event={"ID":"3d338cc2-8d69-4b14-99d1-370710f8cb99","Type":"ContainerStarted","Data":"ea8884b55cebe30dc596e0566583a46ef5526e56107231126a175854bb0f661e"} Oct 30 17:00:05.063346 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.063287 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" podStartSLOduration=5.063269002 podStartE2EDuration="5.063269002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:05.061940403 +0000 UTC m=+2318.772395149" watchObservedRunningTime="2026-10-30 17:00:05.063269002 +0000 UTC m=+2318.773723781" Oct 30 17:00:05.156061 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.155631 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" podStartSLOduration=5.155600579 podStartE2EDuration="5.155600579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:05.155521329 +0000 UTC m=+2318.865976074" watchObservedRunningTime="2026-10-30 17:00:05.155600579 +0000 UTC m=+2318.866055347" Oct 30 17:00:05.253478 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.253400 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" podStartSLOduration=5.253385149 podStartE2EDuration="5.253385149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:05.244585837 +0000 UTC m=+2318.955040582" watchObservedRunningTime="2026-10-30 17:00:05.253385149 +0000 UTC m=+2318.963839892" Oct 30 17:00:05.285987 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.285879 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl" podStartSLOduration=3.285859676 podStartE2EDuration="3.285859676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:05.284177805 +0000 UTC m=+2318.994632552" watchObservedRunningTime="2026-10-30 17:00:05.285859676 +0000 UTC m=+2318.996314421" Oct 30 17:00:05.332854 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.332790 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" podStartSLOduration=5.332752209 podStartE2EDuration="5.332752209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:05.328007046 +0000 UTC m=+2319.038461793" watchObservedRunningTime="2026-10-30 17:00:05.332752209 +0000 UTC m=+2319.043206954" Oct 30 17:00:05.467841 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.467718 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" podStartSLOduration=5.467694502 podStartE2EDuration="5.467694502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:05.465038592 +0000 UTC m=+2319.175493339" watchObservedRunningTime="2026-10-30 17:00:05.467694502 +0000 UTC m=+2319.178149247" Oct 30 17:00:05.473375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:05.473061 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:06.071118 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.071082 6202 generic.go:356] "Generic (PLEG): container finished" podID="5ba09798-de28-4d76-8fea-839bbd8ea7e4" containerID="9f0340876d419cacb0be50e95574582fdcee83cc6ce5786efc7ad18984a7832e" exitCode=60 Oct 30 17:00:06.071575 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.071166 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" event={"ID":"5ba09798-de28-4d76-8fea-839bbd8ea7e4","Type":"ContainerDied","Data":"9f0340876d419cacb0be50e95574582fdcee83cc6ce5786efc7ad18984a7832e"} Oct 30 17:00:06.071575 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.071226 6202 scope.go:122] "RemoveContainer" containerID="d4a20258d07150a34f3e4c858d6680caef504b06e78081f11c55cf62b2925332" Oct 30 17:00:06.071575 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.071527 6202 scope.go:122] "RemoveContainer" containerID="9f0340876d419cacb0be50e95574582fdcee83cc6ce5786efc7ad18984a7832e" Oct 30 17:00:06.072015 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:06.071988 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889660-l5wqc_odh-ai-gateway-infra(5ba09798-de28-4d76-8fea-839bbd8ea7e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" podUID="5ba09798-de28-4d76-8fea-839bbd8ea7e4" Oct 30 17:00:06.073677 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.073382 6202 generic.go:356] "Generic (PLEG): container finished" podID="6957d4c0-8197-4aab-a89a-ad17414cd011" containerID="4eeee6b375a206c0296015b431aa464978d137a35e4bf7e2b707492aa9da3405" exitCode=60 Oct 30 17:00:06.073677 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.073460 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" event={"ID":"6957d4c0-8197-4aab-a89a-ad17414cd011","Type":"ContainerDied","Data":"4eeee6b375a206c0296015b431aa464978d137a35e4bf7e2b707492aa9da3405"} Oct 30 17:00:06.073837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.073825 6202 scope.go:122] "RemoveContainer" containerID="4eeee6b375a206c0296015b431aa464978d137a35e4bf7e2b707492aa9da3405" Oct 30 17:00:06.074360 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:06.074327 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj_odh-ai-gateway-infra(6957d4c0-8197-4aab-a89a-ad17414cd011)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" podUID="6957d4c0-8197-4aab-a89a-ad17414cd011" Oct 30 17:00:06.075855 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.075831 6202 generic.go:356] "Generic (PLEG): container finished" podID="c17f9002-4b15-4e46-a720-7a298c9c3f68" containerID="3a5cb5cb4c040e61566da3d768d17d6fa862ec4badc18d0cdcb91fa1b2f8e1ef" exitCode=60 Oct 30 17:00:06.075956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.075914 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" event={"ID":"c17f9002-4b15-4e46-a720-7a298c9c3f68","Type":"ContainerDied","Data":"3a5cb5cb4c040e61566da3d768d17d6fa862ec4badc18d0cdcb91fa1b2f8e1ef"} Oct 30 17:00:06.076234 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.076219 6202 scope.go:122] "RemoveContainer" containerID="3a5cb5cb4c040e61566da3d768d17d6fa862ec4badc18d0cdcb91fa1b2f8e1ef" Oct 30 17:00:06.076634 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:06.076610 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k_odh-ai-gateway-infra(c17f9002-4b15-4e46-a720-7a298c9c3f68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" podUID="c17f9002-4b15-4e46-a720-7a298c9c3f68" Oct 30 17:00:06.077975 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.077954 6202 generic.go:356] "Generic (PLEG): container finished" podID="ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9" containerID="7fd04118ae9210a0885d8015555ed0263dd03a04c0b1bfa4abda639b0ffb6a03" exitCode=60 Oct 30 17:00:06.078081 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.078032 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" event={"ID":"ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9","Type":"ContainerDied","Data":"7fd04118ae9210a0885d8015555ed0263dd03a04c0b1bfa4abda639b0ffb6a03"} Oct 30 17:00:06.078328 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.078311 6202 scope.go:122] "RemoveContainer" containerID="7fd04118ae9210a0885d8015555ed0263dd03a04c0b1bfa4abda639b0ffb6a03" Oct 30 17:00:06.078679 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:06.078652 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2_odh-ai-gateway-infra(ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" podUID="ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9" Oct 30 17:00:06.080007 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.079985 6202 generic.go:356] "Generic (PLEG): container finished" podID="3d338cc2-8d69-4b14-99d1-370710f8cb99" containerID="ea8884b55cebe30dc596e0566583a46ef5526e56107231126a175854bb0f661e" exitCode=60 Oct 30 17:00:06.080120 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.080060 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" event={"ID":"3d338cc2-8d69-4b14-99d1-370710f8cb99","Type":"ContainerDied","Data":"ea8884b55cebe30dc596e0566583a46ef5526e56107231126a175854bb0f661e"} Oct 30 17:00:06.080358 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.080341 6202 scope.go:122] "RemoveContainer" containerID="ea8884b55cebe30dc596e0566583a46ef5526e56107231126a175854bb0f661e" Oct 30 17:00:06.080676 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:06.080651 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47_odh-ai-gateway-infra(3d338cc2-8d69-4b14-99d1-370710f8cb99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" podUID="3d338cc2-8d69-4b14-99d1-370710f8cb99" Oct 30 17:00:06.081913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.081891 6202 generic.go:356] "Generic (PLEG): container finished" podID="0015d61c-9d93-4652-9414-3b817ab96276" containerID="aa1d0e08671907f48810b40b89681ea8acd0a39f885bf5bc3a3a1eae1c188d50" exitCode=60 Oct 30 17:00:06.082007 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.081959 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" event={"ID":"0015d61c-9d93-4652-9414-3b817ab96276","Type":"ContainerDied","Data":"aa1d0e08671907f48810b40b89681ea8acd0a39f885bf5bc3a3a1eae1c188d50"} Oct 30 17:00:06.082203 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.082188 6202 scope.go:122] "RemoveContainer" containerID="aa1d0e08671907f48810b40b89681ea8acd0a39f885bf5bc3a3a1eae1c188d50" Oct 30 17:00:06.082541 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:06.082521 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm_odh-ai-gateway-infra(0015d61c-9d93-4652-9414-3b817ab96276)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" podUID="0015d61c-9d93-4652-9414-3b817ab96276" Oct 30 17:00:06.083711 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.083692 6202 generic.go:356] "Generic (PLEG): container finished" podID="c7550e3d-3bc1-4ff5-a641-0d68b4bf6127" containerID="b9cdeb971897919dc7da636d827d41a76c3dd08b4e1cb7ac266874c7b1a8f251" exitCode=60 Oct 30 17:00:06.083809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.083757 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" event={"ID":"c7550e3d-3bc1-4ff5-a641-0d68b4bf6127","Type":"ContainerDied","Data":"b9cdeb971897919dc7da636d827d41a76c3dd08b4e1cb7ac266874c7b1a8f251"} Oct 30 17:00:06.084086 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.084067 6202 scope.go:122] "RemoveContainer" containerID="b9cdeb971897919dc7da636d827d41a76c3dd08b4e1cb7ac266874c7b1a8f251" Oct 30 17:00:06.084428 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:06.084408 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr_odh-ai-gateway-infra(c7550e3d-3bc1-4ff5-a641-0d68b4bf6127)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" podUID="c7550e3d-3bc1-4ff5-a641-0d68b4bf6127" Oct 30 17:00:06.086540 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.085847 6202 generic.go:356] "Generic (PLEG): container finished" podID="c0b2402f-79e8-4586-9f36-2b86f307f1cb" containerID="24bf9e58d54ea4ed0abf0b7e57ef5beb8951ebb54ad07413972d8314be4acfc9" exitCode=60 Oct 30 17:00:06.086540 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.085923 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" event={"ID":"c0b2402f-79e8-4586-9f36-2b86f307f1cb","Type":"ContainerDied","Data":"24bf9e58d54ea4ed0abf0b7e57ef5beb8951ebb54ad07413972d8314be4acfc9"} Oct 30 17:00:06.086540 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.086160 6202 scope.go:122] "RemoveContainer" containerID="24bf9e58d54ea4ed0abf0b7e57ef5beb8951ebb54ad07413972d8314be4acfc9" Oct 30 17:00:06.086540 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:06.086470 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t_odh-ai-gateway-infra(c0b2402f-79e8-4586-9f36-2b86f307f1cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" podUID="c0b2402f-79e8-4586-9f36-2b86f307f1cb" Oct 30 17:00:06.088192 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.088161 6202 generic.go:356] "Generic (PLEG): container finished" podID="bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5" containerID="4fbfc7d452355aee1a469bade80d249260ef0961687867612677eb4750b5584f" exitCode=60 Oct 30 17:00:06.088277 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.088244 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" event={"ID":"bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5","Type":"ContainerDied","Data":"4fbfc7d452355aee1a469bade80d249260ef0961687867612677eb4750b5584f"} Oct 30 17:00:06.088494 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.088466 6202 scope.go:122] "RemoveContainer" containerID="4fbfc7d452355aee1a469bade80d249260ef0961687867612677eb4750b5584f" Oct 30 17:00:06.088821 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:06.088797 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz_odh-ai-gateway-infra(bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" podUID="bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5" Oct 30 17:00:06.090346 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.090327 6202 scope.go:122] "RemoveContainer" containerID="abdd5c043c18619f8be1cb8b70e5e927882242ca6bd9190737984ba7a471ea8d" Oct 30 17:00:06.090594 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:06.090576 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4_odh-ai-gateway-infra(660c4258-30bc-4a2f-b130-766dcbbd88f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" podUID="660c4258-30bc-4a2f-b130-766dcbbd88f4" Oct 30 17:00:06.092048 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.092030 6202 generic.go:356] "Generic (PLEG): container finished" podID="7aadca90-6dbe-4fe5-9dee-10431e9210a0" containerID="4c3473df9be92d62380d4ed6bf05c7aa17c78370916ae619ed19e2b4f34ddc83" exitCode=60 Oct 30 17:00:06.092221 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.092203 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" event={"ID":"7aadca90-6dbe-4fe5-9dee-10431e9210a0","Type":"ContainerDied","Data":"4c3473df9be92d62380d4ed6bf05c7aa17c78370916ae619ed19e2b4f34ddc83"} Oct 30 17:00:06.092372 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.092316 6202 scope.go:122] "RemoveContainer" containerID="4c3473df9be92d62380d4ed6bf05c7aa17c78370916ae619ed19e2b4f34ddc83" Oct 30 17:00:06.092585 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:06.092564 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v_odh-ai-gateway-infra(7aadca90-6dbe-4fe5-9dee-10431e9210a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" podUID="7aadca90-6dbe-4fe5-9dee-10431e9210a0" Oct 30 17:00:06.094130 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.094108 6202 generic.go:356] "Generic (PLEG): container finished" podID="f9872bb2-dae7-43b4-9a03-6d0c9804cb79" containerID="1b7cac64d71077169b81fc41df654c1870c84e6335d484a1078041354ff0c28b" exitCode=60 Oct 30 17:00:06.094237 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.094187 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp" event={"ID":"f9872bb2-dae7-43b4-9a03-6d0c9804cb79","Type":"ContainerDied","Data":"1b7cac64d71077169b81fc41df654c1870c84e6335d484a1078041354ff0c28b"} Oct 30 17:00:06.094422 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.094405 6202 scope.go:122] "RemoveContainer" containerID="1b7cac64d71077169b81fc41df654c1870c84e6335d484a1078041354ff0c28b" Oct 30 17:00:06.094677 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:06.094655 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp_odh-ai-gateway-infra(f9872bb2-dae7-43b4-9a03-6d0c9804cb79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp" podUID="f9872bb2-dae7-43b4-9a03-6d0c9804cb79" Oct 30 17:00:06.096168 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.096150 6202 generic.go:356] "Generic (PLEG): container finished" podID="5c75b279-d01e-43e8-9a3e-8dff3dae29c7" containerID="f57ca7fd3cfc4571285d62a55f62b6e3c02e53dd4ec17b1985449f2642623d6b" exitCode=60 Oct 30 17:00:06.096272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.096218 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" event={"ID":"5c75b279-d01e-43e8-9a3e-8dff3dae29c7","Type":"ContainerDied","Data":"f57ca7fd3cfc4571285d62a55f62b6e3c02e53dd4ec17b1985449f2642623d6b"} Oct 30 17:00:06.096455 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.096436 6202 scope.go:122] "RemoveContainer" containerID="f57ca7fd3cfc4571285d62a55f62b6e3c02e53dd4ec17b1985449f2642623d6b" Oct 30 17:00:06.096701 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:06.096680 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh_odh-ai-gateway-infra(5c75b279-d01e-43e8-9a3e-8dff3dae29c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" podUID="5c75b279-d01e-43e8-9a3e-8dff3dae29c7" Oct 30 17:00:06.098251 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.098232 6202 generic.go:356] "Generic (PLEG): container finished" podID="8c799e0b-668f-4c55-a254-69651dd5b005" containerID="3d415d4226ca895027a5d5ce28b1d948cd9e6d077220d537e269938018a23e65" exitCode=60 Oct 30 17:00:06.098352 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.098306 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" event={"ID":"8c799e0b-668f-4c55-a254-69651dd5b005","Type":"ContainerDied","Data":"3d415d4226ca895027a5d5ce28b1d948cd9e6d077220d537e269938018a23e65"} Oct 30 17:00:06.098572 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.098556 6202 scope.go:122] "RemoveContainer" containerID="3d415d4226ca895027a5d5ce28b1d948cd9e6d077220d537e269938018a23e65" Oct 30 17:00:06.098900 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:06.098875 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r_odh-ai-gateway-infra(8c799e0b-668f-4c55-a254-69651dd5b005)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" podUID="8c799e0b-668f-4c55-a254-69651dd5b005" Oct 30 17:00:06.100403 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.100380 6202 generic.go:356] "Generic (PLEG): container finished" podID="75af19e6-9fae-4786-86d2-3ef7f4433f08" containerID="5494b76687547e1856d8261910b0a37ef5c68020dc1bf7ff5f0f4d0a8bc40fc4" exitCode=60 Oct 30 17:00:06.100501 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.100452 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" event={"ID":"75af19e6-9fae-4786-86d2-3ef7f4433f08","Type":"ContainerDied","Data":"5494b76687547e1856d8261910b0a37ef5c68020dc1bf7ff5f0f4d0a8bc40fc4"} Oct 30 17:00:06.100729 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.100712 6202 scope.go:122] "RemoveContainer" containerID="5494b76687547e1856d8261910b0a37ef5c68020dc1bf7ff5f0f4d0a8bc40fc4" Oct 30 17:00:06.101011 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:06.100987 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh_odh-ai-gateway-infra(75af19e6-9fae-4786-86d2-3ef7f4433f08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" podUID="75af19e6-9fae-4786-86d2-3ef7f4433f08" Oct 30 17:00:06.102816 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.102759 6202 generic.go:356] "Generic (PLEG): container finished" podID="b09a458f-a69b-441a-b91a-11854c180a4e" containerID="d3f58a5a36aed980ceb99e50148481fc32db504eac8cb7feb4cdfe371177486c" exitCode=60 Oct 30 17:00:06.102922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.102884 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" event={"ID":"b09a458f-a69b-441a-b91a-11854c180a4e","Type":"ContainerDied","Data":"d3f58a5a36aed980ceb99e50148481fc32db504eac8cb7feb4cdfe371177486c"} Oct 30 17:00:06.103469 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.103450 6202 scope.go:122] "RemoveContainer" containerID="d3f58a5a36aed980ceb99e50148481fc32db504eac8cb7feb4cdfe371177486c" Oct 30 17:00:06.103940 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:06.103741 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n_odh-ai-gateway-infra(b09a458f-a69b-441a-b91a-11854c180a4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" podUID="b09a458f-a69b-441a-b91a-11854c180a4e" Oct 30 17:00:06.107933 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.107907 6202 generic.go:356] "Generic (PLEG): container finished" podID="abac9b81-c250-4d4d-b1ed-f0615a7f2dd9" containerID="c6c85e7b4f2fe1aa5ebbb97c071987c8f1fcf96f18bb6d92a1d3b223c64268b1" exitCode=0 Oct 30 17:00:06.108037 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.107952 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl" event={"ID":"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9","Type":"ContainerDied","Data":"c6c85e7b4f2fe1aa5ebbb97c071987c8f1fcf96f18bb6d92a1d3b223c64268b1"} Oct 30 17:00:06.206595 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.206565 6202 scope.go:122] "RemoveContainer" containerID="9b8556fd75623fc8ada6f051c0b9f8ca43601110ea19fdeb81cb50c0878817e9" Oct 30 17:00:06.224687 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.224660 6202 scope.go:122] "RemoveContainer" containerID="2e8582d4955d41f39706e842a7428d1b1a9a62d3373476a5907f3307e7b97d87" Oct 30 17:00:06.242561 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.242538 6202 scope.go:122] "RemoveContainer" containerID="77c95c7442109687546f15316c58e77e3bb154b76974785fe20d3ff46536a9e9" Oct 30 17:00:06.260761 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.260738 6202 scope.go:122] "RemoveContainer" containerID="357d3053de235095e30968b0ea65013ae6df2106a81728360b6d67f6933af928" Oct 30 17:00:06.277984 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.277962 6202 scope.go:122] "RemoveContainer" containerID="64e6c91817cb6c6095717a21ca934d9be7453a9eca9ef6ef18620125f15ba947" Oct 30 17:00:06.297969 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.297944 6202 scope.go:122] "RemoveContainer" containerID="6f437e27c06f1a51b4ea24d766f6d4f6a7e22ee4b76a11d6ad945d09c59de929" Oct 30 17:00:06.313984 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.313947 6202 scope.go:122] "RemoveContainer" containerID="74dcfcfb9de4d947a3838c599fb9a730a26382ecd0c6222fbe894804043956fe" Oct 30 17:00:06.332007 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.331977 6202 scope.go:122] "RemoveContainer" containerID="92a2d78b38953e80f839f7a5ad0a2bb6178600058510da010814cbcc1d558d1b" Oct 30 17:00:06.353217 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.353195 6202 scope.go:122] "RemoveContainer" containerID="02130268047bafb47dab524b2d25b5267cf6b05ec6b8b5b617f3ec9229e51bc1" Oct 30 17:00:06.373502 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.373478 6202 scope.go:122] "RemoveContainer" containerID="e045cc150f79af9dde89740c42463a28f90a8f9c3b61082b27aa84f1010017ad" Oct 30 17:00:06.389010 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.388979 6202 scope.go:122] "RemoveContainer" containerID="9ee66921bc67e156541f3454a99f5e38d9b0a0319ac670be4436cadc5208e23d" Oct 30 17:00:06.405352 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.405329 6202 scope.go:122] "RemoveContainer" containerID="41ae8cc2af1a49d34d00c46f406725bf89074caebc041895d42da53a55ed9f9f" Oct 30 17:00:06.423420 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.423398 6202 scope.go:122] "RemoveContainer" containerID="600c4aeff11ce7372075c01a2cff5a881c3246a1c9ca24832748bd1afeb4ca33" Oct 30 17:00:06.439603 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:06.439580 6202 scope.go:122] "RemoveContainer" containerID="84cfa8e617668f761d091ea2b830f9f9a58074a9ea045ede540c464800010cfe" Oct 30 17:00:07.019046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.019003 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq"] Oct 30 17:00:07.051127 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.051092 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:07.133333 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.133305 6202 scope.go:122] "RemoveContainer" containerID="1b7cac64d71077169b81fc41df654c1870c84e6335d484a1078041354ff0c28b" Oct 30 17:00:07.134403 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:07.134307 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp_odh-ai-gateway-infra(f9872bb2-dae7-43b4-9a03-6d0c9804cb79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp" podUID="f9872bb2-dae7-43b4-9a03-6d0c9804cb79" Oct 30 17:00:07.138003 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.137959 6202 scope.go:122] "RemoveContainer" containerID="f57ca7fd3cfc4571285d62a55f62b6e3c02e53dd4ec17b1985449f2642623d6b" Oct 30 17:00:07.138909 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:07.138873 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh_odh-ai-gateway-infra(5c75b279-d01e-43e8-9a3e-8dff3dae29c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" podUID="5c75b279-d01e-43e8-9a3e-8dff3dae29c7" Oct 30 17:00:07.144391 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.143832 6202 scope.go:122] "RemoveContainer" containerID="3d415d4226ca895027a5d5ce28b1d948cd9e6d077220d537e269938018a23e65" Oct 30 17:00:07.144391 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:07.144329 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r_odh-ai-gateway-infra(8c799e0b-668f-4c55-a254-69651dd5b005)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" podUID="8c799e0b-668f-4c55-a254-69651dd5b005" Oct 30 17:00:07.147173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.147146 6202 scope.go:122] "RemoveContainer" containerID="5494b76687547e1856d8261910b0a37ef5c68020dc1bf7ff5f0f4d0a8bc40fc4" Oct 30 17:00:07.147844 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:07.147815 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh_odh-ai-gateway-infra(75af19e6-9fae-4786-86d2-3ef7f4433f08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" podUID="75af19e6-9fae-4786-86d2-3ef7f4433f08" Oct 30 17:00:07.153909 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.153868 6202 scope.go:122] "RemoveContainer" containerID="d3f58a5a36aed980ceb99e50148481fc32db504eac8cb7feb4cdfe371177486c" Oct 30 17:00:07.154357 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:07.154319 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n_odh-ai-gateway-infra(b09a458f-a69b-441a-b91a-11854c180a4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" podUID="b09a458f-a69b-441a-b91a-11854c180a4e" Oct 30 17:00:07.161719 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.161685 6202 scope.go:122] "RemoveContainer" containerID="9f0340876d419cacb0be50e95574582fdcee83cc6ce5786efc7ad18984a7832e" Oct 30 17:00:07.162263 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:07.162212 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889660-l5wqc_odh-ai-gateway-infra(5ba09798-de28-4d76-8fea-839bbd8ea7e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" podUID="5ba09798-de28-4d76-8fea-839bbd8ea7e4" Oct 30 17:00:07.164302 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.164276 6202 scope.go:122] "RemoveContainer" containerID="4eeee6b375a206c0296015b431aa464978d137a35e4bf7e2b707492aa9da3405" Oct 30 17:00:07.164846 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:07.164821 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj_odh-ai-gateway-infra(6957d4c0-8197-4aab-a89a-ad17414cd011)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" podUID="6957d4c0-8197-4aab-a89a-ad17414cd011" Oct 30 17:00:07.165962 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.165936 6202 scope.go:122] "RemoveContainer" containerID="3a5cb5cb4c040e61566da3d768d17d6fa862ec4badc18d0cdcb91fa1b2f8e1ef" Oct 30 17:00:07.166286 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:07.166258 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k_odh-ai-gateway-infra(c17f9002-4b15-4e46-a720-7a298c9c3f68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" podUID="c17f9002-4b15-4e46-a720-7a298c9c3f68" Oct 30 17:00:07.169287 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.168871 6202 scope.go:122] "RemoveContainer" containerID="7fd04118ae9210a0885d8015555ed0263dd03a04c0b1bfa4abda639b0ffb6a03" Oct 30 17:00:07.169287 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:07.169234 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2_odh-ai-gateway-infra(ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" podUID="ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9" Oct 30 17:00:07.173151 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.173117 6202 scope.go:122] "RemoveContainer" containerID="ea8884b55cebe30dc596e0566583a46ef5526e56107231126a175854bb0f661e" Oct 30 17:00:07.173570 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:07.173543 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47_odh-ai-gateway-infra(3d338cc2-8d69-4b14-99d1-370710f8cb99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" podUID="3d338cc2-8d69-4b14-99d1-370710f8cb99" Oct 30 17:00:07.176585 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.176556 6202 scope.go:122] "RemoveContainer" containerID="aa1d0e08671907f48810b40b89681ea8acd0a39f885bf5bc3a3a1eae1c188d50" Oct 30 17:00:07.177028 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:07.177002 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm_odh-ai-gateway-infra(0015d61c-9d93-4652-9414-3b817ab96276)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" podUID="0015d61c-9d93-4652-9414-3b817ab96276" Oct 30 17:00:07.185993 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.185958 6202 scope.go:122] "RemoveContainer" containerID="b9cdeb971897919dc7da636d827d41a76c3dd08b4e1cb7ac266874c7b1a8f251" Oct 30 17:00:07.186552 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:07.186512 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr_odh-ai-gateway-infra(c7550e3d-3bc1-4ff5-a641-0d68b4bf6127)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" podUID="c7550e3d-3bc1-4ff5-a641-0d68b4bf6127" Oct 30 17:00:07.188657 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.188539 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:07.190239 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.190180 6202 scope.go:122] "RemoveContainer" containerID="24bf9e58d54ea4ed0abf0b7e57ef5beb8951ebb54ad07413972d8314be4acfc9" Oct 30 17:00:07.190891 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:07.190864 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t_odh-ai-gateway-infra(c0b2402f-79e8-4586-9f36-2b86f307f1cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" podUID="c0b2402f-79e8-4586-9f36-2b86f307f1cb" Oct 30 17:00:07.192892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.192863 6202 scope.go:122] "RemoveContainer" containerID="4fbfc7d452355aee1a469bade80d249260ef0961687867612677eb4750b5584f" Oct 30 17:00:07.193335 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:07.193304 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz_odh-ai-gateway-infra(bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" podUID="bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5" Oct 30 17:00:07.195425 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.195357 6202 scope.go:122] "RemoveContainer" containerID="4c3473df9be92d62380d4ed6bf05c7aa17c78370916ae619ed19e2b4f34ddc83" Oct 30 17:00:07.195847 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:07.195791 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v_odh-ai-gateway-infra(7aadca90-6dbe-4fe5-9dee-10431e9210a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" podUID="7aadca90-6dbe-4fe5-9dee-10431e9210a0" Oct 30 17:00:07.407699 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.407661 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:07.473396 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.473366 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl" Oct 30 17:00:07.502871 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.502833 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-maas-api-openshift-ca\") pod \"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9\" (UID: \"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9\") " Oct 30 17:00:07.503066 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.503045 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-maas-api-xks-ca\") pod \"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9\" (UID: \"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9\") " Oct 30 17:00:07.503256 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.503235 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-maas-api-openshift-ca" pod "abac9b81-c250-4d4d-b1ed-f0615a7f2dd9" (UID: "abac9b81-c250-4d4d-b1ed-f0615a7f2dd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:07.503318 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.503264 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-kube-api-access-hn9lc\" (UniqueName: \"kubernetes.io/projected/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-kube-api-access-hn9lc\") pod \"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9\" (UID: \"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9\") " Oct 30 17:00:07.504223 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.504188 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:07.505544 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.505506 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-maas-api-xks-ca" pod "abac9b81-c250-4d4d-b1ed-f0615a7f2dd9" (UID: "abac9b81-c250-4d4d-b1ed-f0615a7f2dd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:07.505742 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.505715 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-kube-api-access-hn9lc" pod "abac9b81-c250-4d4d-b1ed-f0615a7f2dd9" (UID: "abac9b81-c250-4d4d-b1ed-f0615a7f2dd9"). InnerVolumeSpecName "kube-api-access-hn9lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:07.605981 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.605881 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:07.605981 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:07.605911 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn9lc\" (UniqueName: \"kubernetes.io/projected/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9-kube-api-access-hn9lc\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:08.204974 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:08.204943 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl" Oct 30 17:00:08.205477 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:08.204978 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl" event={"ID":"abac9b81-c250-4d4d-b1ed-f0615a7f2dd9","Type":"ContainerDied","Data":"c21297396157778082fbaddbf2d17a469ca355957467b01721e8750c59c9940f"} Oct 30 17:00:08.205477 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:08.205023 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c21297396157778082fbaddbf2d17a469ca355957467b01721e8750c59c9940f" Oct 30 17:00:08.817965 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:08.817915 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" podUID="3ddb2df8-3531-4d65-a0fe-7fe1facc43ee" containerName="payload-pre-processing" containerID="cri-o://a28c6990e01344e61b4c79d43cb62bf0ef4f692cb32791d1844fb9ed834b8744" gracePeriod=25 Oct 30 17:00:08.877516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:08.877480 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:10.482432 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:10.482396 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:12.019918 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.019865 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" podUID="242e1339-d85b-4360-99c9-6aab93ac561d" containerName="payload-processing" containerID="cri-o://ea0e811746fe7c4e37a8ccd1628b13deafdcbfbfb97277f978ae6ef29f699b2b" gracePeriod=25 Oct 30 17:00:12.068420 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.068381 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:12.673883 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.673838 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn"] Oct 30 17:00:12.674811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.674758 6202 cpu_manager.go:414] "containerMap: removing container" podUID="abac9b81-c250-4d4d-b1ed-f0615a7f2dd9" containerName="revoke-keys" Oct 30 17:00:12.674811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.674790 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abac9b81-c250-4d4d-b1ed-f0615a7f2dd9" containerName="revoke-keys" Oct 30 17:00:12.674811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.674810 6202 cpu_manager.go:414] "containerMap: removing container" podUID="927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" containerName="istio-proxy" Oct 30 17:00:12.674811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.674815 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="927cb01d-5133-4ecd-abb7-fa13f1fbe8c5" containerName="istio-proxy" Oct 30 17:00:12.675098 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.674992 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="abac9b81-c250-4d4d-b1ed-f0615a7f2dd9" containerName="revoke-keys" Oct 30 17:00:12.679331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.679306 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn" Oct 30 17:00:12.694474 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.694447 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:00:12.712579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.712545 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn"] Oct 30 17:00:12.804505 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.804458 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-376c46dc-nqkpn\" (UID: \"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn" Oct 30 17:00:12.804676 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.804611 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27fvr\" (UniqueName: \"kubernetes.io/projected/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-kube-api-access-27fvr\") pod \"maas-api-revoke-sub-376c46dc-nqkpn\" (UID: \"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn" Oct 30 17:00:12.804676 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.804672 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-376c46dc-nqkpn\" (UID: \"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn" Oct 30 17:00:12.906338 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.906306 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27fvr\" (UniqueName: \"kubernetes.io/projected/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-kube-api-access-27fvr\") pod \"maas-api-revoke-sub-376c46dc-nqkpn\" (UID: \"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn" Oct 30 17:00:12.906553 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.906378 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-376c46dc-nqkpn\" (UID: \"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn" Oct 30 17:00:12.906674 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.906655 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-376c46dc-nqkpn\" (UID: \"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn" Oct 30 17:00:12.907254 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.907233 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-376c46dc-nqkpn\" (UID: \"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn" Oct 30 17:00:12.908995 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.908976 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-376c46dc-nqkpn\" (UID: \"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn" Oct 30 17:00:12.930321 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.930255 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27fvr\" (UniqueName: \"kubernetes.io/projected/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-kube-api-access-27fvr\") pod \"maas-api-revoke-sub-376c46dc-nqkpn\" (UID: \"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn" Oct 30 17:00:12.997119 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:12.997072 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn" Oct 30 17:00:13.333255 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:00:13.333225 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6497d9_6147_4773_bc38_5e0f6bf7eb2f.slice/crio-af1de6e929aa21e4543dd6db20c7a0611bd1ef2f8e393cd428f533922b2a3d93 WatchSource:0}: Error finding container af1de6e929aa21e4543dd6db20c7a0611bd1ef2f8e393cd428f533922b2a3d93: Status 404 returned error can't find the container with id af1de6e929aa21e4543dd6db20c7a0611bd1ef2f8e393cd428f533922b2a3d93 Oct 30 17:00:13.334202 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:13.334156 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn"] Oct 30 17:00:13.855871 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:13.855833 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:14.071400 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:14.071363 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:00:14.271779 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:14.271731 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn" event={"ID":"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f","Type":"ContainerStarted","Data":"b60c30e7c6cb74346696e303b842e03ea4a81a28cd3e46a68fc8ea411dfd9455"} Oct 30 17:00:14.271952 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:14.271810 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn" event={"ID":"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f","Type":"ContainerStarted","Data":"af1de6e929aa21e4543dd6db20c7a0611bd1ef2f8e393cd428f533922b2a3d93"} Oct 30 17:00:14.315921 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:14.315856 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn" podStartSLOduration=2.315838476 podStartE2EDuration="2.315838476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:14.314064075 +0000 UTC m=+2328.024518819" watchObservedRunningTime="2026-10-30 17:00:14.315838476 +0000 UTC m=+2328.026293221" Oct 30 17:00:14.693793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:14.693683 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" podUID="3ddb2df8-3531-4d65-a0fe-7fe1facc43ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:00:15.020344 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:15.020262 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:15.285579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:15.285481 6202 generic.go:356] "Generic (PLEG): container finished" podID="9b6497d9-6147-4773-bc38-5e0f6bf7eb2f" containerID="b60c30e7c6cb74346696e303b842e03ea4a81a28cd3e46a68fc8ea411dfd9455" exitCode=0 Oct 30 17:00:15.285804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:15.285653 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn" event={"ID":"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f","Type":"ContainerDied","Data":"b60c30e7c6cb74346696e303b842e03ea4a81a28cd3e46a68fc8ea411dfd9455"} Oct 30 17:00:15.579708 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:15.579664 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:16.413907 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:16.413871 6202 scope.go:122] "RemoveContainer" containerID="7fd04118ae9210a0885d8015555ed0263dd03a04c0b1bfa4abda639b0ffb6a03" Oct 30 17:00:16.414374 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:16.414073 6202 scope.go:122] "RemoveContainer" containerID="3d415d4226ca895027a5d5ce28b1d948cd9e6d077220d537e269938018a23e65" Oct 30 17:00:16.414374 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:16.414272 6202 scope.go:122] "RemoveContainer" containerID="9f0340876d419cacb0be50e95574582fdcee83cc6ce5786efc7ad18984a7832e" Oct 30 17:00:16.414854 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:16.414752 6202 scope.go:122] "RemoveContainer" containerID="4eeee6b375a206c0296015b431aa464978d137a35e4bf7e2b707492aa9da3405" Oct 30 17:00:16.422585 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:16.416917 6202 scope.go:122] "RemoveContainer" containerID="aa1d0e08671907f48810b40b89681ea8acd0a39f885bf5bc3a3a1eae1c188d50" Oct 30 17:00:16.527888 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:16.527733 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn" Oct 30 17:00:16.590680 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:16.590645 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-kube-api-access-27fvr\" (UniqueName: \"kubernetes.io/projected/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-kube-api-access-27fvr\") pod \"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f\" (UID: \"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f\") " Oct 30 17:00:16.590907 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:16.590848 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-maas-api-xks-ca\") pod \"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f\" (UID: \"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f\") " Oct 30 17:00:16.591116 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:16.591090 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-maas-api-openshift-ca\") pod \"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f\" (UID: \"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f\") " Oct 30 17:00:16.592373 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:16.592338 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-maas-api-openshift-ca" pod "9b6497d9-6147-4773-bc38-5e0f6bf7eb2f" (UID: "9b6497d9-6147-4773-bc38-5e0f6bf7eb2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:16.641321 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:16.641244 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-kube-api-access-27fvr" pod "9b6497d9-6147-4773-bc38-5e0f6bf7eb2f" (UID: "9b6497d9-6147-4773-bc38-5e0f6bf7eb2f"). InnerVolumeSpecName "kube-api-access-27fvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:16.641733 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:16.641701 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-maas-api-xks-ca" pod "9b6497d9-6147-4773-bc38-5e0f6bf7eb2f" (UID: "9b6497d9-6147-4773-bc38-5e0f6bf7eb2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:16.693249 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:16.693216 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:16.693249 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:16.693243 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:16.693249 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:16.693254 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27fvr\" (UniqueName: \"kubernetes.io/projected/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f-kube-api-access-27fvr\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:17.306115 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:17.306080 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:17.309472 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:17.309439 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" event={"ID":"ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9","Type":"ContainerStarted","Data":"01149e372985bcc4f83f0d33714577eeb7e96af14122fe0550f14b50d6ffee9b"} Oct 30 17:00:17.311142 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:17.311113 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn" Oct 30 17:00:17.311280 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:17.311121 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn" event={"ID":"9b6497d9-6147-4773-bc38-5e0f6bf7eb2f","Type":"ContainerDied","Data":"af1de6e929aa21e4543dd6db20c7a0611bd1ef2f8e393cd428f533922b2a3d93"} Oct 30 17:00:17.311280 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:17.311247 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af1de6e929aa21e4543dd6db20c7a0611bd1ef2f8e393cd428f533922b2a3d93" Oct 30 17:00:17.356850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:17.354286 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" event={"ID":"0015d61c-9d93-4652-9414-3b817ab96276","Type":"ContainerStarted","Data":"78883660a996f0973e2889ec762ed032d509ec493808b3bf3f9ceeb946e318ad"} Oct 30 17:00:17.398790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:17.395052 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" event={"ID":"8c799e0b-668f-4c55-a254-69651dd5b005","Type":"ContainerStarted","Data":"8c52c60e2f444b88e4589a13171e7343e9b902acd83a8b9ae620d5fc8f237d24"} Oct 30 17:00:17.398790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:17.398679 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" event={"ID":"5ba09798-de28-4d76-8fea-839bbd8ea7e4","Type":"ContainerStarted","Data":"b4a34ab0ee69606279cccdb73f2134b3544b79bd25cc39499fccd11a2e150516"} Oct 30 17:00:17.401223 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:17.401196 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" event={"ID":"6957d4c0-8197-4aab-a89a-ad17414cd011","Type":"ContainerStarted","Data":"4f1d17d26e02ce0fbdaee7a58a07eca9080878d956179084e83fbe8f9a03036f"} Oct 30 17:00:17.409067 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:17.408679 6202 scope.go:122] "RemoveContainer" containerID="5494b76687547e1856d8261910b0a37ef5c68020dc1bf7ff5f0f4d0a8bc40fc4" Oct 30 17:00:17.410127 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:17.410098 6202 scope.go:122] "RemoveContainer" containerID="24bf9e58d54ea4ed0abf0b7e57ef5beb8951ebb54ad07413972d8314be4acfc9" Oct 30 17:00:17.410266 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:17.410189 6202 scope.go:122] "RemoveContainer" containerID="3a5cb5cb4c040e61566da3d768d17d6fa862ec4badc18d0cdcb91fa1b2f8e1ef" Oct 30 17:00:17.410545 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:17.410524 6202 scope.go:122] "RemoveContainer" containerID="f57ca7fd3cfc4571285d62a55f62b6e3c02e53dd4ec17b1985449f2642623d6b" Oct 30 17:00:17.525196 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:17.525137 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" podStartSLOduration=17.525118492 podStartE2EDuration="17.525118492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:17.479987718 +0000 UTC m=+2331.190442463" watchObservedRunningTime="2026-10-30 17:00:17.525118492 +0000 UTC m=+2331.235573236" Oct 30 17:00:18.379241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.379199 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2"] Oct 30 17:00:18.403629 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.403584 6202 scope.go:122] "RemoveContainer" containerID="abdd5c043c18619f8be1cb8b70e5e927882242ca6bd9190737984ba7a471ea8d" Oct 30 17:00:18.404544 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.404060 6202 scope.go:122] "RemoveContainer" containerID="ea8884b55cebe30dc596e0566583a46ef5526e56107231126a175854bb0f661e" Oct 30 17:00:18.404544 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.404184 6202 scope.go:122] "RemoveContainer" containerID="b9cdeb971897919dc7da636d827d41a76c3dd08b4e1cb7ac266874c7b1a8f251" Oct 30 17:00:18.423731 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.423691 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" event={"ID":"5c75b279-d01e-43e8-9a3e-8dff3dae29c7","Type":"ContainerStarted","Data":"ba56572ce8cb11a1ea9bfdbd3fd7d24122b8884075603c4ef87fb96247768c1a"} Oct 30 17:00:18.426267 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.426233 6202 generic.go:356] "Generic (PLEG): container finished" podID="8c799e0b-668f-4c55-a254-69651dd5b005" containerID="8c52c60e2f444b88e4589a13171e7343e9b902acd83a8b9ae620d5fc8f237d24" exitCode=60 Oct 30 17:00:18.426503 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.426316 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" event={"ID":"8c799e0b-668f-4c55-a254-69651dd5b005","Type":"ContainerDied","Data":"8c52c60e2f444b88e4589a13171e7343e9b902acd83a8b9ae620d5fc8f237d24"} Oct 30 17:00:18.426503 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.426382 6202 scope.go:122] "RemoveContainer" containerID="3d415d4226ca895027a5d5ce28b1d948cd9e6d077220d537e269938018a23e65" Oct 30 17:00:18.427031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.427007 6202 scope.go:122] "RemoveContainer" containerID="8c52c60e2f444b88e4589a13171e7343e9b902acd83a8b9ae620d5fc8f237d24" Oct 30 17:00:18.427445 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:18.427418 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r_odh-ai-gateway-infra(8c799e0b-668f-4c55-a254-69651dd5b005)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" podUID="8c799e0b-668f-4c55-a254-69651dd5b005" Oct 30 17:00:18.429046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.429021 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" event={"ID":"75af19e6-9fae-4786-86d2-3ef7f4433f08","Type":"ContainerStarted","Data":"b32bf17af3915951b649e2580ace183fb3cc72866ad90a89d8acd2c0a0ea949e"} Oct 30 17:00:18.431133 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.431112 6202 generic.go:356] "Generic (PLEG): container finished" podID="5ba09798-de28-4d76-8fea-839bbd8ea7e4" containerID="b4a34ab0ee69606279cccdb73f2134b3544b79bd25cc39499fccd11a2e150516" exitCode=60 Oct 30 17:00:18.431223 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.431181 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" event={"ID":"5ba09798-de28-4d76-8fea-839bbd8ea7e4","Type":"ContainerDied","Data":"b4a34ab0ee69606279cccdb73f2134b3544b79bd25cc39499fccd11a2e150516"} Oct 30 17:00:18.431516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.431500 6202 scope.go:122] "RemoveContainer" containerID="b4a34ab0ee69606279cccdb73f2134b3544b79bd25cc39499fccd11a2e150516" Oct 30 17:00:18.431848 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:18.431823 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889660-l5wqc_odh-ai-gateway-infra(5ba09798-de28-4d76-8fea-839bbd8ea7e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" podUID="5ba09798-de28-4d76-8fea-839bbd8ea7e4" Oct 30 17:00:18.433352 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.433329 6202 generic.go:356] "Generic (PLEG): container finished" podID="6957d4c0-8197-4aab-a89a-ad17414cd011" containerID="4f1d17d26e02ce0fbdaee7a58a07eca9080878d956179084e83fbe8f9a03036f" exitCode=60 Oct 30 17:00:18.433435 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.433412 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" event={"ID":"6957d4c0-8197-4aab-a89a-ad17414cd011","Type":"ContainerDied","Data":"4f1d17d26e02ce0fbdaee7a58a07eca9080878d956179084e83fbe8f9a03036f"} Oct 30 17:00:18.433869 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.433852 6202 scope.go:122] "RemoveContainer" containerID="4f1d17d26e02ce0fbdaee7a58a07eca9080878d956179084e83fbe8f9a03036f" Oct 30 17:00:18.434186 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:18.434163 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj_odh-ai-gateway-infra(6957d4c0-8197-4aab-a89a-ad17414cd011)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" podUID="6957d4c0-8197-4aab-a89a-ad17414cd011" Oct 30 17:00:18.435670 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.435647 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" event={"ID":"c17f9002-4b15-4e46-a720-7a298c9c3f68","Type":"ContainerStarted","Data":"0b9e050e5d155bbdbcab2c832001f640f9d90e8acb36e56cbda1b7114cc4a5f6"} Oct 30 17:00:18.437759 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.437734 6202 generic.go:356] "Generic (PLEG): container finished" podID="ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9" containerID="01149e372985bcc4f83f0d33714577eeb7e96af14122fe0550f14b50d6ffee9b" exitCode=60 Oct 30 17:00:18.438016 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.437959 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" event={"ID":"ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9","Type":"ContainerDied","Data":"01149e372985bcc4f83f0d33714577eeb7e96af14122fe0550f14b50d6ffee9b"} Oct 30 17:00:18.444666 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.444631 6202 generic.go:356] "Generic (PLEG): container finished" podID="0015d61c-9d93-4652-9414-3b817ab96276" containerID="78883660a996f0973e2889ec762ed032d509ec493808b3bf3f9ceeb946e318ad" exitCode=60 Oct 30 17:00:18.444805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.444727 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" event={"ID":"0015d61c-9d93-4652-9414-3b817ab96276","Type":"ContainerDied","Data":"78883660a996f0973e2889ec762ed032d509ec493808b3bf3f9ceeb946e318ad"} Oct 30 17:00:18.445262 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.445236 6202 scope.go:122] "RemoveContainer" containerID="78883660a996f0973e2889ec762ed032d509ec493808b3bf3f9ceeb946e318ad" Oct 30 17:00:18.445721 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:18.445690 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm_odh-ai-gateway-infra(0015d61c-9d93-4652-9414-3b817ab96276)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" podUID="0015d61c-9d93-4652-9414-3b817ab96276" Oct 30 17:00:18.448104 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.448048 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm"] Oct 30 17:00:18.451027 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.450999 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" event={"ID":"c0b2402f-79e8-4586-9f36-2b86f307f1cb","Type":"ContainerStarted","Data":"218af457aac26e4028572ac654bff9abf34b8a95160a125dda5b2b279d1e3dd7"} Oct 30 17:00:18.467924 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.467734 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" podStartSLOduration=18.467718332 podStartE2EDuration="18.467718332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:18.466127744 +0000 UTC m=+2332.176582489" watchObservedRunningTime="2026-10-30 17:00:18.467718332 +0000 UTC m=+2332.178173070" Oct 30 17:00:18.493869 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.493818 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r"] Oct 30 17:00:18.533306 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.533196 6202 scope.go:122] "RemoveContainer" containerID="9f0340876d419cacb0be50e95574582fdcee83cc6ce5786efc7ad18984a7832e" Oct 30 17:00:18.559592 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.557901 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj"] Oct 30 17:00:18.575263 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.574456 6202 scope.go:122] "RemoveContainer" containerID="4eeee6b375a206c0296015b431aa464978d137a35e4bf7e2b707492aa9da3405" Oct 30 17:00:18.582054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.580676 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc"] Oct 30 17:00:18.606779 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.606733 6202 scope.go:122] "RemoveContainer" containerID="7fd04118ae9210a0885d8015555ed0263dd03a04c0b1bfa4abda639b0ffb6a03" Oct 30 17:00:18.636868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.635079 6202 scope.go:122] "RemoveContainer" containerID="aa1d0e08671907f48810b40b89681ea8acd0a39f885bf5bc3a3a1eae1c188d50" Oct 30 17:00:18.857480 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.857447 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" Oct 30 17:00:18.878709 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.878671 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9-maas-api-service-ca\") pod \"ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9\" (UID: \"ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9\") " Oct 30 17:00:18.879123 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.879088 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9-kube-api-access-k5px6\" (UniqueName: \"kubernetes.io/projected/ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9-kube-api-access-k5px6\") pod \"ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9\" (UID: \"ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9\") " Oct 30 17:00:18.879259 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.879133 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9-maas-api-service-ca" pod "ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9" (UID: "ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:18.879830 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.879810 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:18.882252 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.882218 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9-kube-api-access-k5px6" pod "ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9" (UID: "ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9"). InnerVolumeSpecName "kube-api-access-k5px6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:18.984096 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:18.984056 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5px6\" (UniqueName: \"kubernetes.io/projected/ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9-kube-api-access-k5px6\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:19.064554 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.064519 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:19.403988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.403896 6202 scope.go:122] "RemoveContainer" containerID="4fbfc7d452355aee1a469bade80d249260ef0961687867612677eb4750b5584f" Oct 30 17:00:19.404158 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.404072 6202 scope.go:122] "RemoveContainer" containerID="d3f58a5a36aed980ceb99e50148481fc32db504eac8cb7feb4cdfe371177486c" Oct 30 17:00:19.464878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.464840 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" event={"ID":"660c4258-30bc-4a2f-b130-766dcbbd88f4","Type":"ContainerStarted","Data":"cd09df53d3bf9d7804d0f94c2a4af6e3d7b9a246b3bd2daba7a79646d36a9085"} Oct 30 17:00:19.467516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.467485 6202 generic.go:356] "Generic (PLEG): container finished" podID="5c75b279-d01e-43e8-9a3e-8dff3dae29c7" containerID="ba56572ce8cb11a1ea9bfdbd3fd7d24122b8884075603c4ef87fb96247768c1a" exitCode=60 Oct 30 17:00:19.467689 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.467563 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" event={"ID":"5c75b279-d01e-43e8-9a3e-8dff3dae29c7","Type":"ContainerDied","Data":"ba56572ce8cb11a1ea9bfdbd3fd7d24122b8884075603c4ef87fb96247768c1a"} Oct 30 17:00:19.467689 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.467633 6202 scope.go:122] "RemoveContainer" containerID="f57ca7fd3cfc4571285d62a55f62b6e3c02e53dd4ec17b1985449f2642623d6b" Oct 30 17:00:19.468256 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.468229 6202 scope.go:122] "RemoveContainer" containerID="ba56572ce8cb11a1ea9bfdbd3fd7d24122b8884075603c4ef87fb96247768c1a" Oct 30 17:00:19.468932 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:19.468697 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh_odh-ai-gateway-infra(5c75b279-d01e-43e8-9a3e-8dff3dae29c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" podUID="5c75b279-d01e-43e8-9a3e-8dff3dae29c7" Oct 30 17:00:19.472383 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.472359 6202 generic.go:356] "Generic (PLEG): container finished" podID="75af19e6-9fae-4786-86d2-3ef7f4433f08" containerID="b32bf17af3915951b649e2580ace183fb3cc72866ad90a89d8acd2c0a0ea949e" exitCode=60 Oct 30 17:00:19.472486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.472435 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" event={"ID":"75af19e6-9fae-4786-86d2-3ef7f4433f08","Type":"ContainerDied","Data":"b32bf17af3915951b649e2580ace183fb3cc72866ad90a89d8acd2c0a0ea949e"} Oct 30 17:00:19.473022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.472989 6202 scope.go:122] "RemoveContainer" containerID="b32bf17af3915951b649e2580ace183fb3cc72866ad90a89d8acd2c0a0ea949e" Oct 30 17:00:19.473485 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:19.473452 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh_odh-ai-gateway-infra(75af19e6-9fae-4786-86d2-3ef7f4433f08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" podUID="75af19e6-9fae-4786-86d2-3ef7f4433f08" Oct 30 17:00:19.482496 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.482466 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh"] Oct 30 17:00:19.495284 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.495241 6202 generic.go:356] "Generic (PLEG): container finished" podID="c17f9002-4b15-4e46-a720-7a298c9c3f68" containerID="0b9e050e5d155bbdbcab2c832001f640f9d90e8acb36e56cbda1b7114cc4a5f6" exitCode=60 Oct 30 17:00:19.495284 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.495264 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" event={"ID":"c17f9002-4b15-4e46-a720-7a298c9c3f68","Type":"ContainerDied","Data":"0b9e050e5d155bbdbcab2c832001f640f9d90e8acb36e56cbda1b7114cc4a5f6"} Oct 30 17:00:19.495863 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.495839 6202 scope.go:122] "RemoveContainer" containerID="0b9e050e5d155bbdbcab2c832001f640f9d90e8acb36e56cbda1b7114cc4a5f6" Oct 30 17:00:19.496300 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:19.496270 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k_odh-ai-gateway-infra(c17f9002-4b15-4e46-a720-7a298c9c3f68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" podUID="c17f9002-4b15-4e46-a720-7a298c9c3f68" Oct 30 17:00:19.497754 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.497722 6202 scope.go:122] "RemoveContainer" containerID="5494b76687547e1856d8261910b0a37ef5c68020dc1bf7ff5f0f4d0a8bc40fc4" Oct 30 17:00:19.506810 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.506449 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" event={"ID":"ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9","Type":"ContainerDied","Data":"9ea9932c9100021fe5a7eafac7733e84c7e534e1e4c94c22af9a0d2ba91f364e"} Oct 30 17:00:19.506810 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.506538 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2" Oct 30 17:00:19.516543 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.516498 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" event={"ID":"3d338cc2-8d69-4b14-99d1-370710f8cb99","Type":"ContainerStarted","Data":"1554ffaef1464ad6b562565d6ffbc579d43a8c924f5cd211877574cc9ab3ad85"} Oct 30 17:00:19.526558 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.525553 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" event={"ID":"c7550e3d-3bc1-4ff5-a641-0d68b4bf6127","Type":"ContainerStarted","Data":"14f9389e04aec67275ac7127a097af1551d9e2dd54da95353372974f0d3c9c7d"} Oct 30 17:00:19.531231 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.530121 6202 generic.go:356] "Generic (PLEG): container finished" podID="c0b2402f-79e8-4586-9f36-2b86f307f1cb" containerID="218af457aac26e4028572ac654bff9abf34b8a95160a125dda5b2b279d1e3dd7" exitCode=60 Oct 30 17:00:19.531231 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.530227 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" event={"ID":"c0b2402f-79e8-4586-9f36-2b86f307f1cb","Type":"ContainerDied","Data":"218af457aac26e4028572ac654bff9abf34b8a95160a125dda5b2b279d1e3dd7"} Oct 30 17:00:19.531231 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.530699 6202 scope.go:122] "RemoveContainer" containerID="218af457aac26e4028572ac654bff9abf34b8a95160a125dda5b2b279d1e3dd7" Oct 30 17:00:19.531231 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:19.531166 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t_odh-ai-gateway-infra(c0b2402f-79e8-4586-9f36-2b86f307f1cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" podUID="c0b2402f-79e8-4586-9f36-2b86f307f1cb" Oct 30 17:00:19.532858 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.532842 6202 scope.go:122] "RemoveContainer" containerID="3a5cb5cb4c040e61566da3d768d17d6fa862ec4badc18d0cdcb91fa1b2f8e1ef" Oct 30 17:00:19.577120 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.577088 6202 scope.go:122] "RemoveContainer" containerID="01149e372985bcc4f83f0d33714577eeb7e96af14122fe0550f14b50d6ffee9b" Oct 30 17:00:19.598825 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.598474 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t"] Oct 30 17:00:19.608468 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.607926 6202 scope.go:122] "RemoveContainer" containerID="24bf9e58d54ea4ed0abf0b7e57ef5beb8951ebb54ad07413972d8314be4acfc9" Oct 30 17:00:19.721989 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.721894 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k"] Oct 30 17:00:19.815170 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.815139 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh"] Oct 30 17:00:19.895083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.895050 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2"] Oct 30 17:00:19.900823 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.900789 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d693d0-29889660-xc4j2"] Oct 30 17:00:19.932891 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:19.932828 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" podStartSLOduration=19.932811845 podStartE2EDuration="19.932811845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:19.929695907 +0000 UTC m=+2333.640150655" watchObservedRunningTime="2026-10-30 17:00:19.932811845 +0000 UTC m=+2333.643266588" Oct 30 17:00:20.031753 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.031721 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" Oct 30 17:00:20.036293 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.036268 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" Oct 30 17:00:20.041781 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.041732 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" Oct 30 17:00:20.048322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.048242 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" Oct 30 17:00:20.115876 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.115835 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0015d61c-9d93-4652-9414-3b817ab96276-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0015d61c-9d93-4652-9414-3b817ab96276-maas-api-service-ca\") pod \"0015d61c-9d93-4652-9414-3b817ab96276\" (UID: \"0015d61c-9d93-4652-9414-3b817ab96276\") " Oct 30 17:00:20.116069 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.116001 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c799e0b-668f-4c55-a254-69651dd5b005-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c799e0b-668f-4c55-a254-69651dd5b005-maas-api-service-ca\") pod \"8c799e0b-668f-4c55-a254-69651dd5b005\" (UID: \"8c799e0b-668f-4c55-a254-69651dd5b005\") " Oct 30 17:00:20.116135 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.116114 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6957d4c0-8197-4aab-a89a-ad17414cd011-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6957d4c0-8197-4aab-a89a-ad17414cd011-maas-api-service-ca\") pod \"6957d4c0-8197-4aab-a89a-ad17414cd011\" (UID: \"6957d4c0-8197-4aab-a89a-ad17414cd011\") " Oct 30 17:00:20.116207 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.116186 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba09798-de28-4d76-8fea-839bbd8ea7e4-kube-api-access-84r6t\" (UniqueName: \"kubernetes.io/projected/5ba09798-de28-4d76-8fea-839bbd8ea7e4-kube-api-access-84r6t\") pod \"5ba09798-de28-4d76-8fea-839bbd8ea7e4\" (UID: \"5ba09798-de28-4d76-8fea-839bbd8ea7e4\") " Oct 30 17:00:20.116301 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.116285 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c799e0b-668f-4c55-a254-69651dd5b005-kube-api-access-7pwvq\" (UniqueName: \"kubernetes.io/projected/8c799e0b-668f-4c55-a254-69651dd5b005-kube-api-access-7pwvq\") pod \"8c799e0b-668f-4c55-a254-69651dd5b005\" (UID: \"8c799e0b-668f-4c55-a254-69651dd5b005\") " Oct 30 17:00:20.116365 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.116332 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6957d4c0-8197-4aab-a89a-ad17414cd011-kube-api-access-bmbcs\" (UniqueName: \"kubernetes.io/projected/6957d4c0-8197-4aab-a89a-ad17414cd011-kube-api-access-bmbcs\") pod \"6957d4c0-8197-4aab-a89a-ad17414cd011\" (UID: \"6957d4c0-8197-4aab-a89a-ad17414cd011\") " Oct 30 17:00:20.116422 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.116405 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0015d61c-9d93-4652-9414-3b817ab96276-kube-api-access-s45pg\" (UniqueName: \"kubernetes.io/projected/0015d61c-9d93-4652-9414-3b817ab96276-kube-api-access-s45pg\") pod \"0015d61c-9d93-4652-9414-3b817ab96276\" (UID: \"0015d61c-9d93-4652-9414-3b817ab96276\") " Oct 30 17:00:20.116483 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.116453 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0015d61c-9d93-4652-9414-3b817ab96276-maas-api-service-ca" pod "0015d61c-9d93-4652-9414-3b817ab96276" (UID: "0015d61c-9d93-4652-9414-3b817ab96276"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:20.116559 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.116540 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba09798-de28-4d76-8fea-839bbd8ea7e4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba09798-de28-4d76-8fea-839bbd8ea7e4-maas-api-service-ca\") pod \"5ba09798-de28-4d76-8fea-839bbd8ea7e4\" (UID: \"5ba09798-de28-4d76-8fea-839bbd8ea7e4\") " Oct 30 17:00:20.117393 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.117366 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0015d61c-9d93-4652-9414-3b817ab96276-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.117755 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.117731 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba09798-de28-4d76-8fea-839bbd8ea7e4-maas-api-service-ca" pod "5ba09798-de28-4d76-8fea-839bbd8ea7e4" (UID: "5ba09798-de28-4d76-8fea-839bbd8ea7e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:20.118981 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.118952 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba09798-de28-4d76-8fea-839bbd8ea7e4-kube-api-access-84r6t" pod "5ba09798-de28-4d76-8fea-839bbd8ea7e4" (UID: "5ba09798-de28-4d76-8fea-839bbd8ea7e4"). InnerVolumeSpecName "kube-api-access-84r6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:20.119244 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.119225 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6957d4c0-8197-4aab-a89a-ad17414cd011-maas-api-service-ca" pod "6957d4c0-8197-4aab-a89a-ad17414cd011" (UID: "6957d4c0-8197-4aab-a89a-ad17414cd011"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:20.119342 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.119262 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c799e0b-668f-4c55-a254-69651dd5b005-maas-api-service-ca" pod "8c799e0b-668f-4c55-a254-69651dd5b005" (UID: "8c799e0b-668f-4c55-a254-69651dd5b005"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:20.121145 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.121106 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c799e0b-668f-4c55-a254-69651dd5b005-kube-api-access-7pwvq" pod "8c799e0b-668f-4c55-a254-69651dd5b005" (UID: "8c799e0b-668f-4c55-a254-69651dd5b005"). InnerVolumeSpecName "kube-api-access-7pwvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:20.121662 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.121636 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6957d4c0-8197-4aab-a89a-ad17414cd011-kube-api-access-bmbcs" pod "6957d4c0-8197-4aab-a89a-ad17414cd011" (UID: "6957d4c0-8197-4aab-a89a-ad17414cd011"). InnerVolumeSpecName "kube-api-access-bmbcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:20.122415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.122394 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0015d61c-9d93-4652-9414-3b817ab96276-kube-api-access-s45pg" pod "0015d61c-9d93-4652-9414-3b817ab96276" (UID: "0015d61c-9d93-4652-9414-3b817ab96276"). InnerVolumeSpecName "kube-api-access-s45pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:20.220187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.220065 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6957d4c0-8197-4aab-a89a-ad17414cd011-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.220187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.220102 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84r6t\" (UniqueName: \"kubernetes.io/projected/5ba09798-de28-4d76-8fea-839bbd8ea7e4-kube-api-access-84r6t\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.220187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.220116 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pwvq\" (UniqueName: \"kubernetes.io/projected/8c799e0b-668f-4c55-a254-69651dd5b005-kube-api-access-7pwvq\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.220187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.220129 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmbcs\" (UniqueName: \"kubernetes.io/projected/6957d4c0-8197-4aab-a89a-ad17414cd011-kube-api-access-bmbcs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.220187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.220141 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s45pg\" (UniqueName: \"kubernetes.io/projected/0015d61c-9d93-4652-9414-3b817ab96276-kube-api-access-s45pg\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.220187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.220152 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba09798-de28-4d76-8fea-839bbd8ea7e4-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.220187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.220164 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c799e0b-668f-4c55-a254-69651dd5b005-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.403410 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.403375 6202 scope.go:122] "RemoveContainer" containerID="1b7cac64d71077169b81fc41df654c1870c84e6335d484a1078041354ff0c28b" Oct 30 17:00:20.403604 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.403447 6202 scope.go:122] "RemoveContainer" containerID="4c3473df9be92d62380d4ed6bf05c7aa17c78370916ae619ed19e2b4f34ddc83" Oct 30 17:00:20.433074 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.433038 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9" path="/var/lib/kubelet/pods/ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9/volumes" Oct 30 17:00:20.526602 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.526571 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4"] Oct 30 17:00:20.552636 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.550718 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" event={"ID":"0015d61c-9d93-4652-9414-3b817ab96276","Type":"ContainerDied","Data":"c7bc739f4037063c450d01b6478295b8973e9eee0f75b142dae83a9dbc89689a"} Oct 30 17:00:20.552636 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.550818 6202 scope.go:122] "RemoveContainer" containerID="78883660a996f0973e2889ec762ed032d509ec493808b3bf3f9ceeb946e318ad" Oct 30 17:00:20.552636 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.550972 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm" Oct 30 17:00:20.558564 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.556289 6202 generic.go:356] "Generic (PLEG): container finished" podID="c7550e3d-3bc1-4ff5-a641-0d68b4bf6127" containerID="14f9389e04aec67275ac7127a097af1551d9e2dd54da95353372974f0d3c9c7d" exitCode=60 Oct 30 17:00:20.558564 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.556383 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" event={"ID":"c7550e3d-3bc1-4ff5-a641-0d68b4bf6127","Type":"ContainerDied","Data":"14f9389e04aec67275ac7127a097af1551d9e2dd54da95353372974f0d3c9c7d"} Oct 30 17:00:20.558564 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.556931 6202 scope.go:122] "RemoveContainer" containerID="14f9389e04aec67275ac7127a097af1551d9e2dd54da95353372974f0d3c9c7d" Oct 30 17:00:20.558564 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:20.557393 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr_odh-ai-gateway-infra(c7550e3d-3bc1-4ff5-a641-0d68b4bf6127)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" podUID="c7550e3d-3bc1-4ff5-a641-0d68b4bf6127" Oct 30 17:00:20.565425 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.565388 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" event={"ID":"bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5","Type":"ContainerStarted","Data":"f7728e6df630a377df6924ee6f5ce492a82ff4929a48dfad0ae1388b09955723"} Oct 30 17:00:20.570439 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.570403 6202 generic.go:356] "Generic (PLEG): container finished" podID="660c4258-30bc-4a2f-b130-766dcbbd88f4" containerID="cd09df53d3bf9d7804d0f94c2a4af6e3d7b9a246b3bd2daba7a79646d36a9085" exitCode=60 Oct 30 17:00:20.571970 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.570860 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" event={"ID":"660c4258-30bc-4a2f-b130-766dcbbd88f4","Type":"ContainerDied","Data":"cd09df53d3bf9d7804d0f94c2a4af6e3d7b9a246b3bd2daba7a79646d36a9085"} Oct 30 17:00:20.581538 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.581502 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" event={"ID":"8c799e0b-668f-4c55-a254-69651dd5b005","Type":"ContainerDied","Data":"4f401a56aa157f20e31cc67511c3cf96f02d022a4a08e89373971fce9dfb0a58"} Oct 30 17:00:20.582193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.582172 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r" Oct 30 17:00:20.590559 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.590115 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" event={"ID":"b09a458f-a69b-441a-b91a-11854c180a4e","Type":"ContainerStarted","Data":"b49f500edf0a61d8328ca43821283e09e55a6aa1a642770154015a99694ef53f"} Oct 30 17:00:20.592821 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.591833 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" event={"ID":"5ba09798-de28-4d76-8fea-839bbd8ea7e4","Type":"ContainerDied","Data":"94835a422ba61196d754951c9120cd9a54db07e5d428aab137c2206388f92325"} Oct 30 17:00:20.592821 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.591896 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc" Oct 30 17:00:20.593929 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.593905 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" Oct 30 17:00:20.594109 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.593930 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj" event={"ID":"6957d4c0-8197-4aab-a89a-ad17414cd011","Type":"ContainerDied","Data":"a66fb2e86c910c9d0af2512fdf19e082553694abd8ff69c7027b5ba4a1df3e88"} Oct 30 17:00:20.603434 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.603399 6202 generic.go:356] "Generic (PLEG): container finished" podID="3d338cc2-8d69-4b14-99d1-370710f8cb99" containerID="1554ffaef1464ad6b562565d6ffbc579d43a8c924f5cd211877574cc9ab3ad85" exitCode=60 Oct 30 17:00:20.603630 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.603437 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" event={"ID":"3d338cc2-8d69-4b14-99d1-370710f8cb99","Type":"ContainerDied","Data":"1554ffaef1464ad6b562565d6ffbc579d43a8c924f5cd211877574cc9ab3ad85"} Oct 30 17:00:20.604146 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.604123 6202 scope.go:122] "RemoveContainer" containerID="1554ffaef1464ad6b562565d6ffbc579d43a8c924f5cd211877574cc9ab3ad85" Oct 30 17:00:20.604568 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:20.604543 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47_odh-ai-gateway-infra(3d338cc2-8d69-4b14-99d1-370710f8cb99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" podUID="3d338cc2-8d69-4b14-99d1-370710f8cb99" Oct 30 17:00:20.607648 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.607620 6202 scope.go:122] "RemoveContainer" containerID="b9cdeb971897919dc7da636d827d41a76c3dd08b4e1cb7ac266874c7b1a8f251" Oct 30 17:00:20.627028 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.626963 6202 scope.go:122] "RemoveContainer" containerID="abdd5c043c18619f8be1cb8b70e5e927882242ca6bd9190737984ba7a471ea8d" Oct 30 17:00:20.655985 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.655948 6202 scope.go:122] "RemoveContainer" containerID="8c52c60e2f444b88e4589a13171e7343e9b902acd83a8b9ae620d5fc8f237d24" Oct 30 17:00:20.657857 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.657760 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm"] Oct 30 17:00:20.672651 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.672612 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889660-j9hgm"] Oct 30 17:00:20.686906 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.686881 6202 scope.go:122] "RemoveContainer" containerID="b4a34ab0ee69606279cccdb73f2134b3544b79bd25cc39499fccd11a2e150516" Oct 30 17:00:20.708612 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.708492 6202 scope.go:122] "RemoveContainer" containerID="4f1d17d26e02ce0fbdaee7a58a07eca9080878d956179084e83fbe8f9a03036f" Oct 30 17:00:20.733346 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.733301 6202 scope.go:122] "RemoveContainer" containerID="ea8884b55cebe30dc596e0566583a46ef5526e56107231126a175854bb0f661e" Oct 30 17:00:20.808083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.807249 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc"] Oct 30 17:00:20.816702 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.813982 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889660-l5wqc"] Oct 30 17:00:20.986820 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.986522 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr"] Oct 30 17:00:20.993535 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:20.993462 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r"] Oct 30 17:00:21.001755 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.001677 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e43400-29889660-r8b9r"] Oct 30 17:00:21.010380 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.010348 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47"] Oct 30 17:00:21.097374 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.097346 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" Oct 30 17:00:21.138471 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.138441 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" Oct 30 17:00:21.139802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.139656 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:21.151786 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.149163 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" Oct 30 17:00:21.153048 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.153022 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" Oct 30 17:00:21.161393 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.161364 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" Oct 30 17:00:21.171124 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.171081 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj"] Oct 30 17:00:21.178061 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.178023 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c75b279-d01e-43e8-9a3e-8dff3dae29c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c75b279-d01e-43e8-9a3e-8dff3dae29c7-maas-api-service-ca\") pod \"5c75b279-d01e-43e8-9a3e-8dff3dae29c7\" (UID: \"5c75b279-d01e-43e8-9a3e-8dff3dae29c7\") " Oct 30 17:00:21.178838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.178800 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c75b279-d01e-43e8-9a3e-8dff3dae29c7-kube-api-access-znkpz\" (UniqueName: \"kubernetes.io/projected/5c75b279-d01e-43e8-9a3e-8dff3dae29c7-kube-api-access-znkpz\") pod \"5c75b279-d01e-43e8-9a3e-8dff3dae29c7\" (UID: \"5c75b279-d01e-43e8-9a3e-8dff3dae29c7\") " Oct 30 17:00:21.179759 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.179289 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c75b279-d01e-43e8-9a3e-8dff3dae29c7-maas-api-service-ca" pod "5c75b279-d01e-43e8-9a3e-8dff3dae29c7" (UID: "5c75b279-d01e-43e8-9a3e-8dff3dae29c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:21.179759 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.179501 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889660-dlfhj"] Oct 30 17:00:21.181302 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.181278 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c75b279-d01e-43e8-9a3e-8dff3dae29c7-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.182564 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.182536 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c75b279-d01e-43e8-9a3e-8dff3dae29c7-kube-api-access-znkpz" pod "5c75b279-d01e-43e8-9a3e-8dff3dae29c7" (UID: "5c75b279-d01e-43e8-9a3e-8dff3dae29c7"). InnerVolumeSpecName "kube-api-access-znkpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:21.282631 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.282540 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75af19e6-9fae-4786-86d2-3ef7f4433f08-kube-api-access-whb9s\" (UniqueName: \"kubernetes.io/projected/75af19e6-9fae-4786-86d2-3ef7f4433f08-kube-api-access-whb9s\") pod \"75af19e6-9fae-4786-86d2-3ef7f4433f08\" (UID: \"75af19e6-9fae-4786-86d2-3ef7f4433f08\") " Oct 30 17:00:21.282993 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.282971 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660c4258-30bc-4a2f-b130-766dcbbd88f4-kube-api-access-lz7t2\" (UniqueName: \"kubernetes.io/projected/660c4258-30bc-4a2f-b130-766dcbbd88f4-kube-api-access-lz7t2\") pod \"660c4258-30bc-4a2f-b130-766dcbbd88f4\" (UID: \"660c4258-30bc-4a2f-b130-766dcbbd88f4\") " Oct 30 17:00:21.283235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.283219 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b2402f-79e8-4586-9f36-2b86f307f1cb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b2402f-79e8-4586-9f36-2b86f307f1cb-maas-api-service-ca\") pod \"c0b2402f-79e8-4586-9f36-2b86f307f1cb\" (UID: \"c0b2402f-79e8-4586-9f36-2b86f307f1cb\") " Oct 30 17:00:21.283446 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.283429 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c17f9002-4b15-4e46-a720-7a298c9c3f68-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17f9002-4b15-4e46-a720-7a298c9c3f68-maas-api-service-ca\") pod \"c17f9002-4b15-4e46-a720-7a298c9c3f68\" (UID: \"c17f9002-4b15-4e46-a720-7a298c9c3f68\") " Oct 30 17:00:21.284006 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.283986 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b2402f-79e8-4586-9f36-2b86f307f1cb-kube-api-access-bfgvs\" (UniqueName: \"kubernetes.io/projected/c0b2402f-79e8-4586-9f36-2b86f307f1cb-kube-api-access-bfgvs\") pod \"c0b2402f-79e8-4586-9f36-2b86f307f1cb\" (UID: \"c0b2402f-79e8-4586-9f36-2b86f307f1cb\") " Oct 30 17:00:21.284476 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.284457 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c17f9002-4b15-4e46-a720-7a298c9c3f68-kube-api-access-94v9q\" (UniqueName: \"kubernetes.io/projected/c17f9002-4b15-4e46-a720-7a298c9c3f68-kube-api-access-94v9q\") pod \"c17f9002-4b15-4e46-a720-7a298c9c3f68\" (UID: \"c17f9002-4b15-4e46-a720-7a298c9c3f68\") " Oct 30 17:00:21.284704 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.284690 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660c4258-30bc-4a2f-b130-766dcbbd88f4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660c4258-30bc-4a2f-b130-766dcbbd88f4-maas-api-service-ca\") pod \"660c4258-30bc-4a2f-b130-766dcbbd88f4\" (UID: \"660c4258-30bc-4a2f-b130-766dcbbd88f4\") " Oct 30 17:00:21.285305 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.283644 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b2402f-79e8-4586-9f36-2b86f307f1cb-maas-api-service-ca" pod "c0b2402f-79e8-4586-9f36-2b86f307f1cb" (UID: "c0b2402f-79e8-4586-9f36-2b86f307f1cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:21.285305 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.283892 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c17f9002-4b15-4e46-a720-7a298c9c3f68-maas-api-service-ca" pod "c17f9002-4b15-4e46-a720-7a298c9c3f68" (UID: "c17f9002-4b15-4e46-a720-7a298c9c3f68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:21.285305 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.285166 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660c4258-30bc-4a2f-b130-766dcbbd88f4-maas-api-service-ca" pod "660c4258-30bc-4a2f-b130-766dcbbd88f4" (UID: "660c4258-30bc-4a2f-b130-766dcbbd88f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:21.285605 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.285589 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75af19e6-9fae-4786-86d2-3ef7f4433f08-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75af19e6-9fae-4786-86d2-3ef7f4433f08-maas-api-service-ca\") pod \"75af19e6-9fae-4786-86d2-3ef7f4433f08\" (UID: \"75af19e6-9fae-4786-86d2-3ef7f4433f08\") " Oct 30 17:00:21.286137 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.286032 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75af19e6-9fae-4786-86d2-3ef7f4433f08-maas-api-service-ca" pod "75af19e6-9fae-4786-86d2-3ef7f4433f08" (UID: "75af19e6-9fae-4786-86d2-3ef7f4433f08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:21.286257 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.286177 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75af19e6-9fae-4786-86d2-3ef7f4433f08-kube-api-access-whb9s" pod "75af19e6-9fae-4786-86d2-3ef7f4433f08" (UID: "75af19e6-9fae-4786-86d2-3ef7f4433f08"). InnerVolumeSpecName "kube-api-access-whb9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:21.287194 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.287172 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whb9s\" (UniqueName: \"kubernetes.io/projected/75af19e6-9fae-4786-86d2-3ef7f4433f08-kube-api-access-whb9s\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.287455 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.287441 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b2402f-79e8-4586-9f36-2b86f307f1cb-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.287579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.287566 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17f9002-4b15-4e46-a720-7a298c9c3f68-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.287693 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.287682 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/660c4258-30bc-4a2f-b130-766dcbbd88f4-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.287819 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.287808 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75af19e6-9fae-4786-86d2-3ef7f4433f08-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.288003 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.287987 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znkpz\" (UniqueName: \"kubernetes.io/projected/5c75b279-d01e-43e8-9a3e-8dff3dae29c7-kube-api-access-znkpz\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.288122 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.287402 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b2402f-79e8-4586-9f36-2b86f307f1cb-kube-api-access-bfgvs" pod "c0b2402f-79e8-4586-9f36-2b86f307f1cb" (UID: "c0b2402f-79e8-4586-9f36-2b86f307f1cb"). InnerVolumeSpecName "kube-api-access-bfgvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:21.288241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.288192 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660c4258-30bc-4a2f-b130-766dcbbd88f4-kube-api-access-lz7t2" pod "660c4258-30bc-4a2f-b130-766dcbbd88f4" (UID: "660c4258-30bc-4a2f-b130-766dcbbd88f4"). InnerVolumeSpecName "kube-api-access-lz7t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:21.288485 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.288463 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c17f9002-4b15-4e46-a720-7a298c9c3f68-kube-api-access-94v9q" pod "c17f9002-4b15-4e46-a720-7a298c9c3f68" (UID: "c17f9002-4b15-4e46-a720-7a298c9c3f68"). InnerVolumeSpecName "kube-api-access-94v9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:21.308674 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.308623 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" podUID="242e1339-d85b-4360-99c9-6aab93ac561d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:00:21.400798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.399372 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz7t2\" (UniqueName: \"kubernetes.io/projected/660c4258-30bc-4a2f-b130-766dcbbd88f4-kube-api-access-lz7t2\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.400798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.399413 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfgvs\" (UniqueName: \"kubernetes.io/projected/c0b2402f-79e8-4586-9f36-2b86f307f1cb-kube-api-access-bfgvs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.400798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.399437 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94v9q\" (UniqueName: \"kubernetes.io/projected/c17f9002-4b15-4e46-a720-7a298c9c3f68-kube-api-access-94v9q\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.618141 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.618100 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" event={"ID":"75af19e6-9fae-4786-86d2-3ef7f4433f08","Type":"ContainerDied","Data":"c746797433518a53eb208665d8f16c411d366b2177f5d17cd9a28ae383df1498"} Oct 30 17:00:21.618804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.618748 6202 scope.go:122] "RemoveContainer" containerID="b32bf17af3915951b649e2580ace183fb3cc72866ad90a89d8acd2c0a0ea949e" Oct 30 17:00:21.618995 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.618149 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh" Oct 30 17:00:21.620879 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.620807 6202 generic.go:356] "Generic (PLEG): container finished" podID="b09a458f-a69b-441a-b91a-11854c180a4e" containerID="b49f500edf0a61d8328ca43821283e09e55a6aa1a642770154015a99694ef53f" exitCode=60 Oct 30 17:00:21.620966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.620916 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" event={"ID":"b09a458f-a69b-441a-b91a-11854c180a4e","Type":"ContainerDied","Data":"b49f500edf0a61d8328ca43821283e09e55a6aa1a642770154015a99694ef53f"} Oct 30 17:00:21.621448 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.621409 6202 scope.go:122] "RemoveContainer" containerID="b49f500edf0a61d8328ca43821283e09e55a6aa1a642770154015a99694ef53f" Oct 30 17:00:21.621901 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:21.621873 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n_odh-ai-gateway-infra(b09a458f-a69b-441a-b91a-11854c180a4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" podUID="b09a458f-a69b-441a-b91a-11854c180a4e" Oct 30 17:00:21.625632 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.625606 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" event={"ID":"c17f9002-4b15-4e46-a720-7a298c9c3f68","Type":"ContainerDied","Data":"2ad9d641a213a1ff614f4d5764ff1c3d7fa2706902fc049cd95d2419cd748205"} Oct 30 17:00:21.625820 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.625617 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k" Oct 30 17:00:21.638506 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.637747 6202 scope.go:122] "RemoveContainer" containerID="d3f58a5a36aed980ceb99e50148481fc32db504eac8cb7feb4cdfe371177486c" Oct 30 17:00:21.641422 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.640710 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" event={"ID":"c0b2402f-79e8-4586-9f36-2b86f307f1cb","Type":"ContainerDied","Data":"152546631ab13644f4e99445bc0c90d4246d7df181b39e1289b41804ee236b14"} Oct 30 17:00:21.641422 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.640728 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t" Oct 30 17:00:21.648727 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.648143 6202 generic.go:356] "Generic (PLEG): container finished" podID="bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5" containerID="f7728e6df630a377df6924ee6f5ce492a82ff4929a48dfad0ae1388b09955723" exitCode=60 Oct 30 17:00:21.648727 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.648345 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" event={"ID":"bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5","Type":"ContainerDied","Data":"f7728e6df630a377df6924ee6f5ce492a82ff4929a48dfad0ae1388b09955723"} Oct 30 17:00:21.649586 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.649471 6202 scope.go:122] "RemoveContainer" containerID="f7728e6df630a377df6924ee6f5ce492a82ff4929a48dfad0ae1388b09955723" Oct 30 17:00:21.650427 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:21.650308 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz_odh-ai-gateway-infra(bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" podUID="bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5" Oct 30 17:00:21.655209 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.655180 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" event={"ID":"660c4258-30bc-4a2f-b130-766dcbbd88f4","Type":"ContainerDied","Data":"117fad5af6d36c80ddddaf7b8e2360b02ca86276ff9824181a5cb8174a1fbf49"} Oct 30 17:00:21.655668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.655418 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4" Oct 30 17:00:21.658692 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.658671 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" event={"ID":"7aadca90-6dbe-4fe5-9dee-10431e9210a0","Type":"ContainerStarted","Data":"9cfcafa909cc52c2bbeb412774e28b4c6b1861f71938a8a6b0d75bd9c067bd78"} Oct 30 17:00:21.664807 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.664095 6202 generic.go:356] "Generic (PLEG): container finished" podID="f9872bb2-dae7-43b4-9a03-6d0c9804cb79" containerID="2f0aacf433faaca88ad916a6ee53bb77927728dd730838c4b3dbcda24e827f40" exitCode=60 Oct 30 17:00:21.664807 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.664253 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp" event={"ID":"f9872bb2-dae7-43b4-9a03-6d0c9804cb79","Type":"ContainerDied","Data":"2f0aacf433faaca88ad916a6ee53bb77927728dd730838c4b3dbcda24e827f40"} Oct 30 17:00:21.664807 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.664691 6202 scope.go:122] "RemoveContainer" containerID="2f0aacf433faaca88ad916a6ee53bb77927728dd730838c4b3dbcda24e827f40" Oct 30 17:00:21.665808 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:21.665729 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp_odh-ai-gateway-infra(f9872bb2-dae7-43b4-9a03-6d0c9804cb79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp" podUID="f9872bb2-dae7-43b4-9a03-6d0c9804cb79" Oct 30 17:00:21.668848 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.667292 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" event={"ID":"5c75b279-d01e-43e8-9a3e-8dff3dae29c7","Type":"ContainerDied","Data":"ed29b314303a5fb6305f4bbdd68b36bb896aeafa43c346666d2736808beb907e"} Oct 30 17:00:21.668848 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.667396 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh" Oct 30 17:00:21.759238 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.759198 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz"] Oct 30 17:00:21.786376 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.784481 6202 scope.go:122] "RemoveContainer" containerID="0b9e050e5d155bbdbcab2c832001f640f9d90e8acb36e56cbda1b7114cc4a5f6" Oct 30 17:00:21.796710 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.796678 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh"] Oct 30 17:00:21.803822 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.803712 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889660-tmwqh"] Oct 30 17:00:21.807301 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.804218 6202 scope.go:122] "RemoveContainer" containerID="218af457aac26e4028572ac654bff9abf34b8a95160a125dda5b2b279d1e3dd7" Oct 30 17:00:21.828219 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.828190 6202 scope.go:122] "RemoveContainer" containerID="4fbfc7d452355aee1a469bade80d249260ef0961687867612677eb4750b5584f" Oct 30 17:00:21.832730 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.832700 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh"] Oct 30 17:00:21.844027 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.843993 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-967fac-29889660-lstlh"] Oct 30 17:00:21.884616 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.883946 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t"] Oct 30 17:00:21.893150 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.893109 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889660-kpt9t"] Oct 30 17:00:21.899639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.899517 6202 scope.go:122] "RemoveContainer" containerID="cd09df53d3bf9d7804d0f94c2a4af6e3d7b9a246b3bd2daba7a79646d36a9085" Oct 30 17:00:21.907363 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.907335 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" Oct 30 17:00:21.916845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.916820 6202 scope.go:122] "RemoveContainer" containerID="1b7cac64d71077169b81fc41df654c1870c84e6335d484a1078041354ff0c28b" Oct 30 17:00:21.920319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.920286 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" Oct 30 17:00:21.932634 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.932612 6202 scope.go:122] "RemoveContainer" containerID="ba56572ce8cb11a1ea9bfdbd3fd7d24122b8884075603c4ef87fb96247768c1a" Oct 30 17:00:21.956232 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.956190 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k"] Oct 30 17:00:21.966146 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:21.966111 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e5bfb0-29889660-mmc6k"] Oct 30 17:00:22.008759 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.008720 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7550e3d-3bc1-4ff5-a641-0d68b4bf6127-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7550e3d-3bc1-4ff5-a641-0d68b4bf6127-maas-api-service-ca\") pod \"c7550e3d-3bc1-4ff5-a641-0d68b4bf6127\" (UID: \"c7550e3d-3bc1-4ff5-a641-0d68b4bf6127\") " Oct 30 17:00:22.008973 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.008917 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7550e3d-3bc1-4ff5-a641-0d68b4bf6127-kube-api-access-p78m5\" (UniqueName: \"kubernetes.io/projected/c7550e3d-3bc1-4ff5-a641-0d68b4bf6127-kube-api-access-p78m5\") pod \"c7550e3d-3bc1-4ff5-a641-0d68b4bf6127\" (UID: \"c7550e3d-3bc1-4ff5-a641-0d68b4bf6127\") " Oct 30 17:00:22.009109 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.009085 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7550e3d-3bc1-4ff5-a641-0d68b4bf6127-maas-api-service-ca" pod "c7550e3d-3bc1-4ff5-a641-0d68b4bf6127" (UID: "c7550e3d-3bc1-4ff5-a641-0d68b4bf6127"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:22.009145 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.009124 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d338cc2-8d69-4b14-99d1-370710f8cb99-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d338cc2-8d69-4b14-99d1-370710f8cb99-maas-api-service-ca\") pod \"3d338cc2-8d69-4b14-99d1-370710f8cb99\" (UID: \"3d338cc2-8d69-4b14-99d1-370710f8cb99\") " Oct 30 17:00:22.009219 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.009205 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d338cc2-8d69-4b14-99d1-370710f8cb99-kube-api-access-2d8bm\" (UniqueName: \"kubernetes.io/projected/3d338cc2-8d69-4b14-99d1-370710f8cb99-kube-api-access-2d8bm\") pod \"3d338cc2-8d69-4b14-99d1-370710f8cb99\" (UID: \"3d338cc2-8d69-4b14-99d1-370710f8cb99\") " Oct 30 17:00:22.009472 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.009443 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d338cc2-8d69-4b14-99d1-370710f8cb99-maas-api-service-ca" pod "3d338cc2-8d69-4b14-99d1-370710f8cb99" (UID: "3d338cc2-8d69-4b14-99d1-370710f8cb99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:22.009897 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.009876 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d338cc2-8d69-4b14-99d1-370710f8cb99-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:22.009897 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.009895 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7550e3d-3bc1-4ff5-a641-0d68b4bf6127-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:22.011253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.011222 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7550e3d-3bc1-4ff5-a641-0d68b4bf6127-kube-api-access-p78m5" pod "c7550e3d-3bc1-4ff5-a641-0d68b4bf6127" (UID: "c7550e3d-3bc1-4ff5-a641-0d68b4bf6127"). InnerVolumeSpecName "kube-api-access-p78m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:22.011424 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.011405 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d338cc2-8d69-4b14-99d1-370710f8cb99-kube-api-access-2d8bm" pod "3d338cc2-8d69-4b14-99d1-370710f8cb99" (UID: "3d338cc2-8d69-4b14-99d1-370710f8cb99"). InnerVolumeSpecName "kube-api-access-2d8bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:22.111496 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.111456 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p78m5\" (UniqueName: \"kubernetes.io/projected/c7550e3d-3bc1-4ff5-a641-0d68b4bf6127-kube-api-access-p78m5\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:22.111496 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.111498 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d8bm\" (UniqueName: \"kubernetes.io/projected/3d338cc2-8d69-4b14-99d1-370710f8cb99-kube-api-access-2d8bm\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:22.114150 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.114120 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n"] Oct 30 17:00:22.144212 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.144168 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4"] Oct 30 17:00:22.150874 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.150842 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-830ba9-29889660-c6tg4"] Oct 30 17:00:22.414124 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.414046 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0015d61c-9d93-4652-9414-3b817ab96276" path="/var/lib/kubelet/pods/0015d61c-9d93-4652-9414-3b817ab96276/volumes" Oct 30 17:00:22.414537 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.414522 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba09798-de28-4d76-8fea-839bbd8ea7e4" path="/var/lib/kubelet/pods/5ba09798-de28-4d76-8fea-839bbd8ea7e4/volumes" Oct 30 17:00:22.414957 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.414942 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c75b279-d01e-43e8-9a3e-8dff3dae29c7" path="/var/lib/kubelet/pods/5c75b279-d01e-43e8-9a3e-8dff3dae29c7/volumes" Oct 30 17:00:22.415639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.415623 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660c4258-30bc-4a2f-b130-766dcbbd88f4" path="/var/lib/kubelet/pods/660c4258-30bc-4a2f-b130-766dcbbd88f4/volumes" Oct 30 17:00:22.416124 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.416106 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6957d4c0-8197-4aab-a89a-ad17414cd011" path="/var/lib/kubelet/pods/6957d4c0-8197-4aab-a89a-ad17414cd011/volumes" Oct 30 17:00:22.416648 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.416632 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75af19e6-9fae-4786-86d2-3ef7f4433f08" path="/var/lib/kubelet/pods/75af19e6-9fae-4786-86d2-3ef7f4433f08/volumes" Oct 30 17:00:22.417473 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.417457 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c799e0b-668f-4c55-a254-69651dd5b005" path="/var/lib/kubelet/pods/8c799e0b-668f-4c55-a254-69651dd5b005/volumes" Oct 30 17:00:22.417883 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.417869 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b2402f-79e8-4586-9f36-2b86f307f1cb" path="/var/lib/kubelet/pods/c0b2402f-79e8-4586-9f36-2b86f307f1cb/volumes" Oct 30 17:00:22.418272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.418257 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c17f9002-4b15-4e46-a720-7a298c9c3f68" path="/var/lib/kubelet/pods/c17f9002-4b15-4e46-a720-7a298c9c3f68/volumes" Oct 30 17:00:22.686652 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.686557 6202 generic.go:356] "Generic (PLEG): container finished" podID="7aadca90-6dbe-4fe5-9dee-10431e9210a0" containerID="9cfcafa909cc52c2bbeb412774e28b4c6b1861f71938a8a6b0d75bd9c067bd78" exitCode=60 Oct 30 17:00:22.686652 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.686633 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" event={"ID":"7aadca90-6dbe-4fe5-9dee-10431e9210a0","Type":"ContainerDied","Data":"9cfcafa909cc52c2bbeb412774e28b4c6b1861f71938a8a6b0d75bd9c067bd78"} Oct 30 17:00:22.687208 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.686699 6202 scope.go:122] "RemoveContainer" containerID="4c3473df9be92d62380d4ed6bf05c7aa17c78370916ae619ed19e2b4f34ddc83" Oct 30 17:00:22.687208 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.687179 6202 scope.go:122] "RemoveContainer" containerID="9cfcafa909cc52c2bbeb412774e28b4c6b1861f71938a8a6b0d75bd9c067bd78" Oct 30 17:00:22.687595 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:22.687571 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v_odh-ai-gateway-infra(7aadca90-6dbe-4fe5-9dee-10431e9210a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" podUID="7aadca90-6dbe-4fe5-9dee-10431e9210a0" Oct 30 17:00:22.695070 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.695045 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" Oct 30 17:00:22.695177 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.695111 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47" event={"ID":"3d338cc2-8d69-4b14-99d1-370710f8cb99","Type":"ContainerDied","Data":"ebd0c2785abe43b3867bdc641517a8543655965cfd165bbb97797e9f7f6a5a1e"} Oct 30 17:00:22.696853 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.696825 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" event={"ID":"c7550e3d-3bc1-4ff5-a641-0d68b4bf6127","Type":"ContainerDied","Data":"4f15ddb65677933cc012f9cc903c878b495bf44ab900b890f48d4d72d70fdc73"} Oct 30 17:00:22.696999 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.696929 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr" Oct 30 17:00:22.708054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.708008 6202 scope.go:122] "RemoveContainer" containerID="1554ffaef1464ad6b562565d6ffbc579d43a8c924f5cd211877574cc9ab3ad85" Oct 30 17:00:22.720938 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.720914 6202 scope.go:122] "RemoveContainer" containerID="14f9389e04aec67275ac7127a097af1551d9e2dd54da95353372974f0d3c9c7d" Oct 30 17:00:22.770713 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.770661 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr"] Oct 30 17:00:22.779493 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.779085 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889660-7zhgr"] Oct 30 17:00:22.795601 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.795415 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:22.935247 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.935214 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47"] Oct 30 17:00:22.948157 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.947999 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889660-fcp47"] Oct 30 17:00:22.972808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:22.972752 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v"] Oct 30 17:00:23.044838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.043438 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" Oct 30 17:00:23.064513 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.064471 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp"] Oct 30 17:00:23.099067 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.096922 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" Oct 30 17:00:23.137298 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.137260 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b09a458f-a69b-441a-b91a-11854c180a4e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09a458f-a69b-441a-b91a-11854c180a4e-maas-api-service-ca\") pod \"b09a458f-a69b-441a-b91a-11854c180a4e\" (UID: \"b09a458f-a69b-441a-b91a-11854c180a4e\") " Oct 30 17:00:23.137524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.137505 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b09a458f-a69b-441a-b91a-11854c180a4e-kube-api-access-ffcx4\" (UniqueName: \"kubernetes.io/projected/b09a458f-a69b-441a-b91a-11854c180a4e-kube-api-access-ffcx4\") pod \"b09a458f-a69b-441a-b91a-11854c180a4e\" (UID: \"b09a458f-a69b-441a-b91a-11854c180a4e\") " Oct 30 17:00:23.154308 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.153081 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b09a458f-a69b-441a-b91a-11854c180a4e-maas-api-service-ca" pod "b09a458f-a69b-441a-b91a-11854c180a4e" (UID: "b09a458f-a69b-441a-b91a-11854c180a4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:23.159647 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.159608 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b09a458f-a69b-441a-b91a-11854c180a4e-kube-api-access-ffcx4" pod "b09a458f-a69b-441a-b91a-11854c180a4e" (UID: "b09a458f-a69b-441a-b91a-11854c180a4e"). InnerVolumeSpecName "kube-api-access-ffcx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:23.262943 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.262851 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5-kube-api-access-z7d7d\" (UniqueName: \"kubernetes.io/projected/bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5-kube-api-access-z7d7d\") pod \"bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5\" (UID: \"bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5\") " Oct 30 17:00:23.263139 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.263072 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5-maas-api-service-ca\") pod \"bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5\" (UID: \"bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5\") " Oct 30 17:00:23.264149 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.264118 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09a458f-a69b-441a-b91a-11854c180a4e-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:23.264278 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.264159 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffcx4\" (UniqueName: \"kubernetes.io/projected/b09a458f-a69b-441a-b91a-11854c180a4e-kube-api-access-ffcx4\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:23.266071 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.266028 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5-maas-api-service-ca" pod "bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5" (UID: "bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:23.268998 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.268955 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5-kube-api-access-z7d7d" pod "bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5" (UID: "bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5"). InnerVolumeSpecName "kube-api-access-z7d7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:23.286518 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.286485 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp" Oct 30 17:00:23.367447 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.365305 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9872bb2-dae7-43b4-9a03-6d0c9804cb79-kube-api-access-wb4nr\" (UniqueName: \"kubernetes.io/projected/f9872bb2-dae7-43b4-9a03-6d0c9804cb79-kube-api-access-wb4nr\") pod \"f9872bb2-dae7-43b4-9a03-6d0c9804cb79\" (UID: \"f9872bb2-dae7-43b4-9a03-6d0c9804cb79\") " Oct 30 17:00:23.367447 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.365578 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9872bb2-dae7-43b4-9a03-6d0c9804cb79-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9872bb2-dae7-43b4-9a03-6d0c9804cb79-maas-api-service-ca\") pod \"f9872bb2-dae7-43b4-9a03-6d0c9804cb79\" (UID: \"f9872bb2-dae7-43b4-9a03-6d0c9804cb79\") " Oct 30 17:00:23.367447 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.366120 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9872bb2-dae7-43b4-9a03-6d0c9804cb79-maas-api-service-ca" pod "f9872bb2-dae7-43b4-9a03-6d0c9804cb79" (UID: "f9872bb2-dae7-43b4-9a03-6d0c9804cb79"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:23.367447 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.366361 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7d7d\" (UniqueName: \"kubernetes.io/projected/bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5-kube-api-access-z7d7d\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:23.367447 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.366385 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:23.367447 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.366400 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9872bb2-dae7-43b4-9a03-6d0c9804cb79-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:23.369197 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.368891 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9872bb2-dae7-43b4-9a03-6d0c9804cb79-kube-api-access-wb4nr" pod "f9872bb2-dae7-43b4-9a03-6d0c9804cb79" (UID: "f9872bb2-dae7-43b4-9a03-6d0c9804cb79"). InnerVolumeSpecName "kube-api-access-wb4nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:23.468217 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.468173 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb4nr\" (UniqueName: \"kubernetes.io/projected/f9872bb2-dae7-43b4-9a03-6d0c9804cb79-kube-api-access-wb4nr\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:23.712306 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.712271 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp" Oct 30 17:00:23.712850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.712322 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp" event={"ID":"f9872bb2-dae7-43b4-9a03-6d0c9804cb79","Type":"ContainerDied","Data":"211eb779680b463ddef24e357fbbfbc647f0e1d6ea4b7f124bc3c45ef3ca7331"} Oct 30 17:00:23.712850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.712395 6202 scope.go:122] "RemoveContainer" containerID="2f0aacf433faaca88ad916a6ee53bb77927728dd730838c4b3dbcda24e827f40" Oct 30 17:00:23.714189 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.714165 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" Oct 30 17:00:23.714342 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.714165 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n" event={"ID":"b09a458f-a69b-441a-b91a-11854c180a4e","Type":"ContainerDied","Data":"7ff08e8f6ef2584961eda65c68b17a3054e9bcd6eb8344cfcc99ebeaea9be883"} Oct 30 17:00:23.723988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.723952 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" event={"ID":"bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5","Type":"ContainerDied","Data":"a4a701f831c04185b540e4465eb181693e4f47581d4435eb3c422810a4b1a5cb"} Oct 30 17:00:23.724127 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.724038 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz" Oct 30 17:00:23.736505 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.732840 6202 scope.go:122] "RemoveContainer" containerID="b49f500edf0a61d8328ca43821283e09e55a6aa1a642770154015a99694ef53f" Oct 30 17:00:23.756571 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.756540 6202 scope.go:122] "RemoveContainer" containerID="f7728e6df630a377df6924ee6f5ce492a82ff4929a48dfad0ae1388b09955723" Oct 30 17:00:23.800009 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.799982 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp"] Oct 30 17:00:23.805182 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.805149 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889660-sfrbp"] Oct 30 17:00:23.897218 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.897178 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n"] Oct 30 17:00:23.906074 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.906041 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ea77d8-29889660-fkd4n"] Oct 30 17:00:23.927756 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.927724 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz"] Oct 30 17:00:23.932114 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.932083 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-bfecdd-29889660-dvzcz"] Oct 30 17:00:23.942943 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:23.942911 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" Oct 30 17:00:24.085081 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:24.085045 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aadca90-6dbe-4fe5-9dee-10431e9210a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aadca90-6dbe-4fe5-9dee-10431e9210a0-maas-api-service-ca\") pod \"7aadca90-6dbe-4fe5-9dee-10431e9210a0\" (UID: \"7aadca90-6dbe-4fe5-9dee-10431e9210a0\") " Oct 30 17:00:24.085360 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:24.085339 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aadca90-6dbe-4fe5-9dee-10431e9210a0-kube-api-access-kxbw7\" (UniqueName: \"kubernetes.io/projected/7aadca90-6dbe-4fe5-9dee-10431e9210a0-kube-api-access-kxbw7\") pod \"7aadca90-6dbe-4fe5-9dee-10431e9210a0\" (UID: \"7aadca90-6dbe-4fe5-9dee-10431e9210a0\") " Oct 30 17:00:24.085445 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:24.085405 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aadca90-6dbe-4fe5-9dee-10431e9210a0-maas-api-service-ca" pod "7aadca90-6dbe-4fe5-9dee-10431e9210a0" (UID: "7aadca90-6dbe-4fe5-9dee-10431e9210a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:24.086247 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:24.086221 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aadca90-6dbe-4fe5-9dee-10431e9210a0-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:24.087827 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:24.087739 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aadca90-6dbe-4fe5-9dee-10431e9210a0-kube-api-access-kxbw7" pod "7aadca90-6dbe-4fe5-9dee-10431e9210a0" (UID: "7aadca90-6dbe-4fe5-9dee-10431e9210a0"). InnerVolumeSpecName "kube-api-access-kxbw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:24.188243 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:24.188205 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxbw7\" (UniqueName: \"kubernetes.io/projected/7aadca90-6dbe-4fe5-9dee-10431e9210a0-kube-api-access-kxbw7\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:24.413082 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:24.412992 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d338cc2-8d69-4b14-99d1-370710f8cb99" path="/var/lib/kubelet/pods/3d338cc2-8d69-4b14-99d1-370710f8cb99/volumes" Oct 30 17:00:24.413588 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:24.413565 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b09a458f-a69b-441a-b91a-11854c180a4e" path="/var/lib/kubelet/pods/b09a458f-a69b-441a-b91a-11854c180a4e/volumes" Oct 30 17:00:24.414091 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:24.414071 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5" path="/var/lib/kubelet/pods/bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5/volumes" Oct 30 17:00:24.414876 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:24.414857 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7550e3d-3bc1-4ff5-a641-0d68b4bf6127" path="/var/lib/kubelet/pods/c7550e3d-3bc1-4ff5-a641-0d68b4bf6127/volumes" Oct 30 17:00:24.415368 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:24.415348 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9872bb2-dae7-43b4-9a03-6d0c9804cb79" path="/var/lib/kubelet/pods/f9872bb2-dae7-43b4-9a03-6d0c9804cb79/volumes" Oct 30 17:00:24.419264 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:24.419243 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:24.694313 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:24.694231 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" podUID="3ddb2df8-3531-4d65-a0fe-7fe1facc43ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:00:24.741125 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:24.741091 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" Oct 30 17:00:24.741624 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:24.741097 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v" event={"ID":"7aadca90-6dbe-4fe5-9dee-10431e9210a0","Type":"ContainerDied","Data":"55fe104d78099de8d226ce1572bf55e0aec8169a9dffe29931206b866fcfe2f1"} Oct 30 17:00:24.741624 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:24.741230 6202 scope.go:122] "RemoveContainer" containerID="9cfcafa909cc52c2bbeb412774e28b4c6b1861f71938a8a6b0d75bd9c067bd78" Oct 30 17:00:24.798481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:24.798406 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v"] Oct 30 17:00:24.803124 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:24.803097 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889660-8gh5v"] Oct 30 17:00:25.645975 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:25.645935 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw"] Oct 30 17:00:25.793506 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:25.793464 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:25.907967 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:25.906795 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:26.275019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:26.274922 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:26.457352 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:26.457011 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aadca90-6dbe-4fe5-9dee-10431e9210a0" path="/var/lib/kubelet/pods/7aadca90-6dbe-4fe5-9dee-10431e9210a0/volumes" Oct 30 17:00:27.030374 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.030338 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk"] Oct 30 17:00:27.031183 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031165 6202 cpu_manager.go:414] "containerMap: removing container" podUID="f9872bb2-dae7-43b4-9a03-6d0c9804cb79" containerName="cleanup" Oct 30 17:00:27.031183 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031183 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9872bb2-dae7-43b4-9a03-6d0c9804cb79" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031192 6202 cpu_manager.go:414] "containerMap: removing container" podUID="6957d4c0-8197-4aab-a89a-ad17414cd011" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031198 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6957d4c0-8197-4aab-a89a-ad17414cd011" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031206 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c17f9002-4b15-4e46-a720-7a298c9c3f68" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031211 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c17f9002-4b15-4e46-a720-7a298c9c3f68" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031217 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c0b2402f-79e8-4586-9f36-2b86f307f1cb" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031222 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b2402f-79e8-4586-9f36-2b86f307f1cb" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031230 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5c75b279-d01e-43e8-9a3e-8dff3dae29c7" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031236 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c75b279-d01e-43e8-9a3e-8dff3dae29c7" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031244 6202 cpu_manager.go:414] "containerMap: removing container" podUID="ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031248 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031260 6202 cpu_manager.go:414] "containerMap: removing container" podUID="bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031264 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031272 6202 cpu_manager.go:414] "containerMap: removing container" podUID="660c4258-30bc-4a2f-b130-766dcbbd88f4" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031276 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660c4258-30bc-4a2f-b130-766dcbbd88f4" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031281 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5ba09798-de28-4d76-8fea-839bbd8ea7e4" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031286 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba09798-de28-4d76-8fea-839bbd8ea7e4" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031292 6202 cpu_manager.go:414] "containerMap: removing container" podUID="75af19e6-9fae-4786-86d2-3ef7f4433f08" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031297 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75af19e6-9fae-4786-86d2-3ef7f4433f08" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031304 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c17f9002-4b15-4e46-a720-7a298c9c3f68" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031308 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c17f9002-4b15-4e46-a720-7a298c9c3f68" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031315 6202 cpu_manager.go:414] "containerMap: removing container" podUID="0015d61c-9d93-4652-9414-3b817ab96276" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031319 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0015d61c-9d93-4652-9414-3b817ab96276" containerName="cleanup" Oct 30 17:00:27.031315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031324 6202 cpu_manager.go:414] "containerMap: removing container" podUID="75af19e6-9fae-4786-86d2-3ef7f4433f08" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031329 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75af19e6-9fae-4786-86d2-3ef7f4433f08" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031334 6202 cpu_manager.go:414] "containerMap: removing container" podUID="660c4258-30bc-4a2f-b130-766dcbbd88f4" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031338 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660c4258-30bc-4a2f-b130-766dcbbd88f4" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031347 6202 cpu_manager.go:414] "containerMap: removing container" podUID="0015d61c-9d93-4652-9414-3b817ab96276" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031352 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0015d61c-9d93-4652-9414-3b817ab96276" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031359 6202 cpu_manager.go:414] "containerMap: removing container" podUID="b09a458f-a69b-441a-b91a-11854c180a4e" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031364 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b09a458f-a69b-441a-b91a-11854c180a4e" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031369 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3d338cc2-8d69-4b14-99d1-370710f8cb99" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031373 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d338cc2-8d69-4b14-99d1-370710f8cb99" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031379 6202 cpu_manager.go:414] "containerMap: removing container" podUID="b09a458f-a69b-441a-b91a-11854c180a4e" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031383 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b09a458f-a69b-441a-b91a-11854c180a4e" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031390 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3d338cc2-8d69-4b14-99d1-370710f8cb99" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031395 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d338cc2-8d69-4b14-99d1-370710f8cb99" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031404 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c7550e3d-3bc1-4ff5-a641-0d68b4bf6127" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031408 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7550e3d-3bc1-4ff5-a641-0d68b4bf6127" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031421 6202 cpu_manager.go:414] "containerMap: removing container" podUID="bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031425 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031431 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5c75b279-d01e-43e8-9a3e-8dff3dae29c7" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031434 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c75b279-d01e-43e8-9a3e-8dff3dae29c7" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031442 6202 cpu_manager.go:414] "containerMap: removing container" podUID="7aadca90-6dbe-4fe5-9dee-10431e9210a0" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031446 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aadca90-6dbe-4fe5-9dee-10431e9210a0" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031453 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5ba09798-de28-4d76-8fea-839bbd8ea7e4" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031457 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba09798-de28-4d76-8fea-839bbd8ea7e4" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031464 6202 cpu_manager.go:414] "containerMap: removing container" podUID="bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031468 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031475 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c7550e3d-3bc1-4ff5-a641-0d68b4bf6127" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031479 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7550e3d-3bc1-4ff5-a641-0d68b4bf6127" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031489 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5ba09798-de28-4d76-8fea-839bbd8ea7e4" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031494 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba09798-de28-4d76-8fea-839bbd8ea7e4" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031503 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8c799e0b-668f-4c55-a254-69651dd5b005" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031507 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c799e0b-668f-4c55-a254-69651dd5b005" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031514 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8c799e0b-668f-4c55-a254-69651dd5b005" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031518 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c799e0b-668f-4c55-a254-69651dd5b005" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031525 6202 cpu_manager.go:414] "containerMap: removing container" podUID="6957d4c0-8197-4aab-a89a-ad17414cd011" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031529 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6957d4c0-8197-4aab-a89a-ad17414cd011" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031535 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8c799e0b-668f-4c55-a254-69651dd5b005" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031540 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c799e0b-668f-4c55-a254-69651dd5b005" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031548 6202 cpu_manager.go:414] "containerMap: removing container" podUID="7aadca90-6dbe-4fe5-9dee-10431e9210a0" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031552 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aadca90-6dbe-4fe5-9dee-10431e9210a0" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031560 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c7550e3d-3bc1-4ff5-a641-0d68b4bf6127" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031564 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7550e3d-3bc1-4ff5-a641-0d68b4bf6127" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031569 6202 cpu_manager.go:414] "containerMap: removing container" podUID="f9872bb2-dae7-43b4-9a03-6d0c9804cb79" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031573 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9872bb2-dae7-43b4-9a03-6d0c9804cb79" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031579 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9b6497d9-6147-4773-bc38-5e0f6bf7eb2f" containerName="revoke-keys" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031583 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b6497d9-6147-4773-bc38-5e0f6bf7eb2f" containerName="revoke-keys" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031598 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3d338cc2-8d69-4b14-99d1-370710f8cb99" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031603 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d338cc2-8d69-4b14-99d1-370710f8cb99" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031610 6202 cpu_manager.go:414] "containerMap: removing container" podUID="ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031614 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031622 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c0b2402f-79e8-4586-9f36-2b86f307f1cb" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031627 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b2402f-79e8-4586-9f36-2b86f307f1cb" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031723 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aadca90-6dbe-4fe5-9dee-10431e9210a0" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031732 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031739 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c799e0b-668f-4c55-a254-69651dd5b005" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031746 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c799e0b-668f-4c55-a254-69651dd5b005" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031752 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="6957d4c0-8197-4aab-a89a-ad17414cd011" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031757 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c799e0b-668f-4c55-a254-69651dd5b005" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031779 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aadca90-6dbe-4fe5-9dee-10431e9210a0" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031787 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7550e3d-3bc1-4ff5-a641-0d68b4bf6127" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031793 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9872bb2-dae7-43b4-9a03-6d0c9804cb79" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031800 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b6497d9-6147-4773-bc38-5e0f6bf7eb2f" containerName="revoke-keys" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031808 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c75b279-d01e-43e8-9a3e-8dff3dae29c7" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031814 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="660c4258-30bc-4a2f-b130-766dcbbd88f4" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031819 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d338cc2-8d69-4b14-99d1-370710f8cb99" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031824 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="0015d61c-9d93-4652-9414-3b817ab96276" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031830 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031834 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9872bb2-dae7-43b4-9a03-6d0c9804cb79" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031841 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b2402f-79e8-4586-9f36-2b86f307f1cb" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031846 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="b09a458f-a69b-441a-b91a-11854c180a4e" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031851 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9872bb2-dae7-43b4-9a03-6d0c9804cb79" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031856 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="6957d4c0-8197-4aab-a89a-ad17414cd011" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031863 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c17f9002-4b15-4e46-a720-7a298c9c3f68" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031867 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b2402f-79e8-4586-9f36-2b86f307f1cb" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031873 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c75b279-d01e-43e8-9a3e-8dff3dae29c7" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031878 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031883 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="660c4258-30bc-4a2f-b130-766dcbbd88f4" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031894 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba09798-de28-4d76-8fea-839bbd8ea7e4" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031899 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="75af19e6-9fae-4786-86d2-3ef7f4433f08" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031905 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c17f9002-4b15-4e46-a720-7a298c9c3f68" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031911 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="75af19e6-9fae-4786-86d2-3ef7f4433f08" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031918 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="0015d61c-9d93-4652-9414-3b817ab96276" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031923 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="660c4258-30bc-4a2f-b130-766dcbbd88f4" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031930 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="0015d61c-9d93-4652-9414-3b817ab96276" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031935 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d338cc2-8d69-4b14-99d1-370710f8cb99" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031939 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="b09a458f-a69b-441a-b91a-11854c180a4e" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031944 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d338cc2-8d69-4b14-99d1-370710f8cb99" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031953 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba09798-de28-4d76-8fea-839bbd8ea7e4" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031958 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031963 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7550e3d-3bc1-4ff5-a641-0d68b4bf6127" containerName="cleanup" Oct 30 17:00:27.032031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.031969 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba09798-de28-4d76-8fea-839bbd8ea7e4" containerName="cleanup" Oct 30 17:00:27.037486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.037466 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk" Oct 30 17:00:27.058816 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.056509 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk"] Oct 30 17:00:27.061180 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.061144 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:00:27.061180 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.061161 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:00:27.073813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.070542 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp2kt\" (UniqueName: \"kubernetes.io/projected/c680bca6-5420-4bcc-8ef5-5bfecb85218e-kube-api-access-rp2kt\") pod \"maas-api-revoke-sub-795ac4c1-95jmk\" (UID: \"c680bca6-5420-4bcc-8ef5-5bfecb85218e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk" Oct 30 17:00:27.073813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.070715 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c680bca6-5420-4bcc-8ef5-5bfecb85218e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-795ac4c1-95jmk\" (UID: \"c680bca6-5420-4bcc-8ef5-5bfecb85218e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk" Oct 30 17:00:27.073813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.070864 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c680bca6-5420-4bcc-8ef5-5bfecb85218e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-795ac4c1-95jmk\" (UID: \"c680bca6-5420-4bcc-8ef5-5bfecb85218e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk" Oct 30 17:00:27.172955 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.172894 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp2kt\" (UniqueName: \"kubernetes.io/projected/c680bca6-5420-4bcc-8ef5-5bfecb85218e-kube-api-access-rp2kt\") pod \"maas-api-revoke-sub-795ac4c1-95jmk\" (UID: \"c680bca6-5420-4bcc-8ef5-5bfecb85218e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk" Oct 30 17:00:27.173188 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.173066 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c680bca6-5420-4bcc-8ef5-5bfecb85218e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-795ac4c1-95jmk\" (UID: \"c680bca6-5420-4bcc-8ef5-5bfecb85218e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk" Oct 30 17:00:27.173188 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.173171 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c680bca6-5420-4bcc-8ef5-5bfecb85218e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-795ac4c1-95jmk\" (UID: \"c680bca6-5420-4bcc-8ef5-5bfecb85218e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk" Oct 30 17:00:27.173733 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.173698 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c680bca6-5420-4bcc-8ef5-5bfecb85218e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-795ac4c1-95jmk\" (UID: \"c680bca6-5420-4bcc-8ef5-5bfecb85218e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk" Oct 30 17:00:27.175802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.175759 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c680bca6-5420-4bcc-8ef5-5bfecb85218e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-795ac4c1-95jmk\" (UID: \"c680bca6-5420-4bcc-8ef5-5bfecb85218e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk" Oct 30 17:00:27.196482 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.196451 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp2kt\" (UniqueName: \"kubernetes.io/projected/c680bca6-5420-4bcc-8ef5-5bfecb85218e-kube-api-access-rp2kt\") pod \"maas-api-revoke-sub-795ac4c1-95jmk\" (UID: \"c680bca6-5420-4bcc-8ef5-5bfecb85218e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk" Oct 30 17:00:27.357537 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.357501 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk" Oct 30 17:00:27.670196 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:00:27.670148 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc680bca6_5420_4bcc_8ef5_5bfecb85218e.slice/crio-8c8ba3ebda6254017c5736a764c8f3b8d023fc73df7a8642bf740c595a9ec8d8 WatchSource:0}: Error finding container 8c8ba3ebda6254017c5736a764c8f3b8d023fc73df7a8642bf740c595a9ec8d8: Status 404 returned error can't find the container with id 8c8ba3ebda6254017c5736a764c8f3b8d023fc73df7a8642bf740c595a9ec8d8 Oct 30 17:00:27.672751 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.672721 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk"] Oct 30 17:00:27.790670 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.790630 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk" event={"ID":"c680bca6-5420-4bcc-8ef5-5bfecb85218e","Type":"ContainerStarted","Data":"8c8ba3ebda6254017c5736a764c8f3b8d023fc73df7a8642bf740c595a9ec8d8"} Oct 30 17:00:27.904797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:27.901679 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:28.802573 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:28.802536 6202 generic.go:356] "Generic (PLEG): container finished" podID="c680bca6-5420-4bcc-8ef5-5bfecb85218e" containerID="76e7b53a4efcc895111b78e525eba994abce07b29a907b0d634ba17c7cc4e62a" exitCode=0 Oct 30 17:00:28.802968 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:28.802625 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk" event={"ID":"c680bca6-5420-4bcc-8ef5-5bfecb85218e","Type":"ContainerDied","Data":"76e7b53a4efcc895111b78e525eba994abce07b29a907b0d634ba17c7cc4e62a"} Oct 30 17:00:29.131175 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:29.131147 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" Oct 30 17:00:29.218890 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:29.218752 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ddb2df8-3531-4d65-a0fe-7fe1facc43ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ddb2df8-3531-4d65-a0fe-7fe1facc43ee-plugins-config-volume\") pod \"3ddb2df8-3531-4d65-a0fe-7fe1facc43ee\" (UID: \"3ddb2df8-3531-4d65-a0fe-7fe1facc43ee\") " Oct 30 17:00:29.219064 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:29.218988 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ddb2df8-3531-4d65-a0fe-7fe1facc43ee-kube-api-access-sv9zn\" (UniqueName: \"kubernetes.io/projected/3ddb2df8-3531-4d65-a0fe-7fe1facc43ee-kube-api-access-sv9zn\") pod \"3ddb2df8-3531-4d65-a0fe-7fe1facc43ee\" (UID: \"3ddb2df8-3531-4d65-a0fe-7fe1facc43ee\") " Oct 30 17:00:29.219176 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:29.219150 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ddb2df8-3531-4d65-a0fe-7fe1facc43ee-plugins-config-volume" pod "3ddb2df8-3531-4d65-a0fe-7fe1facc43ee" (UID: "3ddb2df8-3531-4d65-a0fe-7fe1facc43ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:29.219560 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:29.219540 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ddb2df8-3531-4d65-a0fe-7fe1facc43ee-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:29.221469 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:29.221440 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ddb2df8-3531-4d65-a0fe-7fe1facc43ee-kube-api-access-sv9zn" pod "3ddb2df8-3531-4d65-a0fe-7fe1facc43ee" (UID: "3ddb2df8-3531-4d65-a0fe-7fe1facc43ee"). InnerVolumeSpecName "kube-api-access-sv9zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:29.321105 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:29.321064 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv9zn\" (UniqueName: \"kubernetes.io/projected/3ddb2df8-3531-4d65-a0fe-7fe1facc43ee-kube-api-access-sv9zn\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:29.492390 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:29.492278 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:29.814166 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:29.814074 6202 generic.go:356] "Generic (PLEG): container finished" podID="3ddb2df8-3531-4d65-a0fe-7fe1facc43ee" containerID="a28c6990e01344e61b4c79d43cb62bf0ef4f692cb32791d1844fb9ed834b8744" exitCode=0 Oct 30 17:00:29.814166 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:29.814124 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" Oct 30 17:00:29.814663 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:29.814156 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" event={"ID":"3ddb2df8-3531-4d65-a0fe-7fe1facc43ee","Type":"ContainerDied","Data":"a28c6990e01344e61b4c79d43cb62bf0ef4f692cb32791d1844fb9ed834b8744"} Oct 30 17:00:29.814663 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:29.814230 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5" event={"ID":"3ddb2df8-3531-4d65-a0fe-7fe1facc43ee","Type":"ContainerDied","Data":"e1b147057570e022a1c9d0e0b5719ae0bb165118479c9a416e00f8b8f01ed60e"} Oct 30 17:00:29.814663 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:29.814258 6202 scope.go:122] "RemoveContainer" containerID="a28c6990e01344e61b4c79d43cb62bf0ef4f692cb32791d1844fb9ed834b8744" Oct 30 17:00:29.827152 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:29.827127 6202 scope.go:122] "RemoveContainer" containerID="a28c6990e01344e61b4c79d43cb62bf0ef4f692cb32791d1844fb9ed834b8744" Oct 30 17:00:29.827496 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:29.827464 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a28c6990e01344e61b4c79d43cb62bf0ef4f692cb32791d1844fb9ed834b8744\": container with ID starting with a28c6990e01344e61b4c79d43cb62bf0ef4f692cb32791d1844fb9ed834b8744 not found: ID does not exist" containerID="a28c6990e01344e61b4c79d43cb62bf0ef4f692cb32791d1844fb9ed834b8744" Oct 30 17:00:29.827581 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:29.827509 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a28c6990e01344e61b4c79d43cb62bf0ef4f692cb32791d1844fb9ed834b8744"} err="failed to get container status \"a28c6990e01344e61b4c79d43cb62bf0ef4f692cb32791d1844fb9ed834b8744\": rpc error: code = NotFound desc = could not find container \"a28c6990e01344e61b4c79d43cb62bf0ef4f692cb32791d1844fb9ed834b8744\": container with ID starting with a28c6990e01344e61b4c79d43cb62bf0ef4f692cb32791d1844fb9ed834b8744 not found: ID does not exist" Oct 30 17:00:29.870591 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:29.870554 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5"] Oct 30 17:00:29.882073 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:29.882038 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c1e26-67db849779-prfl5"] Oct 30 17:00:30.010851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:30.010826 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk" Oct 30 17:00:30.132286 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:30.132251 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c680bca6-5420-4bcc-8ef5-5bfecb85218e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c680bca6-5420-4bcc-8ef5-5bfecb85218e-maas-api-openshift-ca\") pod \"c680bca6-5420-4bcc-8ef5-5bfecb85218e\" (UID: \"c680bca6-5420-4bcc-8ef5-5bfecb85218e\") " Oct 30 17:00:30.132548 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:30.132534 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c680bca6-5420-4bcc-8ef5-5bfecb85218e-kube-api-access-rp2kt\" (UniqueName: \"kubernetes.io/projected/c680bca6-5420-4bcc-8ef5-5bfecb85218e-kube-api-access-rp2kt\") pod \"c680bca6-5420-4bcc-8ef5-5bfecb85218e\" (UID: \"c680bca6-5420-4bcc-8ef5-5bfecb85218e\") " Oct 30 17:00:30.132596 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:30.132578 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c680bca6-5420-4bcc-8ef5-5bfecb85218e-maas-api-openshift-ca" pod "c680bca6-5420-4bcc-8ef5-5bfecb85218e" (UID: "c680bca6-5420-4bcc-8ef5-5bfecb85218e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:30.132631 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:30.132613 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c680bca6-5420-4bcc-8ef5-5bfecb85218e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c680bca6-5420-4bcc-8ef5-5bfecb85218e-maas-api-xks-ca\") pod \"c680bca6-5420-4bcc-8ef5-5bfecb85218e\" (UID: \"c680bca6-5420-4bcc-8ef5-5bfecb85218e\") " Oct 30 17:00:30.133043 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:30.133020 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c680bca6-5420-4bcc-8ef5-5bfecb85218e-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:30.134755 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:30.134730 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c680bca6-5420-4bcc-8ef5-5bfecb85218e-maas-api-xks-ca" pod "c680bca6-5420-4bcc-8ef5-5bfecb85218e" (UID: "c680bca6-5420-4bcc-8ef5-5bfecb85218e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:30.134934 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:30.134920 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c680bca6-5420-4bcc-8ef5-5bfecb85218e-kube-api-access-rp2kt" pod "c680bca6-5420-4bcc-8ef5-5bfecb85218e" (UID: "c680bca6-5420-4bcc-8ef5-5bfecb85218e"). InnerVolumeSpecName "kube-api-access-rp2kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:30.234860 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:30.234819 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp2kt\" (UniqueName: \"kubernetes.io/projected/c680bca6-5420-4bcc-8ef5-5bfecb85218e-kube-api-access-rp2kt\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:30.234860 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:30.234853 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c680bca6-5420-4bcc-8ef5-5bfecb85218e-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:30.413749 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:30.413669 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ddb2df8-3531-4d65-a0fe-7fe1facc43ee" path="/var/lib/kubelet/pods/3ddb2df8-3531-4d65-a0fe-7fe1facc43ee/volumes" Oct 30 17:00:30.646310 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:30.646261 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" podUID="17152dfe-8124-40b1-b6b9-163d4901d269" containerName="payload-processing" containerID="cri-o://1a203d503635ffffbf156efa8af11d1d1e95d3af1836a98ac304f9881a932a75" gracePeriod=25 Oct 30 17:00:30.826310 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:30.826274 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk" Oct 30 17:00:30.826310 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:30.826290 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk" event={"ID":"c680bca6-5420-4bcc-8ef5-5bfecb85218e","Type":"ContainerDied","Data":"8c8ba3ebda6254017c5736a764c8f3b8d023fc73df7a8642bf740c595a9ec8d8"} Oct 30 17:00:30.826724 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:30.826347 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c8ba3ebda6254017c5736a764c8f3b8d023fc73df7a8642bf740c595a9ec8d8" Oct 30 17:00:31.308407 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:31.308315 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" podUID="242e1339-d85b-4360-99c9-6aab93ac561d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:00:31.418883 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:31.418840 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:32.408372 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:32.408343 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" Oct 30 17:00:32.490318 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:32.490274 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/242e1339-d85b-4360-99c9-6aab93ac561d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/242e1339-d85b-4360-99c9-6aab93ac561d-plugins-config-volume\") pod \"242e1339-d85b-4360-99c9-6aab93ac561d\" (UID: \"242e1339-d85b-4360-99c9-6aab93ac561d\") " Oct 30 17:00:32.490888 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:32.490539 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/242e1339-d85b-4360-99c9-6aab93ac561d-kube-api-access-gp7k2\" (UniqueName: \"kubernetes.io/projected/242e1339-d85b-4360-99c9-6aab93ac561d-kube-api-access-gp7k2\") pod \"242e1339-d85b-4360-99c9-6aab93ac561d\" (UID: \"242e1339-d85b-4360-99c9-6aab93ac561d\") " Oct 30 17:00:32.490888 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:32.490723 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242e1339-d85b-4360-99c9-6aab93ac561d-plugins-config-volume" pod "242e1339-d85b-4360-99c9-6aab93ac561d" (UID: "242e1339-d85b-4360-99c9-6aab93ac561d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:32.491527 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:32.491375 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/242e1339-d85b-4360-99c9-6aab93ac561d-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:32.493540 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:32.493510 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242e1339-d85b-4360-99c9-6aab93ac561d-kube-api-access-gp7k2" pod "242e1339-d85b-4360-99c9-6aab93ac561d" (UID: "242e1339-d85b-4360-99c9-6aab93ac561d"). InnerVolumeSpecName "kube-api-access-gp7k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:32.593663 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:32.593621 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp7k2\" (UniqueName: \"kubernetes.io/projected/242e1339-d85b-4360-99c9-6aab93ac561d-kube-api-access-gp7k2\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:32.851222 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:32.851131 6202 generic.go:356] "Generic (PLEG): container finished" podID="242e1339-d85b-4360-99c9-6aab93ac561d" containerID="ea0e811746fe7c4e37a8ccd1628b13deafdcbfbfb97277f978ae6ef29f699b2b" exitCode=0 Oct 30 17:00:32.851222 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:32.851211 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" Oct 30 17:00:32.851415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:32.851209 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" event={"ID":"242e1339-d85b-4360-99c9-6aab93ac561d","Type":"ContainerDied","Data":"ea0e811746fe7c4e37a8ccd1628b13deafdcbfbfb97277f978ae6ef29f699b2b"} Oct 30 17:00:32.851415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:32.851342 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq" event={"ID":"242e1339-d85b-4360-99c9-6aab93ac561d","Type":"ContainerDied","Data":"b85eaced89969bc7815f39c3410ac1dc1b35eaf6b53ebde1e2e232ef91f9b09d"} Oct 30 17:00:32.851415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:32.851371 6202 scope.go:122] "RemoveContainer" containerID="ea0e811746fe7c4e37a8ccd1628b13deafdcbfbfb97277f978ae6ef29f699b2b" Oct 30 17:00:32.863924 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:32.863898 6202 scope.go:122] "RemoveContainer" containerID="ea0e811746fe7c4e37a8ccd1628b13deafdcbfbfb97277f978ae6ef29f699b2b" Oct 30 17:00:32.864337 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:32.864310 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea0e811746fe7c4e37a8ccd1628b13deafdcbfbfb97277f978ae6ef29f699b2b\": container with ID starting with ea0e811746fe7c4e37a8ccd1628b13deafdcbfbfb97277f978ae6ef29f699b2b not found: ID does not exist" containerID="ea0e811746fe7c4e37a8ccd1628b13deafdcbfbfb97277f978ae6ef29f699b2b" Oct 30 17:00:32.864446 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:32.864347 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea0e811746fe7c4e37a8ccd1628b13deafdcbfbfb97277f978ae6ef29f699b2b"} err="failed to get container status \"ea0e811746fe7c4e37a8ccd1628b13deafdcbfbfb97277f978ae6ef29f699b2b\": rpc error: code = NotFound desc = could not find container \"ea0e811746fe7c4e37a8ccd1628b13deafdcbfbfb97277f978ae6ef29f699b2b\": container with ID starting with ea0e811746fe7c4e37a8ccd1628b13deafdcbfbfb97277f978ae6ef29f699b2b not found: ID does not exist" Oct 30 17:00:32.914083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:32.914052 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq"] Oct 30 17:00:32.925569 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:32.925536 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9bfcac-79d94dd7c4-ht8mq"] Oct 30 17:00:33.462217 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:33.462170 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:34.421791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:34.420104 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="242e1339-d85b-4360-99c9-6aab93ac561d" path="/var/lib/kubelet/pods/242e1339-d85b-4360-99c9-6aab93ac561d/volumes" Oct 30 17:00:34.426366 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:34.426328 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" podUID="17152dfe-8124-40b1-b6b9-163d4901d269" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:00:36.014972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:36.014879 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:37.032913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:37.032871 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5"] Oct 30 17:00:37.037060 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:37.037031 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a767a9-sbvr5"] Oct 30 17:00:37.315522 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:37.315403 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:38.167514 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:38.167472 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:38.413650 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:38.413613 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a6b947-98de-4432-bdb4-8e70f3719e69" path="/var/lib/kubelet/pods/91a6b947-98de-4432-bdb4-8e70f3719e69/volumes" Oct 30 17:00:39.509495 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:39.509454 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:39.705662 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:39.703861 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:40.436082 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:40.436050 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:40.571128 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:40.571062 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:40.584353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:40.584296 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g"] Oct 30 17:00:40.594481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:40.594437 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9bfcac-skn5g"] Oct 30 17:00:40.705293 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:40.705195 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:40.836620 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:40.836582 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:41.106077 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:41.106043 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:42.051255 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.051217 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:42.342565 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.342530 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:42.413216 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.413183 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="181bd0c6-3a62-4ad2-8244-9987e0ac0606" path="/var/lib/kubelet/pods/181bd0c6-3a62-4ad2-8244-9987e0ac0606/volumes" Oct 30 17:00:42.550605 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.545088 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb"] Oct 30 17:00:42.557537 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.557491 6202 cpu_manager.go:414] "containerMap: removing container" podUID="6957d4c0-8197-4aab-a89a-ad17414cd011" containerName="cleanup" Oct 30 17:00:42.557798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.557779 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6957d4c0-8197-4aab-a89a-ad17414cd011" containerName="cleanup" Oct 30 17:00:42.557940 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.557928 6202 cpu_manager.go:414] "containerMap: removing container" podUID="7aadca90-6dbe-4fe5-9dee-10431e9210a0" containerName="cleanup" Oct 30 17:00:42.558047 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.558035 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aadca90-6dbe-4fe5-9dee-10431e9210a0" containerName="cleanup" Oct 30 17:00:42.558180 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.558168 6202 cpu_manager.go:414] "containerMap: removing container" podUID="242e1339-d85b-4360-99c9-6aab93ac561d" containerName="payload-processing" Oct 30 17:00:42.558291 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.558280 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="242e1339-d85b-4360-99c9-6aab93ac561d" containerName="payload-processing" Oct 30 17:00:42.558406 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.558395 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c0b2402f-79e8-4586-9f36-2b86f307f1cb" containerName="cleanup" Oct 30 17:00:42.558521 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.558509 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b2402f-79e8-4586-9f36-2b86f307f1cb" containerName="cleanup" Oct 30 17:00:42.558630 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.558619 6202 cpu_manager.go:414] "containerMap: removing container" podUID="f9872bb2-dae7-43b4-9a03-6d0c9804cb79" containerName="cleanup" Oct 30 17:00:42.558727 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.558716 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9872bb2-dae7-43b4-9a03-6d0c9804cb79" containerName="cleanup" Oct 30 17:00:42.558866 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.558854 6202 cpu_manager.go:414] "containerMap: removing container" podUID="ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9" containerName="cleanup" Oct 30 17:00:42.558990 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.558977 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9" containerName="cleanup" Oct 30 17:00:42.559114 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.559103 6202 cpu_manager.go:414] "containerMap: removing container" podUID="75af19e6-9fae-4786-86d2-3ef7f4433f08" containerName="cleanup" Oct 30 17:00:42.559218 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.559206 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75af19e6-9fae-4786-86d2-3ef7f4433f08" containerName="cleanup" Oct 30 17:00:42.559335 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.559324 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c17f9002-4b15-4e46-a720-7a298c9c3f68" containerName="cleanup" Oct 30 17:00:42.559433 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.559421 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c17f9002-4b15-4e46-a720-7a298c9c3f68" containerName="cleanup" Oct 30 17:00:42.559540 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.559530 6202 cpu_manager.go:414] "containerMap: removing container" podUID="660c4258-30bc-4a2f-b130-766dcbbd88f4" containerName="cleanup" Oct 30 17:00:42.559639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.559627 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660c4258-30bc-4a2f-b130-766dcbbd88f4" containerName="cleanup" Oct 30 17:00:42.559732 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.559721 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c680bca6-5420-4bcc-8ef5-5bfecb85218e" containerName="revoke-keys" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.588543 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c680bca6-5420-4bcc-8ef5-5bfecb85218e" containerName="revoke-keys" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.588611 6202 cpu_manager.go:414] "containerMap: removing container" podUID="0015d61c-9d93-4652-9414-3b817ab96276" containerName="cleanup" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.588624 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0015d61c-9d93-4652-9414-3b817ab96276" containerName="cleanup" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.588638 6202 cpu_manager.go:414] "containerMap: removing container" podUID="b09a458f-a69b-441a-b91a-11854c180a4e" containerName="cleanup" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.588647 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b09a458f-a69b-441a-b91a-11854c180a4e" containerName="cleanup" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.588699 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5c75b279-d01e-43e8-9a3e-8dff3dae29c7" containerName="cleanup" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.588708 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c75b279-d01e-43e8-9a3e-8dff3dae29c7" containerName="cleanup" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.588717 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3ddb2df8-3531-4d65-a0fe-7fe1facc43ee" containerName="payload-pre-processing" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.588725 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ddb2df8-3531-4d65-a0fe-7fe1facc43ee" containerName="payload-pre-processing" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.589106 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ddb2df8-3531-4d65-a0fe-7fe1facc43ee" containerName="payload-pre-processing" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.589130 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc56b1ba-95e4-4bd7-95f1-ffdf539d28d5" containerName="cleanup" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.589148 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aadca90-6dbe-4fe5-9dee-10431e9210a0" containerName="cleanup" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.589157 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7550e3d-3bc1-4ff5-a641-0d68b4bf6127" containerName="cleanup" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.589166 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c75b279-d01e-43e8-9a3e-8dff3dae29c7" containerName="cleanup" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.589178 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="242e1339-d85b-4360-99c9-6aab93ac561d" containerName="payload-processing" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.589188 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad6a2e21-ad1d-4f11-9613-0e5c5fa536e9" containerName="cleanup" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.589201 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="6957d4c0-8197-4aab-a89a-ad17414cd011" containerName="cleanup" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.589209 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b2402f-79e8-4586-9f36-2b86f307f1cb" containerName="cleanup" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.589224 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c17f9002-4b15-4e46-a720-7a298c9c3f68" containerName="cleanup" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.589235 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="75af19e6-9fae-4786-86d2-3ef7f4433f08" containerName="cleanup" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.589244 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c680bca6-5420-4bcc-8ef5-5bfecb85218e" containerName="revoke-keys" Oct 30 17:00:42.589241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.589254 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="b09a458f-a69b-441a-b91a-11854c180a4e" containerName="cleanup" Oct 30 17:00:42.715177 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.705095 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb"] Oct 30 17:00:42.715177 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.705218 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb" Oct 30 17:00:42.728792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.723645 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:00:42.736064 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.735967 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:00:42.840838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.839500 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24f73874-ks6lb\" (UID: \"77cf4478-e8b2-4a37-afe9-caee2bfae0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb" Oct 30 17:00:42.840838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.839659 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24f73874-ks6lb\" (UID: \"77cf4478-e8b2-4a37-afe9-caee2bfae0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb" Oct 30 17:00:42.840838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.839712 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z67b\" (UniqueName: \"kubernetes.io/projected/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-kube-api-access-5z67b\") pod \"maas-api-revoke-sub-24f73874-ks6lb\" (UID: \"77cf4478-e8b2-4a37-afe9-caee2bfae0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb" Oct 30 17:00:42.946274 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.946138 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24f73874-ks6lb\" (UID: \"77cf4478-e8b2-4a37-afe9-caee2bfae0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb" Oct 30 17:00:42.946274 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.946269 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24f73874-ks6lb\" (UID: \"77cf4478-e8b2-4a37-afe9-caee2bfae0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb" Oct 30 17:00:42.946560 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.946309 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z67b\" (UniqueName: \"kubernetes.io/projected/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-kube-api-access-5z67b\") pod \"maas-api-revoke-sub-24f73874-ks6lb\" (UID: \"77cf4478-e8b2-4a37-afe9-caee2bfae0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb" Oct 30 17:00:42.951305 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.951276 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24f73874-ks6lb\" (UID: \"77cf4478-e8b2-4a37-afe9-caee2bfae0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb" Oct 30 17:00:42.966294 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.958705 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24f73874-ks6lb\" (UID: \"77cf4478-e8b2-4a37-afe9-caee2bfae0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb" Oct 30 17:00:42.970804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:42.970752 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z67b\" (UniqueName: \"kubernetes.io/projected/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-kube-api-access-5z67b\") pod \"maas-api-revoke-sub-24f73874-ks6lb\" (UID: \"77cf4478-e8b2-4a37-afe9-caee2bfae0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb" Oct 30 17:00:43.068726 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:43.068686 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb" Oct 30 17:00:43.237428 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:43.237322 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:43.395658 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:43.394353 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb"] Oct 30 17:00:43.398086 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:43.398061 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:43.532036 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:43.531938 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:43.931845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:43.931812 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:43.991479 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:43.991433 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb" event={"ID":"77cf4478-e8b2-4a37-afe9-caee2bfae0c6","Type":"ContainerStarted","Data":"dd6f18579b0f3ef11dd17eb784c5d22967b10fe2748712728b2e3ee341c14901"} Oct 30 17:00:43.991697 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:43.991527 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb" event={"ID":"77cf4478-e8b2-4a37-afe9-caee2bfae0c6","Type":"ContainerStarted","Data":"b9457111ff8ae6fa1a9f9dec2ee80a1eeb7e324879053c13935cdee68d2ce7a2"} Oct 30 17:00:44.035384 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:44.035322 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb" podStartSLOduration=2.035303437 podStartE2EDuration="2.035303437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:44.033276582 +0000 UTC m=+2357.743731325" watchObservedRunningTime="2026-10-30 17:00:44.035303437 +0000 UTC m=+2357.745758186" Oct 30 17:00:44.072017 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:44.071975 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:00:44.425590 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:44.425551 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" podUID="17152dfe-8124-40b1-b6b9-163d4901d269" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:00:45.001446 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:45.001410 6202 generic.go:356] "Generic (PLEG): container finished" podID="77cf4478-e8b2-4a37-afe9-caee2bfae0c6" containerID="dd6f18579b0f3ef11dd17eb784c5d22967b10fe2748712728b2e3ee341c14901" exitCode=0 Oct 30 17:00:45.001662 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:45.001497 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb" event={"ID":"77cf4478-e8b2-4a37-afe9-caee2bfae0c6","Type":"ContainerDied","Data":"dd6f18579b0f3ef11dd17eb784c5d22967b10fe2748712728b2e3ee341c14901"} Oct 30 17:00:45.374699 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:45.374661 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:46.204230 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:46.204201 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb" Oct 30 17:00:46.301726 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:46.301620 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-maas-api-xks-ca\") pod \"77cf4478-e8b2-4a37-afe9-caee2bfae0c6\" (UID: \"77cf4478-e8b2-4a37-afe9-caee2bfae0c6\") " Oct 30 17:00:46.301726 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:46.301697 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-kube-api-access-5z67b\" (UniqueName: \"kubernetes.io/projected/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-kube-api-access-5z67b\") pod \"77cf4478-e8b2-4a37-afe9-caee2bfae0c6\" (UID: \"77cf4478-e8b2-4a37-afe9-caee2bfae0c6\") " Oct 30 17:00:46.301995 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:46.301857 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-maas-api-openshift-ca\") pod \"77cf4478-e8b2-4a37-afe9-caee2bfae0c6\" (UID: \"77cf4478-e8b2-4a37-afe9-caee2bfae0c6\") " Oct 30 17:00:46.302245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:46.302218 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-maas-api-openshift-ca" pod "77cf4478-e8b2-4a37-afe9-caee2bfae0c6" (UID: "77cf4478-e8b2-4a37-afe9-caee2bfae0c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:46.302587 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:46.302567 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:46.304952 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:46.304924 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-maas-api-xks-ca" pod "77cf4478-e8b2-4a37-afe9-caee2bfae0c6" (UID: "77cf4478-e8b2-4a37-afe9-caee2bfae0c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:46.305051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:46.304991 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-kube-api-access-5z67b" pod "77cf4478-e8b2-4a37-afe9-caee2bfae0c6" (UID: "77cf4478-e8b2-4a37-afe9-caee2bfae0c6"). InnerVolumeSpecName "kube-api-access-5z67b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:46.404139 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:46.404097 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:46.404139 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:46.404138 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z67b\" (UniqueName: \"kubernetes.io/projected/77cf4478-e8b2-4a37-afe9-caee2bfae0c6-kube-api-access-5z67b\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:46.766786 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:46.766740 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:47.073935 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:47.073894 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb" event={"ID":"77cf4478-e8b2-4a37-afe9-caee2bfae0c6","Type":"ContainerDied","Data":"b9457111ff8ae6fa1a9f9dec2ee80a1eeb7e324879053c13935cdee68d2ce7a2"} Oct 30 17:00:47.074212 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:47.074193 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9457111ff8ae6fa1a9f9dec2ee80a1eeb7e324879053c13935cdee68d2ce7a2" Oct 30 17:00:47.078047 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:47.076265 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb" Oct 30 17:00:47.525836 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:47.525709 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:47.653225 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:47.653175 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf"] Oct 30 17:00:47.659457 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:47.659420 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c1e26-tkgbf"] Oct 30 17:00:47.704721 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:47.704682 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:48.081832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:48.077068 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc"] Oct 30 17:00:48.119788 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:48.119295 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:48.350954 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:48.350753 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:48.414216 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:48.414178 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5" path="/var/lib/kubelet/pods/bf2f1a10-33b9-45db-bf19-ddf7b26c3ce5/volumes" Oct 30 17:00:49.253112 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:49.253057 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:49.402956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:49.402916 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:49.486751 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:49.486707 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8"] Oct 30 17:00:49.487242 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:49.487201 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" podUID="5d729b26-7129-4f64-95d9-860074961e2d" containerName="istio-proxy" containerID="cri-o://841b4ffd95bd9502a5e301f4809ff02ba68eb8604a08f7938eca92e0f7bc6f1b" gracePeriod=30 Oct 30 17:00:49.547731 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:49.547610 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:49.664563 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:49.664526 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:50.956615 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:50.956583 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" Oct 30 17:00:51.043795 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.043743 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17152dfe-8124-40b1-b6b9-163d4901d269-kube-api-access-5rl9m\" (UniqueName: \"kubernetes.io/projected/17152dfe-8124-40b1-b6b9-163d4901d269-kube-api-access-5rl9m\") pod \"17152dfe-8124-40b1-b6b9-163d4901d269\" (UID: \"17152dfe-8124-40b1-b6b9-163d4901d269\") " Oct 30 17:00:51.044259 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.044232 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17152dfe-8124-40b1-b6b9-163d4901d269-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17152dfe-8124-40b1-b6b9-163d4901d269-plugins-config-volume\") pod \"17152dfe-8124-40b1-b6b9-163d4901d269\" (UID: \"17152dfe-8124-40b1-b6b9-163d4901d269\") " Oct 30 17:00:51.044878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.044848 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17152dfe-8124-40b1-b6b9-163d4901d269-plugins-config-volume" pod "17152dfe-8124-40b1-b6b9-163d4901d269" (UID: "17152dfe-8124-40b1-b6b9-163d4901d269"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:51.046743 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.046713 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17152dfe-8124-40b1-b6b9-163d4901d269-kube-api-access-5rl9m" pod "17152dfe-8124-40b1-b6b9-163d4901d269" (UID: "17152dfe-8124-40b1-b6b9-163d4901d269"). InnerVolumeSpecName "kube-api-access-5rl9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:51.133622 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.133585 6202 generic.go:356] "Generic (PLEG): container finished" podID="17152dfe-8124-40b1-b6b9-163d4901d269" containerID="1a203d503635ffffbf156efa8af11d1d1e95d3af1836a98ac304f9881a932a75" exitCode=0 Oct 30 17:00:51.133854 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.133642 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" Oct 30 17:00:51.133854 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.133672 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" event={"ID":"17152dfe-8124-40b1-b6b9-163d4901d269","Type":"ContainerDied","Data":"1a203d503635ffffbf156efa8af11d1d1e95d3af1836a98ac304f9881a932a75"} Oct 30 17:00:51.133854 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.133742 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw" event={"ID":"17152dfe-8124-40b1-b6b9-163d4901d269","Type":"ContainerDied","Data":"8fb681c7a1bbcffd6e5bdfa26086b36b3863cf52b75cec574102a1b3c4cba161"} Oct 30 17:00:51.133854 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.133791 6202 scope.go:122] "RemoveContainer" containerID="1a203d503635ffffbf156efa8af11d1d1e95d3af1836a98ac304f9881a932a75" Oct 30 17:00:51.147151 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.147115 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rl9m\" (UniqueName: \"kubernetes.io/projected/17152dfe-8124-40b1-b6b9-163d4901d269-kube-api-access-5rl9m\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:51.147305 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.147153 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17152dfe-8124-40b1-b6b9-163d4901d269-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:51.150514 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.150484 6202 scope.go:122] "RemoveContainer" containerID="1a203d503635ffffbf156efa8af11d1d1e95d3af1836a98ac304f9881a932a75" Oct 30 17:00:51.152291 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:51.152262 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a203d503635ffffbf156efa8af11d1d1e95d3af1836a98ac304f9881a932a75\": container with ID starting with 1a203d503635ffffbf156efa8af11d1d1e95d3af1836a98ac304f9881a932a75 not found: ID does not exist" containerID="1a203d503635ffffbf156efa8af11d1d1e95d3af1836a98ac304f9881a932a75" Oct 30 17:00:51.152420 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.152300 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a203d503635ffffbf156efa8af11d1d1e95d3af1836a98ac304f9881a932a75"} err="failed to get container status \"1a203d503635ffffbf156efa8af11d1d1e95d3af1836a98ac304f9881a932a75\": rpc error: code = NotFound desc = could not find container \"1a203d503635ffffbf156efa8af11d1d1e95d3af1836a98ac304f9881a932a75\": container with ID starting with 1a203d503635ffffbf156efa8af11d1d1e95d3af1836a98ac304f9881a932a75 not found: ID does not exist" Oct 30 17:00:51.187714 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.187626 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw"] Oct 30 17:00:51.196364 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.196328 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17f60f-685665d68d-btqsw"] Oct 30 17:00:51.553220 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.553103 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:51.713551 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.713478 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:51.886552 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.886510 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj"] Oct 30 17:00:51.886810 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.886754 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" podUID="4cead7b4-5a36-4ed4-8550-3c454ace71a3" containerName="istio-proxy" containerID="cri-o://d663a7002638c3ee2565968279436a3b92be88aa1ed03a504f1839c2626c925c" gracePeriod=30 Oct 30 17:00:51.908874 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:51.908844 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:52.040791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.037830 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:52.415190 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.415156 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17152dfe-8124-40b1-b6b9-163d4901d269" path="/var/lib/kubelet/pods/17152dfe-8124-40b1-b6b9-163d4901d269/volumes" Oct 30 17:00:52.448996 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.448956 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq"] Oct 30 17:00:52.449795 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.449778 6202 cpu_manager.go:414] "containerMap: removing container" podUID="77cf4478-e8b2-4a37-afe9-caee2bfae0c6" containerName="revoke-keys" Oct 30 17:00:52.449847 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.449797 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77cf4478-e8b2-4a37-afe9-caee2bfae0c6" containerName="revoke-keys" Oct 30 17:00:52.449847 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.449810 6202 cpu_manager.go:414] "containerMap: removing container" podUID="17152dfe-8124-40b1-b6b9-163d4901d269" containerName="payload-processing" Oct 30 17:00:52.449847 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.449815 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17152dfe-8124-40b1-b6b9-163d4901d269" containerName="payload-processing" Oct 30 17:00:52.449957 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.449948 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="77cf4478-e8b2-4a37-afe9-caee2bfae0c6" containerName="revoke-keys" Oct 30 17:00:52.449993 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.449957 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="17152dfe-8124-40b1-b6b9-163d4901d269" containerName="payload-processing" Oct 30 17:00:52.455797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.455742 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq" Oct 30 17:00:52.478601 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.478574 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq"] Oct 30 17:00:52.555998 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.555960 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:00:52.556228 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.556209 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:00:52.590019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.589977 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fabd6007-9258-4203-82ec-b4c024c9102e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-689899-96kwq\" (UID: \"fabd6007-9258-4203-82ec-b4c024c9102e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq" Oct 30 17:00:52.590209 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.590024 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnfzj\" (UniqueName: \"kubernetes.io/projected/fabd6007-9258-4203-82ec-b4c024c9102e-kube-api-access-hnfzj\") pod \"maas-api-revoke-keys-e2e-inf-b-689899-96kwq\" (UID: \"fabd6007-9258-4203-82ec-b4c024c9102e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq" Oct 30 17:00:52.590209 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.590180 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fabd6007-9258-4203-82ec-b4c024c9102e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-689899-96kwq\" (UID: \"fabd6007-9258-4203-82ec-b4c024c9102e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq" Oct 30 17:00:52.691933 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.691824 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fabd6007-9258-4203-82ec-b4c024c9102e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-689899-96kwq\" (UID: \"fabd6007-9258-4203-82ec-b4c024c9102e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq" Oct 30 17:00:52.692127 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.692094 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fabd6007-9258-4203-82ec-b4c024c9102e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-689899-96kwq\" (UID: \"fabd6007-9258-4203-82ec-b4c024c9102e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq" Oct 30 17:00:52.692127 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.692122 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnfzj\" (UniqueName: \"kubernetes.io/projected/fabd6007-9258-4203-82ec-b4c024c9102e-kube-api-access-hnfzj\") pod \"maas-api-revoke-keys-e2e-inf-b-689899-96kwq\" (UID: \"fabd6007-9258-4203-82ec-b4c024c9102e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq" Oct 30 17:00:52.692613 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.692589 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fabd6007-9258-4203-82ec-b4c024c9102e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-689899-96kwq\" (UID: \"fabd6007-9258-4203-82ec-b4c024c9102e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq" Oct 30 17:00:52.694668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.694642 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fabd6007-9258-4203-82ec-b4c024c9102e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-689899-96kwq\" (UID: \"fabd6007-9258-4203-82ec-b4c024c9102e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq" Oct 30 17:00:52.717891 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.717862 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnfzj\" (UniqueName: \"kubernetes.io/projected/fabd6007-9258-4203-82ec-b4c024c9102e-kube-api-access-hnfzj\") pod \"maas-api-revoke-keys-e2e-inf-b-689899-96kwq\" (UID: \"fabd6007-9258-4203-82ec-b4c024c9102e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq" Oct 30 17:00:52.770884 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:52.770844 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq" Oct 30 17:00:53.077606 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:00:53.077569 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabd6007_9258_4203_82ec_b4c024c9102e.slice/crio-6c90a93112e0cc558073b703f3c7973d1e509be9ab843424893758f3f3eb82e9 WatchSource:0}: Error finding container 6c90a93112e0cc558073b703f3c7973d1e509be9ab843424893758f3f3eb82e9: Status 404 returned error can't find the container with id 6c90a93112e0cc558073b703f3c7973d1e509be9ab843424893758f3f3eb82e9 Oct 30 17:00:53.079063 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:53.079039 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq"] Oct 30 17:00:53.104272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:53.104235 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:53.189972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:53.189935 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq" event={"ID":"fabd6007-9258-4203-82ec-b4c024c9102e","Type":"ContainerStarted","Data":"6c90a93112e0cc558073b703f3c7973d1e509be9ab843424893758f3f3eb82e9"} Oct 30 17:00:53.867254 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:53.867223 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:53.894371 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:53.894332 6202 patch_prober.go:29] interesting pod/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:00:53.894570 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:53.894419 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" podUID="5d729b26-7129-4f64-95d9-860074961e2d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:00:54.203545 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.203450 6202 generic.go:356] "Generic (PLEG): container finished" podID="fabd6007-9258-4203-82ec-b4c024c9102e" containerID="9fae3a4175cb1aef23e97825b8fee2350ecad498c198c5848215a26094794671" exitCode=0 Oct 30 17:00:54.204020 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.203545 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq" event={"ID":"fabd6007-9258-4203-82ec-b4c024c9102e","Type":"ContainerDied","Data":"9fae3a4175cb1aef23e97825b8fee2350ecad498c198c5848215a26094794671"} Oct 30 17:00:54.801082 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.801054 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 17:00:54.936702 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.936669 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d729b26-7129-4f64-95d9-860074961e2d-istio-token\" (UniqueName: \"kubernetes.io/projected/5d729b26-7129-4f64-95d9-860074961e2d-istio-token\") pod \"5d729b26-7129-4f64-95d9-860074961e2d\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " Oct 30 17:00:54.936914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.936731 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d729b26-7129-4f64-95d9-860074961e2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d729b26-7129-4f64-95d9-860074961e2d-istio-podinfo\") pod \"5d729b26-7129-4f64-95d9-860074961e2d\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " Oct 30 17:00:54.936914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.936869 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-workload-certs\") pod \"5d729b26-7129-4f64-95d9-860074961e2d\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " Oct 30 17:00:54.936914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.936901 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-credential-socket\") pod \"5d729b26-7129-4f64-95d9-860074961e2d\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " Oct 30 17:00:54.937111 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.936986 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d729b26-7129-4f64-95d9-860074961e2d-kube-api-access-lqdbn\" (UniqueName: \"kubernetes.io/projected/5d729b26-7129-4f64-95d9-860074961e2d-kube-api-access-lqdbn\") pod \"5d729b26-7129-4f64-95d9-860074961e2d\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " Oct 30 17:00:54.937111 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.937035 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-istio-data\") pod \"5d729b26-7129-4f64-95d9-860074961e2d\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " Oct 30 17:00:54.937217 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.937158 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-istio-envoy\") pod \"5d729b26-7129-4f64-95d9-860074961e2d\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " Oct 30 17:00:54.937217 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.937196 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-workload-certs" pod "5d729b26-7129-4f64-95d9-860074961e2d" (UID: "5d729b26-7129-4f64-95d9-860074961e2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:54.937316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.937226 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-credential-socket" pod "5d729b26-7129-4f64-95d9-860074961e2d" (UID: "5d729b26-7129-4f64-95d9-860074961e2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:54.937316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.937289 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-istio-data" pod "5d729b26-7129-4f64-95d9-860074961e2d" (UID: "5d729b26-7129-4f64-95d9-860074961e2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:54.937692 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.937665 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-workload-socket\") pod \"5d729b26-7129-4f64-95d9-860074961e2d\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " Oct 30 17:00:54.937884 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.937721 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d729b26-7129-4f64-95d9-860074961e2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d729b26-7129-4f64-95d9-860074961e2d-istiod-ca-cert\") pod \"5d729b26-7129-4f64-95d9-860074961e2d\" (UID: \"5d729b26-7129-4f64-95d9-860074961e2d\") " Oct 30 17:00:54.938111 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.938081 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-workload-socket" pod "5d729b26-7129-4f64-95d9-860074961e2d" (UID: "5d729b26-7129-4f64-95d9-860074961e2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:54.938404 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.938370 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d729b26-7129-4f64-95d9-860074961e2d-istiod-ca-cert" pod "5d729b26-7129-4f64-95d9-860074961e2d" (UID: "5d729b26-7129-4f64-95d9-860074961e2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:54.938668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.938644 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:54.938747 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.938676 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d729b26-7129-4f64-95d9-860074961e2d-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:54.938747 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.938692 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:54.938747 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.938706 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:54.938747 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.938721 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:54.939377 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.939356 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-istio-envoy" pod "5d729b26-7129-4f64-95d9-860074961e2d" (UID: "5d729b26-7129-4f64-95d9-860074961e2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:54.939377 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.939359 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d729b26-7129-4f64-95d9-860074961e2d-istio-token" pod "5d729b26-7129-4f64-95d9-860074961e2d" (UID: "5d729b26-7129-4f64-95d9-860074961e2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:54.939636 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.939620 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d729b26-7129-4f64-95d9-860074961e2d-istio-podinfo" pod "5d729b26-7129-4f64-95d9-860074961e2d" (UID: "5d729b26-7129-4f64-95d9-860074961e2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:00:54.940640 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:54.940609 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d729b26-7129-4f64-95d9-860074961e2d-kube-api-access-lqdbn" pod "5d729b26-7129-4f64-95d9-860074961e2d" (UID: "5d729b26-7129-4f64-95d9-860074961e2d"). InnerVolumeSpecName "kube-api-access-lqdbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:55.040501 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.040413 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d729b26-7129-4f64-95d9-860074961e2d-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:55.040501 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.040440 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d729b26-7129-4f64-95d9-860074961e2d-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:55.040501 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.040449 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d729b26-7129-4f64-95d9-860074961e2d-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:55.040501 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.040459 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqdbn\" (UniqueName: \"kubernetes.io/projected/5d729b26-7129-4f64-95d9-860074961e2d-kube-api-access-lqdbn\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:55.191160 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.191126 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:55.214085 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.214052 6202 generic.go:356] "Generic (PLEG): container finished" podID="5d729b26-7129-4f64-95d9-860074961e2d" containerID="841b4ffd95bd9502a5e301f4809ff02ba68eb8604a08f7938eca92e0f7bc6f1b" exitCode=0 Oct 30 17:00:55.214626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.214126 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" Oct 30 17:00:55.214626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.214161 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" event={"ID":"5d729b26-7129-4f64-95d9-860074961e2d","Type":"ContainerDied","Data":"841b4ffd95bd9502a5e301f4809ff02ba68eb8604a08f7938eca92e0f7bc6f1b"} Oct 30 17:00:55.214626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.214236 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8" event={"ID":"5d729b26-7129-4f64-95d9-860074961e2d","Type":"ContainerDied","Data":"976fe0b4665a0752493c68e429bbda9a0321b068d18282d859b5591366e7b34d"} Oct 30 17:00:55.214626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.214272 6202 scope.go:122] "RemoveContainer" containerID="841b4ffd95bd9502a5e301f4809ff02ba68eb8604a08f7938eca92e0f7bc6f1b" Oct 30 17:00:55.230549 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.230522 6202 scope.go:122] "RemoveContainer" containerID="bfba618ec63d7227ae99120abb61ae940f89d53e880cceb04b3b442fab5705f2" Oct 30 17:00:55.272370 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.272331 6202 scope.go:122] "RemoveContainer" containerID="841b4ffd95bd9502a5e301f4809ff02ba68eb8604a08f7938eca92e0f7bc6f1b" Oct 30 17:00:55.272851 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:55.272758 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841b4ffd95bd9502a5e301f4809ff02ba68eb8604a08f7938eca92e0f7bc6f1b\": container with ID starting with 841b4ffd95bd9502a5e301f4809ff02ba68eb8604a08f7938eca92e0f7bc6f1b not found: ID does not exist" containerID="841b4ffd95bd9502a5e301f4809ff02ba68eb8604a08f7938eca92e0f7bc6f1b" Oct 30 17:00:55.272986 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.272862 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841b4ffd95bd9502a5e301f4809ff02ba68eb8604a08f7938eca92e0f7bc6f1b"} err="failed to get container status \"841b4ffd95bd9502a5e301f4809ff02ba68eb8604a08f7938eca92e0f7bc6f1b\": rpc error: code = NotFound desc = could not find container \"841b4ffd95bd9502a5e301f4809ff02ba68eb8604a08f7938eca92e0f7bc6f1b\": container with ID starting with 841b4ffd95bd9502a5e301f4809ff02ba68eb8604a08f7938eca92e0f7bc6f1b not found: ID does not exist" Oct 30 17:00:55.272986 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.272894 6202 scope.go:122] "RemoveContainer" containerID="bfba618ec63d7227ae99120abb61ae940f89d53e880cceb04b3b442fab5705f2" Oct 30 17:00:55.273441 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:00:55.273414 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfba618ec63d7227ae99120abb61ae940f89d53e880cceb04b3b442fab5705f2\": container with ID starting with bfba618ec63d7227ae99120abb61ae940f89d53e880cceb04b3b442fab5705f2 not found: ID does not exist" containerID="bfba618ec63d7227ae99120abb61ae940f89d53e880cceb04b3b442fab5705f2" Oct 30 17:00:55.273543 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.273448 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfba618ec63d7227ae99120abb61ae940f89d53e880cceb04b3b442fab5705f2"} err="failed to get container status \"bfba618ec63d7227ae99120abb61ae940f89d53e880cceb04b3b442fab5705f2\": rpc error: code = NotFound desc = could not find container \"bfba618ec63d7227ae99120abb61ae940f89d53e880cceb04b3b442fab5705f2\": container with ID starting with bfba618ec63d7227ae99120abb61ae940f89d53e880cceb04b3b442fab5705f2 not found: ID does not exist" Oct 30 17:00:55.293300 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.293219 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8"] Oct 30 17:00:55.304973 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.304756 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9bfcac-openshift-default-676454d6b5-4hhw8"] Oct 30 17:00:55.368533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.368485 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:55.407344 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.407317 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq" Oct 30 17:00:55.552636 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.552547 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv"] Oct 30 17:00:55.552994 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.552962 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fabd6007-9258-4203-82ec-b4c024c9102e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fabd6007-9258-4203-82ec-b4c024c9102e-maas-api-openshift-ca\") pod \"fabd6007-9258-4203-82ec-b4c024c9102e\" (UID: \"fabd6007-9258-4203-82ec-b4c024c9102e\") " Oct 30 17:00:55.553167 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.553021 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fabd6007-9258-4203-82ec-b4c024c9102e-kube-api-access-hnfzj\" (UniqueName: \"kubernetes.io/projected/fabd6007-9258-4203-82ec-b4c024c9102e-kube-api-access-hnfzj\") pod \"fabd6007-9258-4203-82ec-b4c024c9102e\" (UID: \"fabd6007-9258-4203-82ec-b4c024c9102e\") " Oct 30 17:00:55.553167 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.553119 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" podUID="c7320e48-7c58-40b0-bd5b-03b9f4bb551f" containerName="istio-proxy" containerID="cri-o://a93ec556596a6308c936c62b119db9f06348514a9c1c8723762d98d97f4707eb" gracePeriod=30 Oct 30 17:00:55.553425 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.553332 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fabd6007-9258-4203-82ec-b4c024c9102e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fabd6007-9258-4203-82ec-b4c024c9102e-maas-api-xks-ca\") pod \"fabd6007-9258-4203-82ec-b4c024c9102e\" (UID: \"fabd6007-9258-4203-82ec-b4c024c9102e\") " Oct 30 17:00:55.553523 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.553480 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fabd6007-9258-4203-82ec-b4c024c9102e-maas-api-openshift-ca" pod "fabd6007-9258-4203-82ec-b4c024c9102e" (UID: "fabd6007-9258-4203-82ec-b4c024c9102e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:55.554658 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.554499 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fabd6007-9258-4203-82ec-b4c024c9102e-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:55.559017 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.558980 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fabd6007-9258-4203-82ec-b4c024c9102e-kube-api-access-hnfzj" pod "fabd6007-9258-4203-82ec-b4c024c9102e" (UID: "fabd6007-9258-4203-82ec-b4c024c9102e"). InnerVolumeSpecName "kube-api-access-hnfzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:55.559519 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.559490 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fabd6007-9258-4203-82ec-b4c024c9102e-maas-api-xks-ca" pod "fabd6007-9258-4203-82ec-b4c024c9102e" (UID: "fabd6007-9258-4203-82ec-b4c024c9102e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:55.607922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.607881 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:55.665395 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.665328 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fabd6007-9258-4203-82ec-b4c024c9102e-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:55.665707 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.665665 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnfzj\" (UniqueName: \"kubernetes.io/projected/fabd6007-9258-4203-82ec-b4c024c9102e-kube-api-access-hnfzj\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:55.752120 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:55.752083 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:56.201267 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.201232 6202 patch_prober.go:29] interesting pod/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:00:56.201430 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.201294 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" podUID="c7320e48-7c58-40b0-bd5b-03b9f4bb551f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:00:56.224604 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.224572 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq" Oct 30 17:00:56.225106 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.224599 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq" event={"ID":"fabd6007-9258-4203-82ec-b4c024c9102e","Type":"ContainerDied","Data":"6c90a93112e0cc558073b703f3c7973d1e509be9ab843424893758f3f3eb82e9"} Oct 30 17:00:56.225106 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.224689 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c90a93112e0cc558073b703f3c7973d1e509be9ab843424893758f3f3eb82e9" Oct 30 17:00:56.413476 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.413445 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d729b26-7129-4f64-95d9-860074961e2d" path="/var/lib/kubelet/pods/5d729b26-7129-4f64-95d9-860074961e2d/volumes" Oct 30 17:00:56.480615 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.480517 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:56.965004 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.964970 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7"] Oct 30 17:00:56.965758 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.965743 6202 cpu_manager.go:414] "containerMap: removing container" podUID="fabd6007-9258-4203-82ec-b4c024c9102e" containerName="revoke-keys" Oct 30 17:00:56.965827 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.965777 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fabd6007-9258-4203-82ec-b4c024c9102e" containerName="revoke-keys" Oct 30 17:00:56.965827 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.965797 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5d729b26-7129-4f64-95d9-860074961e2d" containerName="istio-proxy" Oct 30 17:00:56.965827 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.965803 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d729b26-7129-4f64-95d9-860074961e2d" containerName="istio-proxy" Oct 30 17:00:56.965827 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.965812 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5d729b26-7129-4f64-95d9-860074961e2d" containerName="istio-proxy" Oct 30 17:00:56.965827 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.965817 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d729b26-7129-4f64-95d9-860074961e2d" containerName="istio-proxy" Oct 30 17:00:56.965991 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.965965 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d729b26-7129-4f64-95d9-860074961e2d" containerName="istio-proxy" Oct 30 17:00:56.965991 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.965973 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d729b26-7129-4f64-95d9-860074961e2d" containerName="istio-proxy" Oct 30 17:00:56.965991 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.965990 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="fabd6007-9258-4203-82ec-b4c024c9102e" containerName="revoke-keys" Oct 30 17:00:56.969931 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.969905 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:56.989166 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.988988 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istiod-ca-cert\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:56.989166 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.989126 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-workload-certs\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:56.989480 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.989433 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-podinfo\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:56.989559 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.989493 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-token\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:56.989606 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.989574 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-credential-socket\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:56.989659 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.989622 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzw97\" (UniqueName: \"kubernetes.io/projected/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-kube-api-access-nzw97\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:56.989659 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.989650 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-data\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:56.989751 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.989718 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-workload-socket\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:56.989847 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:56.989828 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-envoy\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.001905 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.001870 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-b4214a-openshift-default-dockercfg-s4nj8\"" Oct 30 17:00:57.006898 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.006786 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7"] Oct 30 17:00:57.121790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.113364 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:57.158163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.151033 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-workload-socket\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.158163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.151126 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-workload-socket\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.158163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.151216 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-envoy\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.158163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.151280 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istiod-ca-cert\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.158163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.151399 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-workload-certs\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.158163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.151542 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-podinfo\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.158163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.151578 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-token\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.158163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.151663 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-credential-socket\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.158163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.151711 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzw97\" (UniqueName: \"kubernetes.io/projected/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-kube-api-access-nzw97\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.158163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.151739 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-data\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.158163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.156327 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-credential-socket\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.158163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.157526 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istiod-ca-cert\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.158163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.158053 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-workload-certs\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.159626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.159559 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-data\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.163870 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.163402 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-podinfo\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.167091 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.167029 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-envoy\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.263181 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.262361 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzw97\" (UniqueName: \"kubernetes.io/projected/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-kube-api-access-nzw97\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.278840 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.276821 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-token\") pod \"e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.278840 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.277956 6202 generic.go:356] "Generic (PLEG): container finished" podID="4cead7b4-5a36-4ed4-8550-3c454ace71a3" containerID="d663a7002638c3ee2565968279436a3b92be88aa1ed03a504f1839c2626c925c" exitCode=0 Oct 30 17:00:57.278840 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.278177 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" event={"ID":"4cead7b4-5a36-4ed4-8550-3c454ace71a3","Type":"ContainerDied","Data":"d663a7002638c3ee2565968279436a3b92be88aa1ed03a504f1839c2626c925c"} Oct 30 17:00:57.305632 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.304699 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:57.446691 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.446658 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 17:00:57.558585 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.558498 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:57.580212 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.580181 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-workload-certs\") pod \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " Oct 30 17:00:57.580212 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.580222 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-workload-socket\") pod \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " Oct 30 17:00:57.580509 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.580272 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-podinfo\") pod \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " Oct 30 17:00:57.580509 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.580418 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cead7b4-5a36-4ed4-8550-3c454ace71a3-kube-api-access-wmshr\" (UniqueName: \"kubernetes.io/projected/4cead7b4-5a36-4ed4-8550-3c454ace71a3-kube-api-access-wmshr\") pod \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " Oct 30 17:00:57.580509 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.580478 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-data\") pod \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " Oct 30 17:00:57.580509 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.580505 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istiod-ca-cert\") pod \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " Oct 30 17:00:57.580737 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.580501 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-workload-certs" pod "4cead7b4-5a36-4ed4-8550-3c454ace71a3" (UID: "4cead7b4-5a36-4ed4-8550-3c454ace71a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:57.580737 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.580594 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-envoy\") pod \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " Oct 30 17:00:57.580737 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.580629 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-token\" (UniqueName: \"kubernetes.io/projected/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-token\") pod \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " Oct 30 17:00:57.580737 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.580674 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-credential-socket\") pod \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\" (UID: \"4cead7b4-5a36-4ed4-8550-3c454ace71a3\") " Oct 30 17:00:57.581212 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.581179 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istiod-ca-cert" pod "4cead7b4-5a36-4ed4-8550-3c454ace71a3" (UID: "4cead7b4-5a36-4ed4-8550-3c454ace71a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:57.581495 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.581434 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-workload-socket" pod "4cead7b4-5a36-4ed4-8550-3c454ace71a3" (UID: "4cead7b4-5a36-4ed4-8550-3c454ace71a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:57.581695 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.581506 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-data" pod "4cead7b4-5a36-4ed4-8550-3c454ace71a3" (UID: "4cead7b4-5a36-4ed4-8550-3c454ace71a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:57.581844 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.581818 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-credential-socket" pod "4cead7b4-5a36-4ed4-8550-3c454ace71a3" (UID: "4cead7b4-5a36-4ed4-8550-3c454ace71a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:57.582329 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.582298 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.582329 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.582326 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.582482 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.582341 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.582482 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.582356 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.582482 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.582370 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.583849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.583817 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-envoy" pod "4cead7b4-5a36-4ed4-8550-3c454ace71a3" (UID: "4cead7b4-5a36-4ed4-8550-3c454ace71a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:57.584379 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.584352 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-podinfo" pod "4cead7b4-5a36-4ed4-8550-3c454ace71a3" (UID: "4cead7b4-5a36-4ed4-8550-3c454ace71a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:00:57.584750 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.584720 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cead7b4-5a36-4ed4-8550-3c454ace71a3-kube-api-access-wmshr" pod "4cead7b4-5a36-4ed4-8550-3c454ace71a3" (UID: "4cead7b4-5a36-4ed4-8550-3c454ace71a3"). InnerVolumeSpecName "kube-api-access-wmshr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:57.584977 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.584947 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-token" pod "4cead7b4-5a36-4ed4-8550-3c454ace71a3" (UID: "4cead7b4-5a36-4ed4-8550-3c454ace71a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:57.672099 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:00:57.672053 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7bcb077_0d1d_4046_9a5f_75fc61dd2886.slice/crio-82924796ddb011f300b8dcc1f72c87727263cb29446f5b39319138c08391d929 WatchSource:0}: Error finding container 82924796ddb011f300b8dcc1f72c87727263cb29446f5b39319138c08391d929: Status 404 returned error can't find the container with id 82924796ddb011f300b8dcc1f72c87727263cb29446f5b39319138c08391d929 Oct 30 17:00:57.673723 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.673581 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7"] Oct 30 17:00:57.677827 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.677783 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:00:57.677960 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.677883 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:00:57.678049 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.677961 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:00:57.684998 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.684970 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.685161 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.685149 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.685313 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.685297 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cead7b4-5a36-4ed4-8550-3c454ace71a3-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.685471 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.685433 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmshr\" (UniqueName: \"kubernetes.io/projected/4cead7b4-5a36-4ed4-8550-3c454ace71a3-kube-api-access-wmshr\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:00:57.764253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:57.764222 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:58.290809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:58.290704 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" event={"ID":"e7bcb077-0d1d-4046-9a5f-75fc61dd2886","Type":"ContainerStarted","Data":"ce8b40adbd789f536bf361e2b3ad0e0616e05d724036925874c9ffa350ed2c5d"} Oct 30 17:00:58.290809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:58.290761 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" event={"ID":"e7bcb077-0d1d-4046-9a5f-75fc61dd2886","Type":"ContainerStarted","Data":"82924796ddb011f300b8dcc1f72c87727263cb29446f5b39319138c08391d929"} Oct 30 17:00:58.292451 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:58.292428 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" Oct 30 17:00:58.292558 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:58.292435 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj" event={"ID":"4cead7b4-5a36-4ed4-8550-3c454ace71a3","Type":"ContainerDied","Data":"6d2d08694cd65a4f8893a582f21fb0f9344309a610c2e30d22c9e97b1291ec74"} Oct 30 17:00:58.292558 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:58.292546 6202 scope.go:122] "RemoveContainer" containerID="d663a7002638c3ee2565968279436a3b92be88aa1ed03a504f1839c2626c925c" Oct 30 17:00:58.307092 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:58.307057 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:58.313574 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:58.313546 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:58.370228 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:58.370158 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" podStartSLOduration=2.370138897 podStartE2EDuration="2.370138897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:58.368518317 +0000 UTC m=+2372.078973082" watchObservedRunningTime="2026-10-30 17:00:58.370138897 +0000 UTC m=+2372.080593641" Oct 30 17:00:58.429845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:58.429807 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj"] Oct 30 17:00:58.433912 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:58.433882 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-17f60f-openshift-default-9c8cfdbb7-z7mcj"] Oct 30 17:00:59.302481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:59.302439 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:59.303670 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:59.303647 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:00:59.476338 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:00:59.476306 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:00.413974 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:00.413938 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cead7b4-5a36-4ed4-8550-3c454ace71a3" path="/var/lib/kubelet/pods/4cead7b4-5a36-4ed4-8550-3c454ace71a3/volumes" Oct 30 17:01:00.883895 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:00.883864 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 17:01:00.892161 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:00.892133 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:01.033148 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.033047 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:01.052531 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.052495 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-podinfo\") pod \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " Oct 30 17:01:01.052738 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.052559 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-data\") pod \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " Oct 30 17:01:01.052738 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.052701 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-workload-socket\") pod \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " Oct 30 17:01:01.052920 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.052756 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istiod-ca-cert\") pod \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " Oct 30 17:01:01.052920 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.052864 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-envoy\") pod \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " Oct 30 17:01:01.053030 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.052935 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-workload-certs\") pod \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " Oct 30 17:01:01.053030 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.052981 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-data" pod "c7320e48-7c58-40b0-bd5b-03b9f4bb551f" (UID: "c7320e48-7c58-40b0-bd5b-03b9f4bb551f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:01.053030 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.052988 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-token\" (UniqueName: \"kubernetes.io/projected/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-token\") pod \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " Oct 30 17:01:01.053196 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.053129 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-workload-socket" pod "c7320e48-7c58-40b0-bd5b-03b9f4bb551f" (UID: "c7320e48-7c58-40b0-bd5b-03b9f4bb551f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:01.053196 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.053177 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-credential-socket\") pod \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " Oct 30 17:01:01.053340 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.053214 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istiod-ca-cert" pod "c7320e48-7c58-40b0-bd5b-03b9f4bb551f" (UID: "c7320e48-7c58-40b0-bd5b-03b9f4bb551f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:01.053402 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.053253 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-kube-api-access-kvk5x\" (UniqueName: \"kubernetes.io/projected/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-kube-api-access-kvk5x\") pod \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\" (UID: \"c7320e48-7c58-40b0-bd5b-03b9f4bb551f\") " Oct 30 17:01:01.053460 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.053299 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-workload-certs" pod "c7320e48-7c58-40b0-bd5b-03b9f4bb551f" (UID: "c7320e48-7c58-40b0-bd5b-03b9f4bb551f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:01.054073 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.054048 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:01.054205 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.054075 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:01.054205 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.054089 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:01.054205 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.054101 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:01.054403 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.054356 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-credential-socket" pod "c7320e48-7c58-40b0-bd5b-03b9f4bb551f" (UID: "c7320e48-7c58-40b0-bd5b-03b9f4bb551f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:01.055369 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.055333 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-podinfo" pod "c7320e48-7c58-40b0-bd5b-03b9f4bb551f" (UID: "c7320e48-7c58-40b0-bd5b-03b9f4bb551f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:01:01.055930 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.055903 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-kube-api-access-kvk5x" pod "c7320e48-7c58-40b0-bd5b-03b9f4bb551f" (UID: "c7320e48-7c58-40b0-bd5b-03b9f4bb551f"). InnerVolumeSpecName "kube-api-access-kvk5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:01.056076 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.056054 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-token" pod "c7320e48-7c58-40b0-bd5b-03b9f4bb551f" (UID: "c7320e48-7c58-40b0-bd5b-03b9f4bb551f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:01.056343 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.056321 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-envoy" pod "c7320e48-7c58-40b0-bd5b-03b9f4bb551f" (UID: "c7320e48-7c58-40b0-bd5b-03b9f4bb551f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:01.156041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.156010 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:01.156041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.156037 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:01.156041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.156047 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:01.156270 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.156055 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvk5x\" (UniqueName: \"kubernetes.io/projected/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-kube-api-access-kvk5x\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:01.156270 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.156065 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7320e48-7c58-40b0-bd5b-03b9f4bb551f-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:01.325891 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.325855 6202 generic.go:356] "Generic (PLEG): container finished" podID="c7320e48-7c58-40b0-bd5b-03b9f4bb551f" containerID="a93ec556596a6308c936c62b119db9f06348514a9c1c8723762d98d97f4707eb" exitCode=0 Oct 30 17:01:01.326076 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.325910 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" Oct 30 17:01:01.326076 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.325942 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" event={"ID":"c7320e48-7c58-40b0-bd5b-03b9f4bb551f","Type":"ContainerDied","Data":"a93ec556596a6308c936c62b119db9f06348514a9c1c8723762d98d97f4707eb"} Oct 30 17:01:01.326076 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.326004 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv" event={"ID":"c7320e48-7c58-40b0-bd5b-03b9f4bb551f","Type":"ContainerDied","Data":"09eba606f8738347fd1c504bad5379cd3100a85f406aa3b6fa1dff565da17fbb"} Oct 30 17:01:01.326076 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.326035 6202 scope.go:122] "RemoveContainer" containerID="a93ec556596a6308c936c62b119db9f06348514a9c1c8723762d98d97f4707eb" Oct 30 17:01:01.338619 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.338594 6202 scope.go:122] "RemoveContainer" containerID="d579cd1a477fd869c51061a833073b281c5b2e206e1911192dfc00a4835ac8b1" Oct 30 17:01:01.351082 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.351039 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:01.351520 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.351498 6202 scope.go:122] "RemoveContainer" containerID="a93ec556596a6308c936c62b119db9f06348514a9c1c8723762d98d97f4707eb" Oct 30 17:01:01.351829 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:01:01.351806 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a93ec556596a6308c936c62b119db9f06348514a9c1c8723762d98d97f4707eb\": container with ID starting with a93ec556596a6308c936c62b119db9f06348514a9c1c8723762d98d97f4707eb not found: ID does not exist" containerID="a93ec556596a6308c936c62b119db9f06348514a9c1c8723762d98d97f4707eb" Oct 30 17:01:01.351915 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.351837 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93ec556596a6308c936c62b119db9f06348514a9c1c8723762d98d97f4707eb"} err="failed to get container status \"a93ec556596a6308c936c62b119db9f06348514a9c1c8723762d98d97f4707eb\": rpc error: code = NotFound desc = could not find container \"a93ec556596a6308c936c62b119db9f06348514a9c1c8723762d98d97f4707eb\": container with ID starting with a93ec556596a6308c936c62b119db9f06348514a9c1c8723762d98d97f4707eb not found: ID does not exist" Oct 30 17:01:01.351915 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.351857 6202 scope.go:122] "RemoveContainer" containerID="d579cd1a477fd869c51061a833073b281c5b2e206e1911192dfc00a4835ac8b1" Oct 30 17:01:01.352169 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:01:01.352142 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d579cd1a477fd869c51061a833073b281c5b2e206e1911192dfc00a4835ac8b1\": container with ID starting with d579cd1a477fd869c51061a833073b281c5b2e206e1911192dfc00a4835ac8b1 not found: ID does not exist" containerID="d579cd1a477fd869c51061a833073b281c5b2e206e1911192dfc00a4835ac8b1" Oct 30 17:01:01.352244 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.352179 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d579cd1a477fd869c51061a833073b281c5b2e206e1911192dfc00a4835ac8b1"} err="failed to get container status \"d579cd1a477fd869c51061a833073b281c5b2e206e1911192dfc00a4835ac8b1\": rpc error: code = NotFound desc = could not find container \"d579cd1a477fd869c51061a833073b281c5b2e206e1911192dfc00a4835ac8b1\": container with ID starting with d579cd1a477fd869c51061a833073b281c5b2e206e1911192dfc00a4835ac8b1 not found: ID does not exist" Oct 30 17:01:01.420499 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.420463 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv"] Oct 30 17:01:01.424924 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:01.424898 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9c1e26-openshift-default-5dcf9bb6c7-rdqcv"] Oct 30 17:01:02.412870 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:02.412830 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7320e48-7c58-40b0-bd5b-03b9f4bb551f" path="/var/lib/kubelet/pods/c7320e48-7c58-40b0-bd5b-03b9f4bb551f/volumes" Oct 30 17:01:02.507937 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:02.507901 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:02.631433 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:02.631395 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:03.116310 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.116273 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:03.117716 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.117680 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" podUID="b229a8cf-cd61-4a38-be7b-d9ae564ac3a6" containerName="main" containerID="cri-o://b4c1685b048a5c527b3e4417252abeb4fd67583f1eb8fa17dad816b3e27acbf0" gracePeriod=45 Oct 30 17:01:03.259109 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.259056 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:03.386114 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.385966 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:03.395086 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.395053 6202 generic.go:356] "Generic (PLEG): container finished" podID="b229a8cf-cd61-4a38-be7b-d9ae564ac3a6" containerID="b4c1685b048a5c527b3e4417252abeb4fd67583f1eb8fa17dad816b3e27acbf0" exitCode=0 Oct 30 17:01:03.395343 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.395325 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" event={"ID":"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6","Type":"ContainerDied","Data":"b4c1685b048a5c527b3e4417252abeb4fd67583f1eb8fa17dad816b3e27acbf0"} Oct 30 17:01:03.472171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.472141 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 17:01:03.615711 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.615671 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-home\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-home\") pod \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " Oct 30 17:01:03.616302 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.615758 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-model-cache\") pod \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " Oct 30 17:01:03.616302 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.615900 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-tmp-dir\") pod \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " Oct 30 17:01:03.616302 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.616043 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-home" pod "b229a8cf-cd61-4a38-be7b-d9ae564ac3a6" (UID: "b229a8cf-cd61-4a38-be7b-d9ae564ac3a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:03.616302 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.616093 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-model-cache" pod "b229a8cf-cd61-4a38-be7b-d9ae564ac3a6" (UID: "b229a8cf-cd61-4a38-be7b-d9ae564ac3a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:03.616302 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.616101 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-tls-certs\") pod \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " Oct 30 17:01:03.616302 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.616178 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-dshm\") pod \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " Oct 30 17:01:03.616302 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.616193 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-tmp-dir" pod "b229a8cf-cd61-4a38-be7b-d9ae564ac3a6" (UID: "b229a8cf-cd61-4a38-be7b-d9ae564ac3a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:03.616302 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.616233 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-kube-api-access-dlbw2\" (UniqueName: \"kubernetes.io/projected/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-kube-api-access-dlbw2\") pod \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\" (UID: \"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6\") " Oct 30 17:01:03.617214 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.617189 6202 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-tmp-dir\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:03.617214 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.617216 6202 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-home\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:03.617409 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.617231 6202 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-model-cache\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:03.618739 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.618663 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-dshm" pod "b229a8cf-cd61-4a38-be7b-d9ae564ac3a6" (UID: "b229a8cf-cd61-4a38-be7b-d9ae564ac3a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:03.618739 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.618663 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-tls-certs" pod "b229a8cf-cd61-4a38-be7b-d9ae564ac3a6" (UID: "b229a8cf-cd61-4a38-be7b-d9ae564ac3a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:03.618927 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.618829 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-kube-api-access-dlbw2" pod "b229a8cf-cd61-4a38-be7b-d9ae564ac3a6" (UID: "b229a8cf-cd61-4a38-be7b-d9ae564ac3a6"). InnerVolumeSpecName "kube-api-access-dlbw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:03.734744 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.734600 6202 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-tls-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:03.735008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.734989 6202 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-dshm\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:03.735126 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:03.735113 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlbw2\" (UniqueName: \"kubernetes.io/projected/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6-kube-api-access-dlbw2\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:04.405532 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:04.405501 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" Oct 30 17:01:04.413376 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:04.413341 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc" event={"ID":"b229a8cf-cd61-4a38-be7b-d9ae564ac3a6","Type":"ContainerDied","Data":"c9e6d67cbece072e6922983c9d20da9e86d4f111c42a42c8a6c9f0edabd8f2a9"} Oct 30 17:01:04.413537 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:04.413389 6202 scope.go:122] "RemoveContainer" containerID="b4c1685b048a5c527b3e4417252abeb4fd67583f1eb8fa17dad816b3e27acbf0" Oct 30 17:01:04.461583 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:04.461542 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc"] Oct 30 17:01:04.472832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:04.472799 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7c4b84/auth-test-model-7c4b84-kserve-59796579b9-95khc"] Oct 30 17:01:04.999391 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:04.997922 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:06.412907 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:06.412877 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b229a8cf-cd61-4a38-be7b-d9ae564ac3a6" path="/var/lib/kubelet/pods/b229a8cf-cd61-4a38-be7b-d9ae564ac3a6/volumes" Oct 30 17:01:06.781659 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:06.781570 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:08.380001 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:08.379966 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:08.950691 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:08.950297 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:09.115820 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.115289 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd"] Oct 30 17:01:09.116481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.116453 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c7320e48-7c58-40b0-bd5b-03b9f4bb551f" containerName="istio-proxy" Oct 30 17:01:09.116481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.116479 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7320e48-7c58-40b0-bd5b-03b9f4bb551f" containerName="istio-proxy" Oct 30 17:01:09.116701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.116511 6202 cpu_manager.go:414] "containerMap: removing container" podUID="b229a8cf-cd61-4a38-be7b-d9ae564ac3a6" containerName="main" Oct 30 17:01:09.116701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.116520 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b229a8cf-cd61-4a38-be7b-d9ae564ac3a6" containerName="main" Oct 30 17:01:09.116701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.116597 6202 cpu_manager.go:414] "containerMap: removing container" podUID="4cead7b4-5a36-4ed4-8550-3c454ace71a3" containerName="istio-proxy" Oct 30 17:01:09.116701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.116605 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cead7b4-5a36-4ed4-8550-3c454ace71a3" containerName="istio-proxy" Oct 30 17:01:09.116982 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.116824 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7320e48-7c58-40b0-bd5b-03b9f4bb551f" containerName="istio-proxy" Oct 30 17:01:09.116982 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.116843 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cead7b4-5a36-4ed4-8550-3c454ace71a3" containerName="istio-proxy" Oct 30 17:01:09.116982 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.116858 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7320e48-7c58-40b0-bd5b-03b9f4bb551f" containerName="istio-proxy" Oct 30 17:01:09.116982 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.116872 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="b229a8cf-cd61-4a38-be7b-d9ae564ac3a6" containerName="main" Oct 30 17:01:09.122619 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.122596 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd" Oct 30 17:01:09.134573 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.134540 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a06535-5f78-45d3-bed8-06dccf6414b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd\" (UID: \"93a06535-5f78-45d3-bed8-06dccf6414b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd" Oct 30 17:01:09.134935 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.134910 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw5zs\" (UniqueName: \"kubernetes.io/projected/93a06535-5f78-45d3-bed8-06dccf6414b9-kube-api-access-vw5zs\") pod \"maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd\" (UID: \"93a06535-5f78-45d3-bed8-06dccf6414b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd" Oct 30 17:01:09.135124 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.135105 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a06535-5f78-45d3-bed8-06dccf6414b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd\" (UID: \"93a06535-5f78-45d3-bed8-06dccf6414b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd" Oct 30 17:01:09.150084 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.149731 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:01:09.150084 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.149845 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:01:09.150401 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.150183 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd"] Oct 30 17:01:09.238293 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.238181 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw5zs\" (UniqueName: \"kubernetes.io/projected/93a06535-5f78-45d3-bed8-06dccf6414b9-kube-api-access-vw5zs\") pod \"maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd\" (UID: \"93a06535-5f78-45d3-bed8-06dccf6414b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd" Oct 30 17:01:09.238293 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.238245 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a06535-5f78-45d3-bed8-06dccf6414b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd\" (UID: \"93a06535-5f78-45d3-bed8-06dccf6414b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd" Oct 30 17:01:09.238561 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.238438 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a06535-5f78-45d3-bed8-06dccf6414b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd\" (UID: \"93a06535-5f78-45d3-bed8-06dccf6414b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd" Oct 30 17:01:09.239150 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.239118 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a06535-5f78-45d3-bed8-06dccf6414b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd\" (UID: \"93a06535-5f78-45d3-bed8-06dccf6414b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd" Oct 30 17:01:09.241246 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.241219 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a06535-5f78-45d3-bed8-06dccf6414b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd\" (UID: \"93a06535-5f78-45d3-bed8-06dccf6414b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd" Oct 30 17:01:09.277494 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.277419 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw5zs\" (UniqueName: \"kubernetes.io/projected/93a06535-5f78-45d3-bed8-06dccf6414b9-kube-api-access-vw5zs\") pod \"maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd\" (UID: \"93a06535-5f78-45d3-bed8-06dccf6414b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd" Oct 30 17:01:09.461118 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.461080 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd" Oct 30 17:01:09.785076 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:09.785044 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd"] Oct 30 17:01:10.114966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:10.114926 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:10.488301 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:10.488257 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd" event={"ID":"93a06535-5f78-45d3-bed8-06dccf6414b9","Type":"ContainerStarted","Data":"6248c45be4c073d36828a7d3be3db72d0d33d04c33daa762a1bccd59694ac3aa"} Oct 30 17:01:10.488717 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:10.488321 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd" event={"ID":"93a06535-5f78-45d3-bed8-06dccf6414b9","Type":"ContainerStarted","Data":"b5243b339018c47602f56480525100ec0cf5627c16fe70f42d8f27a41656c84e"} Oct 30 17:01:10.530281 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:10.530221 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd" podStartSLOduration=1.5302072020000002 podStartE2EDuration="1.530207202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:10.529619945 +0000 UTC m=+2384.240074689" watchObservedRunningTime="2026-10-30 17:01:10.530207202 +0000 UTC m=+2384.240661945" Oct 30 17:01:11.032734 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:11.032616 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v"] Oct 30 17:01:11.044432 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:11.044401 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b38945ae-stj5v"] Oct 30 17:01:11.498741 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:11.498703 6202 generic.go:356] "Generic (PLEG): container finished" podID="93a06535-5f78-45d3-bed8-06dccf6414b9" containerID="6248c45be4c073d36828a7d3be3db72d0d33d04c33daa762a1bccd59694ac3aa" exitCode=0 Oct 30 17:01:11.499208 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:11.498797 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd" event={"ID":"93a06535-5f78-45d3-bed8-06dccf6414b9","Type":"ContainerDied","Data":"6248c45be4c073d36828a7d3be3db72d0d33d04c33daa762a1bccd59694ac3aa"} Oct 30 17:01:11.886538 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:11.886500 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:12.412317 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:12.412280 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61969530-e89d-44aa-801b-78ad6fcc61b1" path="/var/lib/kubelet/pods/61969530-e89d-44aa-801b-78ad6fcc61b1/volumes" Oct 30 17:01:12.699555 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:12.699529 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd" Oct 30 17:01:12.818217 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:12.818184 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93a06535-5f78-45d3-bed8-06dccf6414b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a06535-5f78-45d3-bed8-06dccf6414b9-maas-api-xks-ca\") pod \"93a06535-5f78-45d3-bed8-06dccf6414b9\" (UID: \"93a06535-5f78-45d3-bed8-06dccf6414b9\") " Oct 30 17:01:12.818373 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:12.818331 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a06535-5f78-45d3-bed8-06dccf6414b9-kube-api-access-vw5zs\" (UniqueName: \"kubernetes.io/projected/93a06535-5f78-45d3-bed8-06dccf6414b9-kube-api-access-vw5zs\") pod \"93a06535-5f78-45d3-bed8-06dccf6414b9\" (UID: \"93a06535-5f78-45d3-bed8-06dccf6414b9\") " Oct 30 17:01:12.818522 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:12.818505 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a06535-5f78-45d3-bed8-06dccf6414b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a06535-5f78-45d3-bed8-06dccf6414b9-maas-api-openshift-ca\") pod \"93a06535-5f78-45d3-bed8-06dccf6414b9\" (UID: \"93a06535-5f78-45d3-bed8-06dccf6414b9\") " Oct 30 17:01:12.818850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:12.818819 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a06535-5f78-45d3-bed8-06dccf6414b9-maas-api-openshift-ca" pod "93a06535-5f78-45d3-bed8-06dccf6414b9" (UID: "93a06535-5f78-45d3-bed8-06dccf6414b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:12.819098 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:12.819080 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a06535-5f78-45d3-bed8-06dccf6414b9-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:12.820430 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:12.820396 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a06535-5f78-45d3-bed8-06dccf6414b9-maas-api-xks-ca" pod "93a06535-5f78-45d3-bed8-06dccf6414b9" (UID: "93a06535-5f78-45d3-bed8-06dccf6414b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:12.820579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:12.820562 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a06535-5f78-45d3-bed8-06dccf6414b9-kube-api-access-vw5zs" pod "93a06535-5f78-45d3-bed8-06dccf6414b9" (UID: "93a06535-5f78-45d3-bed8-06dccf6414b9"). InnerVolumeSpecName "kube-api-access-vw5zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:12.920754 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:12.920673 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw5zs\" (UniqueName: \"kubernetes.io/projected/93a06535-5f78-45d3-bed8-06dccf6414b9-kube-api-access-vw5zs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:12.920754 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:12.920703 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a06535-5f78-45d3-bed8-06dccf6414b9-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:13.366722 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:13.366666 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:13.490030 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:13.490000 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:13.517182 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:13.517152 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd" Oct 30 17:01:13.517354 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:13.517182 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd" event={"ID":"93a06535-5f78-45d3-bed8-06dccf6414b9","Type":"ContainerDied","Data":"b5243b339018c47602f56480525100ec0cf5627c16fe70f42d8f27a41656c84e"} Oct 30 17:01:13.517354 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:13.517226 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5243b339018c47602f56480525100ec0cf5627c16fe70f42d8f27a41656c84e" Oct 30 17:01:14.071261 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:01:14.071215 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:01:14.876074 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:14.876038 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:15.041688 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:15.041637 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:16.877028 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:16.876992 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:18.569596 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:18.569560 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:18.953308 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:18.953272 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:19.733717 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:19.733677 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:19.849810 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:19.849753 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:20.363658 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:20.363620 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:21.325362 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:21.325326 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj"] Oct 30 17:01:21.326211 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:21.326175 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" podUID="47902ba0-362e-40c1-9589-c2181308e89a" containerName="istio-proxy" containerID="cri-o://f21ac59678585f01e3ee373b97fe1cb50511979baa4559d7e400aaa6459eb741" gracePeriod=30 Oct 30 17:01:22.773137 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:22.773104 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:24.593417 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:24.593383 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:26.265584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:26.265541 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:26.682204 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:26.682170 6202 generic.go:356] "Generic (PLEG): container finished" podID="47902ba0-362e-40c1-9589-c2181308e89a" containerID="f21ac59678585f01e3ee373b97fe1cb50511979baa4559d7e400aaa6459eb741" exitCode=0 Oct 30 17:01:26.682400 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:26.682257 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" event={"ID":"47902ba0-362e-40c1-9589-c2181308e89a","Type":"ContainerDied","Data":"f21ac59678585f01e3ee373b97fe1cb50511979baa4559d7e400aaa6459eb741"} Oct 30 17:01:27.036588 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.036560 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 17:01:27.109996 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.109641 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47902ba0-362e-40c1-9589-c2181308e89a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47902ba0-362e-40c1-9589-c2181308e89a-istio-podinfo\") pod \"47902ba0-362e-40c1-9589-c2181308e89a\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " Oct 30 17:01:27.110199 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.110112 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-istio-envoy\") pod \"47902ba0-362e-40c1-9589-c2181308e89a\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " Oct 30 17:01:27.110547 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.110523 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-workload-socket\") pod \"47902ba0-362e-40c1-9589-c2181308e89a\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " Oct 30 17:01:27.110865 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.110839 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-workload-socket" pod "47902ba0-362e-40c1-9589-c2181308e89a" (UID: "47902ba0-362e-40c1-9589-c2181308e89a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:27.111096 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.111078 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47902ba0-362e-40c1-9589-c2181308e89a-istio-token\" (UniqueName: \"kubernetes.io/projected/47902ba0-362e-40c1-9589-c2181308e89a-istio-token\") pod \"47902ba0-362e-40c1-9589-c2181308e89a\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " Oct 30 17:01:27.111441 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.111420 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-istio-data\") pod \"47902ba0-362e-40c1-9589-c2181308e89a\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " Oct 30 17:01:27.111673 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.111652 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-istio-data" pod "47902ba0-362e-40c1-9589-c2181308e89a" (UID: "47902ba0-362e-40c1-9589-c2181308e89a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:27.111832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.111813 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47902ba0-362e-40c1-9589-c2181308e89a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47902ba0-362e-40c1-9589-c2181308e89a-istiod-ca-cert\") pod \"47902ba0-362e-40c1-9589-c2181308e89a\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " Oct 30 17:01:27.111931 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.111913 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47902ba0-362e-40c1-9589-c2181308e89a-kube-api-access-ggt4b\" (UniqueName: \"kubernetes.io/projected/47902ba0-362e-40c1-9589-c2181308e89a-kube-api-access-ggt4b\") pod \"47902ba0-362e-40c1-9589-c2181308e89a\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " Oct 30 17:01:27.111998 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.111947 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-workload-certs\") pod \"47902ba0-362e-40c1-9589-c2181308e89a\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " Oct 30 17:01:27.111998 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.111977 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-credential-socket\") pod \"47902ba0-362e-40c1-9589-c2181308e89a\" (UID: \"47902ba0-362e-40c1-9589-c2181308e89a\") " Oct 30 17:01:27.112960 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.112935 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:27.113079 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.112962 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:27.113162 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.113136 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-credential-socket" pod "47902ba0-362e-40c1-9589-c2181308e89a" (UID: "47902ba0-362e-40c1-9589-c2181308e89a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:27.113414 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.113395 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47902ba0-362e-40c1-9589-c2181308e89a-istiod-ca-cert" pod "47902ba0-362e-40c1-9589-c2181308e89a" (UID: "47902ba0-362e-40c1-9589-c2181308e89a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:27.113823 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.113800 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-workload-certs" pod "47902ba0-362e-40c1-9589-c2181308e89a" (UID: "47902ba0-362e-40c1-9589-c2181308e89a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:27.118533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.116175 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47902ba0-362e-40c1-9589-c2181308e89a-istio-podinfo" pod "47902ba0-362e-40c1-9589-c2181308e89a" (UID: "47902ba0-362e-40c1-9589-c2181308e89a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:01:27.118533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.116579 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-istio-envoy" pod "47902ba0-362e-40c1-9589-c2181308e89a" (UID: "47902ba0-362e-40c1-9589-c2181308e89a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:27.118533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.117915 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47902ba0-362e-40c1-9589-c2181308e89a-istio-token" pod "47902ba0-362e-40c1-9589-c2181308e89a" (UID: "47902ba0-362e-40c1-9589-c2181308e89a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:27.118533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.118192 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47902ba0-362e-40c1-9589-c2181308e89a-kube-api-access-ggt4b" pod "47902ba0-362e-40c1-9589-c2181308e89a" (UID: "47902ba0-362e-40c1-9589-c2181308e89a"). InnerVolumeSpecName "kube-api-access-ggt4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:27.215012 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.214955 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47902ba0-362e-40c1-9589-c2181308e89a-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:27.215012 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.214991 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47902ba0-362e-40c1-9589-c2181308e89a-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:27.215012 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.215007 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggt4b\" (UniqueName: \"kubernetes.io/projected/47902ba0-362e-40c1-9589-c2181308e89a-kube-api-access-ggt4b\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:27.215366 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.215020 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:27.215366 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.215034 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:27.215366 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.215047 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47902ba0-362e-40c1-9589-c2181308e89a-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:27.215366 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.215059 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47902ba0-362e-40c1-9589-c2181308e89a-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:27.692657 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.692622 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" Oct 30 17:01:27.693127 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.692655 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj" event={"ID":"47902ba0-362e-40c1-9589-c2181308e89a","Type":"ContainerDied","Data":"17c520598ac215898a66d414a18bfb597894eb17639d637f869b725468e80975"} Oct 30 17:01:27.693127 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.692719 6202 scope.go:122] "RemoveContainer" containerID="f21ac59678585f01e3ee373b97fe1cb50511979baa4559d7e400aaa6459eb741" Oct 30 17:01:27.775138 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.775101 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj"] Oct 30 17:01:27.775362 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.775336 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:27.785872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:27.785848 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8cxrj"] Oct 30 17:01:28.413218 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:28.413177 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47902ba0-362e-40c1-9589-c2181308e89a" path="/var/lib/kubelet/pods/47902ba0-362e-40c1-9589-c2181308e89a/volumes" Oct 30 17:01:28.437030 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:28.436998 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:28.678101 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:28.678021 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:28.707575 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:28.707543 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd"] Oct 30 17:01:28.719278 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:28.719224 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ea77d8-jzmbd"] Oct 30 17:01:28.790668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:28.790624 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:30.412445 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:30.412413 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a06535-5f78-45d3-bed8-06dccf6414b9" path="/var/lib/kubelet/pods/93a06535-5f78-45d3-bed8-06dccf6414b9/volumes" Oct 30 17:01:30.717347 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:30.717268 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:32.427948 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:32.427916 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:32.438219 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:32.438192 6202 scope.go:122] "RemoveContainer" containerID="dbb1c351fa2afcdb6a90414cfa5f868102d346e6112f4ffd8604e15070c2cc53" Oct 30 17:01:32.449942 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:32.449910 6202 scope.go:122] "RemoveContainer" containerID="9bb9461572ce3c78001e17323b934618d651b9b0f116399ec909b09bc3f0a69e" Oct 30 17:01:32.460283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:32.460256 6202 scope.go:122] "RemoveContainer" containerID="a1d8db655965566d92b3ac4d43003fbd35d49c82dace75769e2e65d1aba800bc" Oct 30 17:01:34.180533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:34.180494 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:35.849975 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:35.849941 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:37.660803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:37.660736 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:39.313714 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:39.313676 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:39.500138 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:39.500107 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:40.941306 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:40.941271 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:41.153685 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:41.153640 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:41.264673 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:41.264565 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:42.804312 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:42.802181 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:43.393677 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:43.393575 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:43.552488 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:43.552450 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:43.571393 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:43.571364 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq"] Oct 30 17:01:43.592224 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:43.592181 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-689899-96kwq"] Oct 30 17:01:43.677355 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:43.677278 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:44.071076 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:01:44.070984 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:01:44.104111 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:44.104066 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:44.413566 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:44.413484 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fabd6007-9258-4203-82ec-b4c024c9102e" path="/var/lib/kubelet/pods/fabd6007-9258-4203-82ec-b4c024c9102e/volumes" Oct 30 17:01:45.313912 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.313879 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm"] Oct 30 17:01:45.314641 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.314626 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c7320e48-7c58-40b0-bd5b-03b9f4bb551f" containerName="istio-proxy" Oct 30 17:01:45.314700 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.314642 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7320e48-7c58-40b0-bd5b-03b9f4bb551f" containerName="istio-proxy" Oct 30 17:01:45.314700 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.314658 6202 cpu_manager.go:414] "containerMap: removing container" podUID="47902ba0-362e-40c1-9589-c2181308e89a" containerName="istio-proxy" Oct 30 17:01:45.314700 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.314662 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47902ba0-362e-40c1-9589-c2181308e89a" containerName="istio-proxy" Oct 30 17:01:45.314829 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.314702 6202 cpu_manager.go:414] "containerMap: removing container" podUID="93a06535-5f78-45d3-bed8-06dccf6414b9" containerName="revoke-keys" Oct 30 17:01:45.314829 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.314707 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a06535-5f78-45d3-bed8-06dccf6414b9" containerName="revoke-keys" Oct 30 17:01:45.314896 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.314834 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="47902ba0-362e-40c1-9589-c2181308e89a" containerName="istio-proxy" Oct 30 17:01:45.314896 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.314851 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a06535-5f78-45d3-bed8-06dccf6414b9" containerName="revoke-keys" Oct 30 17:01:45.320817 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.320799 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm" Oct 30 17:01:45.342150 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.342118 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm"] Oct 30 17:01:45.347497 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.347472 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:01:45.347626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.347560 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:01:45.388509 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.388471 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01c02b63-xf7nm\" (UID: \"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm" Oct 30 17:01:45.388755 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.388537 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01c02b63-xf7nm\" (UID: \"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm" Oct 30 17:01:45.388755 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.388716 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp4r5\" (UniqueName: \"kubernetes.io/projected/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-kube-api-access-cp4r5\") pod \"maas-api-revoke-sub-01c02b63-xf7nm\" (UID: \"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm" Oct 30 17:01:45.491035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.490613 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01c02b63-xf7nm\" (UID: \"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm" Oct 30 17:01:45.491035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.490708 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01c02b63-xf7nm\" (UID: \"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm" Oct 30 17:01:45.491035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.490879 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp4r5\" (UniqueName: \"kubernetes.io/projected/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-kube-api-access-cp4r5\") pod \"maas-api-revoke-sub-01c02b63-xf7nm\" (UID: \"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm" Oct 30 17:01:45.491546 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.491512 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01c02b63-xf7nm\" (UID: \"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm" Oct 30 17:01:45.494325 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.494299 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01c02b63-xf7nm\" (UID: \"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm" Oct 30 17:01:45.523316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.523288 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp4r5\" (UniqueName: \"kubernetes.io/projected/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-kube-api-access-cp4r5\") pod \"maas-api-revoke-sub-01c02b63-xf7nm\" (UID: \"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm" Oct 30 17:01:45.612790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.612189 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:45.639860 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.639821 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm" Oct 30 17:01:45.963488 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:01:45.963450 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1bf2ff_180e_4b58_bd3e_078ae6e8201b.slice/crio-6db55912776c41fcbf404635dc106e22a5e615ac63b0bc84f0bd913529855745 WatchSource:0}: Error finding container 6db55912776c41fcbf404635dc106e22a5e615ac63b0bc84f0bd913529855745: Status 404 returned error can't find the container with id 6db55912776c41fcbf404635dc106e22a5e615ac63b0bc84f0bd913529855745 Oct 30 17:01:45.964545 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:45.964518 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm"] Oct 30 17:01:46.157391 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:46.157357 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:46.863676 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:46.863635 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm" event={"ID":"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b","Type":"ContainerStarted","Data":"072c6cb23b019931b588a13b20b07e6791ad6d820c78efec4de9889cd9e8757c"} Oct 30 17:01:46.864203 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:46.863706 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm" event={"ID":"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b","Type":"ContainerStarted","Data":"6db55912776c41fcbf404635dc106e22a5e615ac63b0bc84f0bd913529855745"} Oct 30 17:01:46.906413 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:46.906357 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm" podStartSLOduration=1.9063410140000001 podStartE2EDuration="1.906341014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:46.903232481 +0000 UTC m=+2420.613687225" watchObservedRunningTime="2026-10-30 17:01:46.906341014 +0000 UTC m=+2420.616795759" Oct 30 17:01:47.089075 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:47.089037 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:47.201529 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:47.201422 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:47.426828 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:47.426793 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg"] Oct 30 17:01:47.427317 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:47.427287 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" podUID="323c4d64-a380-4e62-9779-1a1a87ae3308" containerName="istio-proxy" containerID="cri-o://dde14991980aa79145cacb4f5218fd017677b3d0af5c194ae9937a3467003515" gracePeriod=30 Oct 30 17:01:47.501350 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:47.501254 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:47.648792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:47.634624 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:47.874389 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:47.874353 6202 generic.go:356] "Generic (PLEG): container finished" podID="4a1bf2ff-180e-4b58-bd3e-078ae6e8201b" containerID="072c6cb23b019931b588a13b20b07e6791ad6d820c78efec4de9889cd9e8757c" exitCode=0 Oct 30 17:01:47.874932 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:47.874439 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm" event={"ID":"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b","Type":"ContainerDied","Data":"072c6cb23b019931b588a13b20b07e6791ad6d820c78efec4de9889cd9e8757c"} Oct 30 17:01:47.982996 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:47.982959 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6"] Oct 30 17:01:47.989788 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:47.989743 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6" Oct 30 17:01:48.005533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:48.005499 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6"] Oct 30 17:01:48.045509 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:48.045469 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bbe06bf-8e56-4399-9969-41ade4c450ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6\" (UID: \"9bbe06bf-8e56-4399-9969-41ade4c450ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6" Oct 30 17:01:48.046239 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:48.046211 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prkjb\" (UniqueName: \"kubernetes.io/projected/9bbe06bf-8e56-4399-9969-41ade4c450ef-kube-api-access-prkjb\") pod \"maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6\" (UID: \"9bbe06bf-8e56-4399-9969-41ade4c450ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6" Oct 30 17:01:48.047080 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:48.047057 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bbe06bf-8e56-4399-9969-41ade4c450ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6\" (UID: \"9bbe06bf-8e56-4399-9969-41ade4c450ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6" Oct 30 17:01:48.149581 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:48.149483 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bbe06bf-8e56-4399-9969-41ade4c450ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6\" (UID: \"9bbe06bf-8e56-4399-9969-41ade4c450ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6" Oct 30 17:01:48.149756 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:48.149601 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prkjb\" (UniqueName: \"kubernetes.io/projected/9bbe06bf-8e56-4399-9969-41ade4c450ef-kube-api-access-prkjb\") pod \"maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6\" (UID: \"9bbe06bf-8e56-4399-9969-41ade4c450ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6" Oct 30 17:01:48.149756 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:48.149748 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bbe06bf-8e56-4399-9969-41ade4c450ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6\" (UID: \"9bbe06bf-8e56-4399-9969-41ade4c450ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6" Oct 30 17:01:48.150430 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:48.150407 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bbe06bf-8e56-4399-9969-41ade4c450ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6\" (UID: \"9bbe06bf-8e56-4399-9969-41ade4c450ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6" Oct 30 17:01:48.152309 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:48.152283 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bbe06bf-8e56-4399-9969-41ade4c450ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6\" (UID: \"9bbe06bf-8e56-4399-9969-41ade4c450ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6" Oct 30 17:01:48.173675 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:48.173643 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prkjb\" (UniqueName: \"kubernetes.io/projected/9bbe06bf-8e56-4399-9969-41ade4c450ef-kube-api-access-prkjb\") pod \"maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6\" (UID: \"9bbe06bf-8e56-4399-9969-41ade4c450ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6" Oct 30 17:01:48.315199 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:48.315161 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6" Oct 30 17:01:48.625166 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:48.625134 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6"] Oct 30 17:01:48.628681 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:01:48.628651 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbe06bf_8e56_4399_9969_41ade4c450ef.slice/crio-6385e2a3c9c7a1738d80efd07d924be8f8195e8ecdb034446d8323ca3656c06e WatchSource:0}: Error finding container 6385e2a3c9c7a1738d80efd07d924be8f8195e8ecdb034446d8323ca3656c06e: Status 404 returned error can't find the container with id 6385e2a3c9c7a1738d80efd07d924be8f8195e8ecdb034446d8323ca3656c06e Oct 30 17:01:48.888170 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:48.888084 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6" event={"ID":"9bbe06bf-8e56-4399-9969-41ade4c450ef","Type":"ContainerStarted","Data":"6385e2a3c9c7a1738d80efd07d924be8f8195e8ecdb034446d8323ca3656c06e"} Oct 30 17:01:49.112915 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:49.112878 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm" Oct 30 17:01:49.169881 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:49.169798 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-kube-api-access-cp4r5\" (UniqueName: \"kubernetes.io/projected/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-kube-api-access-cp4r5\") pod \"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b\" (UID: \"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b\") " Oct 30 17:01:49.170031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:49.169890 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-maas-api-xks-ca\") pod \"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b\" (UID: \"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b\") " Oct 30 17:01:49.170107 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:49.170088 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-maas-api-openshift-ca\") pod \"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b\" (UID: \"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b\") " Oct 30 17:01:49.170638 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:49.170607 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-maas-api-openshift-ca" pod "4a1bf2ff-180e-4b58-bd3e-078ae6e8201b" (UID: "4a1bf2ff-180e-4b58-bd3e-078ae6e8201b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:49.171433 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:49.171032 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:49.173245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:49.173217 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-maas-api-xks-ca" pod "4a1bf2ff-180e-4b58-bd3e-078ae6e8201b" (UID: "4a1bf2ff-180e-4b58-bd3e-078ae6e8201b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:49.173732 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:49.173694 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-kube-api-access-cp4r5" pod "4a1bf2ff-180e-4b58-bd3e-078ae6e8201b" (UID: "4a1bf2ff-180e-4b58-bd3e-078ae6e8201b"). InnerVolumeSpecName "kube-api-access-cp4r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:49.272453 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:49.272414 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp4r5\" (UniqueName: \"kubernetes.io/projected/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-kube-api-access-cp4r5\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:49.272453 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:49.272448 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:49.535198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:49.535124 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:49.898101 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:49.898075 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm" Oct 30 17:01:49.898604 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:49.898113 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm" event={"ID":"4a1bf2ff-180e-4b58-bd3e-078ae6e8201b","Type":"ContainerDied","Data":"6db55912776c41fcbf404635dc106e22a5e615ac63b0bc84f0bd913529855745"} Oct 30 17:01:49.898604 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:49.898170 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6db55912776c41fcbf404635dc106e22a5e615ac63b0bc84f0bd913529855745" Oct 30 17:01:49.900088 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:49.900053 6202 generic.go:356] "Generic (PLEG): container finished" podID="9bbe06bf-8e56-4399-9969-41ade4c450ef" containerID="91299af34799ddd367f45b7112f35af8d88d4d77d930232978752b5993968812" exitCode=0 Oct 30 17:01:49.900223 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:49.900125 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6" event={"ID":"9bbe06bf-8e56-4399-9969-41ade4c450ef","Type":"ContainerDied","Data":"91299af34799ddd367f45b7112f35af8d88d4d77d930232978752b5993968812"} Oct 30 17:01:50.649223 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:50.649180 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:50.757187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:50.757151 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:50.988792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:50.984862 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x"] Oct 30 17:01:50.996307 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:50.992954 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" podUID="051d25d4-9b2f-482e-a9f7-1c630eec241c" containerName="istio-proxy" containerID="cri-o://ac831c619ea4ce9946bad503fcde433894b9b9af5a4ccb0c894c8ad233ee147b" gracePeriod=30 Oct 30 17:01:50.996307 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:50.993951 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9"] Oct 30 17:01:50.996307 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:50.994165 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" podUID="33abe911-bb76-44ba-a61e-6c8c87c95bb8" containerName="istio-proxy" containerID="cri-o://ea18800c7daca6729a6b9b53ff51414e4f1a738ee007693550e4254796bbb51b" gracePeriod=30 Oct 30 17:01:51.000306 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:51.000281 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r"] Oct 30 17:01:51.000717 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:51.000688 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" podUID="318e8454-0cf7-485e-b774-9ce1b368cac6" containerName="istio-proxy" containerID="cri-o://bb4aa15c5ed737c36f2aa3d71aca44d0126b5debb4eac2aa352fde9ff2989172" gracePeriod=30 Oct 30 17:01:51.029493 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:51.029401 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:51.132994 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:51.132968 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6" Oct 30 17:01:51.228839 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:51.220338 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bbe06bf-8e56-4399-9969-41ade4c450ef-kube-api-access-prkjb\" (UniqueName: \"kubernetes.io/projected/9bbe06bf-8e56-4399-9969-41ade4c450ef-kube-api-access-prkjb\") pod \"9bbe06bf-8e56-4399-9969-41ade4c450ef\" (UID: \"9bbe06bf-8e56-4399-9969-41ade4c450ef\") " Oct 30 17:01:51.228839 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:51.220453 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bbe06bf-8e56-4399-9969-41ade4c450ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bbe06bf-8e56-4399-9969-41ade4c450ef-maas-api-openshift-ca\") pod \"9bbe06bf-8e56-4399-9969-41ade4c450ef\" (UID: \"9bbe06bf-8e56-4399-9969-41ade4c450ef\") " Oct 30 17:01:51.228839 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:51.220509 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bbe06bf-8e56-4399-9969-41ade4c450ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bbe06bf-8e56-4399-9969-41ade4c450ef-maas-api-xks-ca\") pod \"9bbe06bf-8e56-4399-9969-41ade4c450ef\" (UID: \"9bbe06bf-8e56-4399-9969-41ade4c450ef\") " Oct 30 17:01:51.228839 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:51.221867 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bbe06bf-8e56-4399-9969-41ade4c450ef-maas-api-openshift-ca" pod "9bbe06bf-8e56-4399-9969-41ade4c450ef" (UID: "9bbe06bf-8e56-4399-9969-41ade4c450ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:51.290400 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:51.290312 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bbe06bf-8e56-4399-9969-41ade4c450ef-kube-api-access-prkjb" pod "9bbe06bf-8e56-4399-9969-41ade4c450ef" (UID: "9bbe06bf-8e56-4399-9969-41ade4c450ef"). InnerVolumeSpecName "kube-api-access-prkjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:51.290845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:51.290817 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bbe06bf-8e56-4399-9969-41ade4c450ef-maas-api-xks-ca" pod "9bbe06bf-8e56-4399-9969-41ade4c450ef" (UID: "9bbe06bf-8e56-4399-9969-41ade4c450ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:51.323530 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:51.323484 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prkjb\" (UniqueName: \"kubernetes.io/projected/9bbe06bf-8e56-4399-9969-41ade4c450ef-kube-api-access-prkjb\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:51.323530 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:51.323526 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bbe06bf-8e56-4399-9969-41ade4c450ef-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:51.323530 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:51.323540 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bbe06bf-8e56-4399-9969-41ade4c450ef-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:51.485366 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:51.485277 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:51.921000 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:51.920969 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6" Oct 30 17:01:51.921154 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:51.921002 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6" event={"ID":"9bbe06bf-8e56-4399-9969-41ade4c450ef","Type":"ContainerDied","Data":"6385e2a3c9c7a1738d80efd07d924be8f8195e8ecdb034446d8323ca3656c06e"} Oct 30 17:01:51.921154 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:51.921080 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6385e2a3c9c7a1738d80efd07d924be8f8195e8ecdb034446d8323ca3656c06e" Oct 30 17:01:52.753513 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.753483 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 17:01:52.848073 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.848042 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-podinfo\") pod \"323c4d64-a380-4e62-9779-1a1a87ae3308\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " Oct 30 17:01:52.848289 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.848128 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323c4d64-a380-4e62-9779-1a1a87ae3308-kube-api-access-tcq79\" (UniqueName: \"kubernetes.io/projected/323c4d64-a380-4e62-9779-1a1a87ae3308-kube-api-access-tcq79\") pod \"323c4d64-a380-4e62-9779-1a1a87ae3308\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " Oct 30 17:01:52.848289 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.848172 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-credential-socket\") pod \"323c4d64-a380-4e62-9779-1a1a87ae3308\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " Oct 30 17:01:52.848289 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.848220 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-workload-socket\") pod \"323c4d64-a380-4e62-9779-1a1a87ae3308\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " Oct 30 17:01:52.848289 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.848243 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-token\" (UniqueName: \"kubernetes.io/projected/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-token\") pod \"323c4d64-a380-4e62-9779-1a1a87ae3308\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " Oct 30 17:01:52.848289 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.848271 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/323c4d64-a380-4e62-9779-1a1a87ae3308-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323c4d64-a380-4e62-9779-1a1a87ae3308-istiod-ca-cert\") pod \"323c4d64-a380-4e62-9779-1a1a87ae3308\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " Oct 30 17:01:52.848547 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.848317 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-envoy\") pod \"323c4d64-a380-4e62-9779-1a1a87ae3308\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " Oct 30 17:01:52.848547 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.848477 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-data\") pod \"323c4d64-a380-4e62-9779-1a1a87ae3308\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " Oct 30 17:01:52.848648 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.848582 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-workload-certs\") pod \"323c4d64-a380-4e62-9779-1a1a87ae3308\" (UID: \"323c4d64-a380-4e62-9779-1a1a87ae3308\") " Oct 30 17:01:52.848743 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.848715 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-workload-socket" pod "323c4d64-a380-4e62-9779-1a1a87ae3308" (UID: "323c4d64-a380-4e62-9779-1a1a87ae3308"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:52.849859 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.849290 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323c4d64-a380-4e62-9779-1a1a87ae3308-istiod-ca-cert" pod "323c4d64-a380-4e62-9779-1a1a87ae3308" (UID: "323c4d64-a380-4e62-9779-1a1a87ae3308"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:52.849859 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.849457 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:52.849859 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.849812 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-workload-certs" pod "323c4d64-a380-4e62-9779-1a1a87ae3308" (UID: "323c4d64-a380-4e62-9779-1a1a87ae3308"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:52.849859 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.849826 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-data" pod "323c4d64-a380-4e62-9779-1a1a87ae3308" (UID: "323c4d64-a380-4e62-9779-1a1a87ae3308"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:52.850376 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.850358 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-credential-socket" pod "323c4d64-a380-4e62-9779-1a1a87ae3308" (UID: "323c4d64-a380-4e62-9779-1a1a87ae3308"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:52.852149 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.852121 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-envoy" pod "323c4d64-a380-4e62-9779-1a1a87ae3308" (UID: "323c4d64-a380-4e62-9779-1a1a87ae3308"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:52.852238 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.852195 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323c4d64-a380-4e62-9779-1a1a87ae3308-kube-api-access-tcq79" pod "323c4d64-a380-4e62-9779-1a1a87ae3308" (UID: "323c4d64-a380-4e62-9779-1a1a87ae3308"). InnerVolumeSpecName "kube-api-access-tcq79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:52.852984 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.852949 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-token" pod "323c4d64-a380-4e62-9779-1a1a87ae3308" (UID: "323c4d64-a380-4e62-9779-1a1a87ae3308"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:52.867784 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.863390 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-podinfo" pod "323c4d64-a380-4e62-9779-1a1a87ae3308" (UID: "323c4d64-a380-4e62-9779-1a1a87ae3308"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:01:52.931040 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.931008 6202 generic.go:356] "Generic (PLEG): container finished" podID="323c4d64-a380-4e62-9779-1a1a87ae3308" containerID="dde14991980aa79145cacb4f5218fd017677b3d0af5c194ae9937a3467003515" exitCode=0 Oct 30 17:01:52.931208 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.931057 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" Oct 30 17:01:52.931208 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.931091 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" event={"ID":"323c4d64-a380-4e62-9779-1a1a87ae3308","Type":"ContainerDied","Data":"dde14991980aa79145cacb4f5218fd017677b3d0af5c194ae9937a3467003515"} Oct 30 17:01:52.931208 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.931146 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg" event={"ID":"323c4d64-a380-4e62-9779-1a1a87ae3308","Type":"ContainerDied","Data":"a65441b9a4547e143efe66b23b151f8d20cdcfbb942bba27bf1d6fc376d48f17"} Oct 30 17:01:52.931208 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.931175 6202 scope.go:122] "RemoveContainer" containerID="dde14991980aa79145cacb4f5218fd017677b3d0af5c194ae9937a3467003515" Oct 30 17:01:52.943978 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.943950 6202 scope.go:122] "RemoveContainer" containerID="381811ac503f75277f36a76ce038e7b9d3cc91bcf5273179879969a09538c8c7" Oct 30 17:01:52.951060 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.951033 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:52.951182 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.951063 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323c4d64-a380-4e62-9779-1a1a87ae3308-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:52.951182 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.951077 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:52.951182 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.951089 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:52.951182 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.951101 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:52.951182 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.951111 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323c4d64-a380-4e62-9779-1a1a87ae3308-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:52.951182 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.951122 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcq79\" (UniqueName: \"kubernetes.io/projected/323c4d64-a380-4e62-9779-1a1a87ae3308-kube-api-access-tcq79\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:52.951182 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.951136 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c4d64-a380-4e62-9779-1a1a87ae3308-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:52.955611 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.955592 6202 scope.go:122] "RemoveContainer" containerID="dde14991980aa79145cacb4f5218fd017677b3d0af5c194ae9937a3467003515" Oct 30 17:01:52.955930 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:01:52.955907 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde14991980aa79145cacb4f5218fd017677b3d0af5c194ae9937a3467003515\": container with ID starting with dde14991980aa79145cacb4f5218fd017677b3d0af5c194ae9937a3467003515 not found: ID does not exist" containerID="dde14991980aa79145cacb4f5218fd017677b3d0af5c194ae9937a3467003515" Oct 30 17:01:52.956038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.955938 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde14991980aa79145cacb4f5218fd017677b3d0af5c194ae9937a3467003515"} err="failed to get container status \"dde14991980aa79145cacb4f5218fd017677b3d0af5c194ae9937a3467003515\": rpc error: code = NotFound desc = could not find container \"dde14991980aa79145cacb4f5218fd017677b3d0af5c194ae9937a3467003515\": container with ID starting with dde14991980aa79145cacb4f5218fd017677b3d0af5c194ae9937a3467003515 not found: ID does not exist" Oct 30 17:01:52.956038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.955956 6202 scope.go:122] "RemoveContainer" containerID="381811ac503f75277f36a76ce038e7b9d3cc91bcf5273179879969a09538c8c7" Oct 30 17:01:52.956243 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:01:52.956223 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"381811ac503f75277f36a76ce038e7b9d3cc91bcf5273179879969a09538c8c7\": container with ID starting with 381811ac503f75277f36a76ce038e7b9d3cc91bcf5273179879969a09538c8c7 not found: ID does not exist" containerID="381811ac503f75277f36a76ce038e7b9d3cc91bcf5273179879969a09538c8c7" Oct 30 17:01:52.956362 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.956248 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"381811ac503f75277f36a76ce038e7b9d3cc91bcf5273179879969a09538c8c7"} err="failed to get container status \"381811ac503f75277f36a76ce038e7b9d3cc91bcf5273179879969a09538c8c7\": rpc error: code = NotFound desc = could not find container \"381811ac503f75277f36a76ce038e7b9d3cc91bcf5273179879969a09538c8c7\": container with ID starting with 381811ac503f75277f36a76ce038e7b9d3cc91bcf5273179879969a09538c8c7 not found: ID does not exist" Oct 30 17:01:52.987016 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.986961 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg"] Oct 30 17:01:52.992839 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:52.992816 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ea77d8-openshift-default-68fd566499-mdglg"] Oct 30 17:01:53.246027 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:53.245919 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:54.412202 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:54.412169 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="323c4d64-a380-4e62-9779-1a1a87ae3308" path="/var/lib/kubelet/pods/323c4d64-a380-4e62-9779-1a1a87ae3308/volumes" Oct 30 17:01:55.049321 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.049280 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:55.759453 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.759409 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw"] Oct 30 17:01:55.760214 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.760196 6202 cpu_manager.go:414] "containerMap: removing container" podUID="323c4d64-a380-4e62-9779-1a1a87ae3308" containerName="istio-proxy" Oct 30 17:01:55.760214 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.760213 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323c4d64-a380-4e62-9779-1a1a87ae3308" containerName="istio-proxy" Oct 30 17:01:55.760473 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.760234 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9bbe06bf-8e56-4399-9969-41ade4c450ef" containerName="revoke-keys" Oct 30 17:01:55.760473 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.760239 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bbe06bf-8e56-4399-9969-41ade4c450ef" containerName="revoke-keys" Oct 30 17:01:55.760473 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.760264 6202 cpu_manager.go:414] "containerMap: removing container" podUID="4a1bf2ff-180e-4b58-bd3e-078ae6e8201b" containerName="revoke-keys" Oct 30 17:01:55.760473 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.760269 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a1bf2ff-180e-4b58-bd3e-078ae6e8201b" containerName="revoke-keys" Oct 30 17:01:55.760473 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.760379 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bbe06bf-8e56-4399-9969-41ade4c450ef" containerName="revoke-keys" Oct 30 17:01:55.760473 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.760391 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a1bf2ff-180e-4b58-bd3e-078ae6e8201b" containerName="revoke-keys" Oct 30 17:01:55.760473 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.760402 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="323c4d64-a380-4e62-9779-1a1a87ae3308" containerName="istio-proxy" Oct 30 17:01:55.760473 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.760409 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="323c4d64-a380-4e62-9779-1a1a87ae3308" containerName="istio-proxy" Oct 30 17:01:55.765815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.765794 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw" Oct 30 17:01:55.787569 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.786033 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw"] Oct 30 17:01:55.791568 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.791540 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:01:55.791754 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.791651 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:01:55.893938 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.893911 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5caf3f97-2914-46bf-a72c-ca024714d5d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77e226e9-pq9rw\" (UID: \"5caf3f97-2914-46bf-a72c-ca024714d5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw" Oct 30 17:01:55.894107 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.893974 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns2cn\" (UniqueName: \"kubernetes.io/projected/5caf3f97-2914-46bf-a72c-ca024714d5d4-kube-api-access-ns2cn\") pod \"maas-api-revoke-sub-77e226e9-pq9rw\" (UID: \"5caf3f97-2914-46bf-a72c-ca024714d5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw" Oct 30 17:01:55.894107 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.894079 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5caf3f97-2914-46bf-a72c-ca024714d5d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77e226e9-pq9rw\" (UID: \"5caf3f97-2914-46bf-a72c-ca024714d5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw" Oct 30 17:01:55.996197 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.996159 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5caf3f97-2914-46bf-a72c-ca024714d5d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77e226e9-pq9rw\" (UID: \"5caf3f97-2914-46bf-a72c-ca024714d5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw" Oct 30 17:01:55.996353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.996233 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns2cn\" (UniqueName: \"kubernetes.io/projected/5caf3f97-2914-46bf-a72c-ca024714d5d4-kube-api-access-ns2cn\") pod \"maas-api-revoke-sub-77e226e9-pq9rw\" (UID: \"5caf3f97-2914-46bf-a72c-ca024714d5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw" Oct 30 17:01:55.996353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.996253 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5caf3f97-2914-46bf-a72c-ca024714d5d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77e226e9-pq9rw\" (UID: \"5caf3f97-2914-46bf-a72c-ca024714d5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw" Oct 30 17:01:55.996917 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.996897 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5caf3f97-2914-46bf-a72c-ca024714d5d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77e226e9-pq9rw\" (UID: \"5caf3f97-2914-46bf-a72c-ca024714d5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw" Oct 30 17:01:55.998896 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:55.998870 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5caf3f97-2914-46bf-a72c-ca024714d5d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77e226e9-pq9rw\" (UID: \"5caf3f97-2914-46bf-a72c-ca024714d5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw" Oct 30 17:01:56.027655 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.027630 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns2cn\" (UniqueName: \"kubernetes.io/projected/5caf3f97-2914-46bf-a72c-ca024714d5d4-kube-api-access-ns2cn\") pod \"maas-api-revoke-sub-77e226e9-pq9rw\" (UID: \"5caf3f97-2914-46bf-a72c-ca024714d5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw" Oct 30 17:01:56.084529 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.084495 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw" Oct 30 17:01:56.358052 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.358020 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 17:01:56.385188 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.383011 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 17:01:56.391089 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.391063 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 17:01:56.461536 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.461503 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw"] Oct 30 17:01:56.507265 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.507227 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/318e8454-0cf7-485e-b774-9ce1b368cac6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318e8454-0cf7-485e-b774-9ce1b368cac6-istiod-ca-cert\") pod \"318e8454-0cf7-485e-b774-9ce1b368cac6\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " Oct 30 17:01:56.507433 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.507289 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istiod-ca-cert\") pod \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " Oct 30 17:01:56.507433 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.507368 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-workload-certs\") pod \"051d25d4-9b2f-482e-a9f7-1c630eec241c\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " Oct 30 17:01:56.507550 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.507437 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-envoy\") pod \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " Oct 30 17:01:56.507550 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.507468 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051d25d4-9b2f-482e-a9f7-1c630eec241c-kube-api-access-5nv4n\" (UniqueName: \"kubernetes.io/projected/051d25d4-9b2f-482e-a9f7-1c630eec241c-kube-api-access-5nv4n\") pod \"051d25d4-9b2f-482e-a9f7-1c630eec241c\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " Oct 30 17:01:56.507550 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.507531 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-data\") pod \"318e8454-0cf7-485e-b774-9ce1b368cac6\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " Oct 30 17:01:56.507714 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.507566 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-data\") pod \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " Oct 30 17:01:56.507714 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.507607 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-workload-socket\") pod \"318e8454-0cf7-485e-b774-9ce1b368cac6\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " Oct 30 17:01:56.507714 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.507633 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33abe911-bb76-44ba-a61e-6c8c87c95bb8-kube-api-access-857kc\" (UniqueName: \"kubernetes.io/projected/33abe911-bb76-44ba-a61e-6c8c87c95bb8-kube-api-access-857kc\") pod \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " Oct 30 17:01:56.507714 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.507668 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318e8454-0cf7-485e-b774-9ce1b368cac6-kube-api-access-84mjj\" (UniqueName: \"kubernetes.io/projected/318e8454-0cf7-485e-b774-9ce1b368cac6-kube-api-access-84mjj\") pod \"318e8454-0cf7-485e-b774-9ce1b368cac6\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " Oct 30 17:01:56.507714 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.507697 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-workload-certs\") pod \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " Oct 30 17:01:56.508060 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.507739 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-workload-certs\") pod \"318e8454-0cf7-485e-b774-9ce1b368cac6\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " Oct 30 17:01:56.508060 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.507827 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-token\" (UniqueName: \"kubernetes.io/projected/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-token\") pod \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " Oct 30 17:01:56.508060 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.507859 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-podinfo\") pod \"051d25d4-9b2f-482e-a9f7-1c630eec241c\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " Oct 30 17:01:56.508060 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.507888 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-credential-socket\") pod \"318e8454-0cf7-485e-b774-9ce1b368cac6\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " Oct 30 17:01:56.508060 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.507928 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051d25d4-9b2f-482e-a9f7-1c630eec241c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/051d25d4-9b2f-482e-a9f7-1c630eec241c-istiod-ca-cert\") pod \"051d25d4-9b2f-482e-a9f7-1c630eec241c\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " Oct 30 17:01:56.508060 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.507968 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-credential-socket\") pod \"051d25d4-9b2f-482e-a9f7-1c630eec241c\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " Oct 30 17:01:56.508060 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.508026 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-token\" (UniqueName: \"kubernetes.io/projected/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-token\") pod \"051d25d4-9b2f-482e-a9f7-1c630eec241c\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " Oct 30 17:01:56.508429 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.508062 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-credential-socket\") pod \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " Oct 30 17:01:56.508429 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.508159 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-envoy\") pod \"051d25d4-9b2f-482e-a9f7-1c630eec241c\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " Oct 30 17:01:56.508429 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.508201 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-podinfo\") pod \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " Oct 30 17:01:56.508429 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.508230 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-data\") pod \"051d25d4-9b2f-482e-a9f7-1c630eec241c\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " Oct 30 17:01:56.508429 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.508322 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-token\" (UniqueName: \"kubernetes.io/projected/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-token\") pod \"318e8454-0cf7-485e-b774-9ce1b368cac6\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " Oct 30 17:01:56.508429 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.508382 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-envoy\") pod \"318e8454-0cf7-485e-b774-9ce1b368cac6\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " Oct 30 17:01:56.508429 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.508414 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-workload-socket\") pod \"051d25d4-9b2f-482e-a9f7-1c630eec241c\" (UID: \"051d25d4-9b2f-482e-a9f7-1c630eec241c\") " Oct 30 17:01:56.508846 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.508444 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-podinfo\") pod \"318e8454-0cf7-485e-b774-9ce1b368cac6\" (UID: \"318e8454-0cf7-485e-b774-9ce1b368cac6\") " Oct 30 17:01:56.508846 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.508546 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-workload-socket\") pod \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\" (UID: \"33abe911-bb76-44ba-a61e-6c8c87c95bb8\") " Oct 30 17:01:56.510942 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.510654 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istiod-ca-cert" pod "33abe911-bb76-44ba-a61e-6c8c87c95bb8" (UID: "33abe911-bb76-44ba-a61e-6c8c87c95bb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:56.511389 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.511173 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051d25d4-9b2f-482e-a9f7-1c630eec241c-istiod-ca-cert" pod "051d25d4-9b2f-482e-a9f7-1c630eec241c" (UID: "051d25d4-9b2f-482e-a9f7-1c630eec241c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:56.511389 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.511259 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-workload-certs" pod "318e8454-0cf7-485e-b774-9ce1b368cac6" (UID: "318e8454-0cf7-485e-b774-9ce1b368cac6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.516288 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.514657 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-workload-certs" pod "051d25d4-9b2f-482e-a9f7-1c630eec241c" (UID: "051d25d4-9b2f-482e-a9f7-1c630eec241c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.516288 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.514723 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-workload-certs" pod "33abe911-bb76-44ba-a61e-6c8c87c95bb8" (UID: "33abe911-bb76-44ba-a61e-6c8c87c95bb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.516288 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.515523 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318e8454-0cf7-485e-b774-9ce1b368cac6-istiod-ca-cert" pod "318e8454-0cf7-485e-b774-9ce1b368cac6" (UID: "318e8454-0cf7-485e-b774-9ce1b368cac6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:56.516288 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.515572 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-credential-socket" pod "051d25d4-9b2f-482e-a9f7-1c630eec241c" (UID: "051d25d4-9b2f-482e-a9f7-1c630eec241c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.516577 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.516445 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-credential-socket" pod "318e8454-0cf7-485e-b774-9ce1b368cac6" (UID: "318e8454-0cf7-485e-b774-9ce1b368cac6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.517604 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.517090 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-data" pod "318e8454-0cf7-485e-b774-9ce1b368cac6" (UID: "318e8454-0cf7-485e-b774-9ce1b368cac6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.517604 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.517434 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-workload-socket" pod "318e8454-0cf7-485e-b774-9ce1b368cac6" (UID: "318e8454-0cf7-485e-b774-9ce1b368cac6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.517783 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.517599 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-credential-socket" pod "33abe911-bb76-44ba-a61e-6c8c87c95bb8" (UID: "33abe911-bb76-44ba-a61e-6c8c87c95bb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.521484 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.518924 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-workload-socket" pod "051d25d4-9b2f-482e-a9f7-1c630eec241c" (UID: "051d25d4-9b2f-482e-a9f7-1c630eec241c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.521484 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.518973 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-workload-socket" pod "33abe911-bb76-44ba-a61e-6c8c87c95bb8" (UID: "33abe911-bb76-44ba-a61e-6c8c87c95bb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.521484 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.519009 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-data" pod "33abe911-bb76-44ba-a61e-6c8c87c95bb8" (UID: "33abe911-bb76-44ba-a61e-6c8c87c95bb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.521484 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.519907 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-data" pod "051d25d4-9b2f-482e-a9f7-1c630eec241c" (UID: "051d25d4-9b2f-482e-a9f7-1c630eec241c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.524805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.524549 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-podinfo" pod "33abe911-bb76-44ba-a61e-6c8c87c95bb8" (UID: "33abe911-bb76-44ba-a61e-6c8c87c95bb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:01:56.526061 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.525942 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051d25d4-9b2f-482e-a9f7-1c630eec241c-kube-api-access-5nv4n" pod "051d25d4-9b2f-482e-a9f7-1c630eec241c" (UID: "051d25d4-9b2f-482e-a9f7-1c630eec241c"). InnerVolumeSpecName "kube-api-access-5nv4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.526061 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.526010 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-envoy" pod "051d25d4-9b2f-482e-a9f7-1c630eec241c" (UID: "051d25d4-9b2f-482e-a9f7-1c630eec241c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.527329 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.527261 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-token" pod "318e8454-0cf7-485e-b774-9ce1b368cac6" (UID: "318e8454-0cf7-485e-b774-9ce1b368cac6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.527329 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.527267 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33abe911-bb76-44ba-a61e-6c8c87c95bb8-kube-api-access-857kc" pod "33abe911-bb76-44ba-a61e-6c8c87c95bb8" (UID: "33abe911-bb76-44ba-a61e-6c8c87c95bb8"). InnerVolumeSpecName "kube-api-access-857kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.527329 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.527296 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-token" pod "051d25d4-9b2f-482e-a9f7-1c630eec241c" (UID: "051d25d4-9b2f-482e-a9f7-1c630eec241c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.527961 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.527935 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-envoy" pod "33abe911-bb76-44ba-a61e-6c8c87c95bb8" (UID: "33abe911-bb76-44ba-a61e-6c8c87c95bb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.528042 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.527979 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-token" pod "33abe911-bb76-44ba-a61e-6c8c87c95bb8" (UID: "33abe911-bb76-44ba-a61e-6c8c87c95bb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.528348 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.528296 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-podinfo" pod "051d25d4-9b2f-482e-a9f7-1c630eec241c" (UID: "051d25d4-9b2f-482e-a9f7-1c630eec241c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:01:56.533979 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.533941 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-podinfo" pod "318e8454-0cf7-485e-b774-9ce1b368cac6" (UID: "318e8454-0cf7-485e-b774-9ce1b368cac6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:01:56.536329 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.536292 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318e8454-0cf7-485e-b774-9ce1b368cac6-kube-api-access-84mjj" pod "318e8454-0cf7-485e-b774-9ce1b368cac6" (UID: "318e8454-0cf7-485e-b774-9ce1b368cac6"). InnerVolumeSpecName "kube-api-access-84mjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.537182 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.537147 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-envoy" pod "318e8454-0cf7-485e-b774-9ce1b368cac6" (UID: "318e8454-0cf7-485e-b774-9ce1b368cac6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.611195 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611118 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611195 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611147 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611195 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611156 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611195 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611164 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611195 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611172 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611195 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611180 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318e8454-0cf7-485e-b774-9ce1b368cac6-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611195 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611188 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611195 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611195 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611195 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611203 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611211 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nv4n\" (UniqueName: \"kubernetes.io/projected/051d25d4-9b2f-482e-a9f7-1c630eec241c-kube-api-access-5nv4n\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611220 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611228 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611235 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611243 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-857kc\" (UniqueName: \"kubernetes.io/projected/33abe911-bb76-44ba-a61e-6c8c87c95bb8-kube-api-access-857kc\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611251 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84mjj\" (UniqueName: \"kubernetes.io/projected/318e8454-0cf7-485e-b774-9ce1b368cac6-kube-api-access-84mjj\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611259 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611266 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611273 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611280 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611287 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318e8454-0cf7-485e-b774-9ce1b368cac6-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611295 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/051d25d4-9b2f-482e-a9f7-1c630eec241c-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611302 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611310 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611319 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33abe911-bb76-44ba-a61e-6c8c87c95bb8-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611328 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611335 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33abe911-bb76-44ba-a61e-6c8c87c95bb8-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.611694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.611342 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/051d25d4-9b2f-482e-a9f7-1c630eec241c-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.918010 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.916010 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:56.991322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.990424 6202 generic.go:356] "Generic (PLEG): container finished" podID="051d25d4-9b2f-482e-a9f7-1c630eec241c" containerID="ac831c619ea4ce9946bad503fcde433894b9b9af5a4ccb0c894c8ad233ee147b" exitCode=0 Oct 30 17:01:56.991322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.990539 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" Oct 30 17:01:56.991322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.990633 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" event={"ID":"051d25d4-9b2f-482e-a9f7-1c630eec241c","Type":"ContainerDied","Data":"ac831c619ea4ce9946bad503fcde433894b9b9af5a4ccb0c894c8ad233ee147b"} Oct 30 17:01:56.991322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.990682 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x" event={"ID":"051d25d4-9b2f-482e-a9f7-1c630eec241c","Type":"ContainerDied","Data":"58274fa80b0a7f894fd0776a5441368c263c3e21e40730fb6569c0be574aec59"} Oct 30 17:01:56.991322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.990710 6202 scope.go:122] "RemoveContainer" containerID="ac831c619ea4ce9946bad503fcde433894b9b9af5a4ccb0c894c8ad233ee147b" Oct 30 17:01:56.999907 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.999595 6202 generic.go:356] "Generic (PLEG): container finished" podID="318e8454-0cf7-485e-b774-9ce1b368cac6" containerID="bb4aa15c5ed737c36f2aa3d71aca44d0126b5debb4eac2aa352fde9ff2989172" exitCode=0 Oct 30 17:01:57.000202 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.000178 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" Oct 30 17:01:57.008065 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:56.999879 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" event={"ID":"318e8454-0cf7-485e-b774-9ce1b368cac6","Type":"ContainerDied","Data":"bb4aa15c5ed737c36f2aa3d71aca44d0126b5debb4eac2aa352fde9ff2989172"} Oct 30 17:01:57.008065 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.007971 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r" event={"ID":"318e8454-0cf7-485e-b774-9ce1b368cac6","Type":"ContainerDied","Data":"4e9f533b4aaab210475f76ad937b5f1f26bb10cb2bcf27dcc053c444dfc460e9"} Oct 30 17:01:57.010858 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.010801 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw" event={"ID":"5caf3f97-2914-46bf-a72c-ca024714d5d4","Type":"ContainerStarted","Data":"e77327528ab328225d3159db38506a1f9d5f9a8454a6f76c9251a7d1a1b5375c"} Oct 30 17:01:57.012601 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.012568 6202 generic.go:356] "Generic (PLEG): container finished" podID="33abe911-bb76-44ba-a61e-6c8c87c95bb8" containerID="ea18800c7daca6729a6b9b53ff51414e4f1a738ee007693550e4254796bbb51b" exitCode=0 Oct 30 17:01:57.012701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.012602 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" Oct 30 17:01:57.012701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.012611 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" event={"ID":"33abe911-bb76-44ba-a61e-6c8c87c95bb8","Type":"ContainerDied","Data":"ea18800c7daca6729a6b9b53ff51414e4f1a738ee007693550e4254796bbb51b"} Oct 30 17:01:57.012868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.012738 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9" event={"ID":"33abe911-bb76-44ba-a61e-6c8c87c95bb8","Type":"ContainerDied","Data":"80291eeadd4eec71104c462fd5a9c172e324ae4fe90b780437f21649b91ef75d"} Oct 30 17:01:57.050906 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.050883 6202 scope.go:122] "RemoveContainer" containerID="ac831c619ea4ce9946bad503fcde433894b9b9af5a4ccb0c894c8ad233ee147b" Oct 30 17:01:57.051221 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:01:57.051192 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac831c619ea4ce9946bad503fcde433894b9b9af5a4ccb0c894c8ad233ee147b\": container with ID starting with ac831c619ea4ce9946bad503fcde433894b9b9af5a4ccb0c894c8ad233ee147b not found: ID does not exist" containerID="ac831c619ea4ce9946bad503fcde433894b9b9af5a4ccb0c894c8ad233ee147b" Oct 30 17:01:57.051305 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.051234 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac831c619ea4ce9946bad503fcde433894b9b9af5a4ccb0c894c8ad233ee147b"} err="failed to get container status \"ac831c619ea4ce9946bad503fcde433894b9b9af5a4ccb0c894c8ad233ee147b\": rpc error: code = NotFound desc = could not find container \"ac831c619ea4ce9946bad503fcde433894b9b9af5a4ccb0c894c8ad233ee147b\": container with ID starting with ac831c619ea4ce9946bad503fcde433894b9b9af5a4ccb0c894c8ad233ee147b not found: ID does not exist" Oct 30 17:01:57.051305 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.051258 6202 scope.go:122] "RemoveContainer" containerID="bb4aa15c5ed737c36f2aa3d71aca44d0126b5debb4eac2aa352fde9ff2989172" Oct 30 17:01:57.064265 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.064242 6202 scope.go:122] "RemoveContainer" containerID="fcaf838b4520c9a9d6431c3843ebbe8f76a404c6e31aa532650a80b032f843c7" Oct 30 17:01:57.081339 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.081301 6202 scope.go:122] "RemoveContainer" containerID="bb4aa15c5ed737c36f2aa3d71aca44d0126b5debb4eac2aa352fde9ff2989172" Oct 30 17:01:57.082637 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:01:57.081647 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb4aa15c5ed737c36f2aa3d71aca44d0126b5debb4eac2aa352fde9ff2989172\": container with ID starting with bb4aa15c5ed737c36f2aa3d71aca44d0126b5debb4eac2aa352fde9ff2989172 not found: ID does not exist" containerID="bb4aa15c5ed737c36f2aa3d71aca44d0126b5debb4eac2aa352fde9ff2989172" Oct 30 17:01:57.082637 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.081681 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb4aa15c5ed737c36f2aa3d71aca44d0126b5debb4eac2aa352fde9ff2989172"} err="failed to get container status \"bb4aa15c5ed737c36f2aa3d71aca44d0126b5debb4eac2aa352fde9ff2989172\": rpc error: code = NotFound desc = could not find container \"bb4aa15c5ed737c36f2aa3d71aca44d0126b5debb4eac2aa352fde9ff2989172\": container with ID starting with bb4aa15c5ed737c36f2aa3d71aca44d0126b5debb4eac2aa352fde9ff2989172 not found: ID does not exist" Oct 30 17:01:57.082637 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.081703 6202 scope.go:122] "RemoveContainer" containerID="fcaf838b4520c9a9d6431c3843ebbe8f76a404c6e31aa532650a80b032f843c7" Oct 30 17:01:57.082637 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:01:57.082169 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcaf838b4520c9a9d6431c3843ebbe8f76a404c6e31aa532650a80b032f843c7\": container with ID starting with fcaf838b4520c9a9d6431c3843ebbe8f76a404c6e31aa532650a80b032f843c7 not found: ID does not exist" containerID="fcaf838b4520c9a9d6431c3843ebbe8f76a404c6e31aa532650a80b032f843c7" Oct 30 17:01:57.082637 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.082204 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcaf838b4520c9a9d6431c3843ebbe8f76a404c6e31aa532650a80b032f843c7"} err="failed to get container status \"fcaf838b4520c9a9d6431c3843ebbe8f76a404c6e31aa532650a80b032f843c7\": rpc error: code = NotFound desc = could not find container \"fcaf838b4520c9a9d6431c3843ebbe8f76a404c6e31aa532650a80b032f843c7\": container with ID starting with fcaf838b4520c9a9d6431c3843ebbe8f76a404c6e31aa532650a80b032f843c7 not found: ID does not exist" Oct 30 17:01:57.082637 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.082238 6202 scope.go:122] "RemoveContainer" containerID="ea18800c7daca6729a6b9b53ff51414e4f1a738ee007693550e4254796bbb51b" Oct 30 17:01:57.087634 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.087611 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r"] Oct 30 17:01:57.092351 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.092326 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-spj5r"] Oct 30 17:01:57.097816 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.097616 6202 scope.go:122] "RemoveContainer" containerID="ea18800c7daca6729a6b9b53ff51414e4f1a738ee007693550e4254796bbb51b" Oct 30 17:01:57.097973 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:01:57.097953 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea18800c7daca6729a6b9b53ff51414e4f1a738ee007693550e4254796bbb51b\": container with ID starting with ea18800c7daca6729a6b9b53ff51414e4f1a738ee007693550e4254796bbb51b not found: ID does not exist" containerID="ea18800c7daca6729a6b9b53ff51414e4f1a738ee007693550e4254796bbb51b" Oct 30 17:01:57.098029 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.097981 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea18800c7daca6729a6b9b53ff51414e4f1a738ee007693550e4254796bbb51b"} err="failed to get container status \"ea18800c7daca6729a6b9b53ff51414e4f1a738ee007693550e4254796bbb51b\": rpc error: code = NotFound desc = could not find container \"ea18800c7daca6729a6b9b53ff51414e4f1a738ee007693550e4254796bbb51b\": container with ID starting with ea18800c7daca6729a6b9b53ff51414e4f1a738ee007693550e4254796bbb51b not found: ID does not exist" Oct 30 17:01:57.129623 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.129582 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9"] Oct 30 17:01:57.135332 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.135308 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-jzkv9"] Oct 30 17:01:57.173842 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.173757 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x"] Oct 30 17:01:57.179692 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:57.179663 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-8rh7x"] Oct 30 17:01:58.022964 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:58.022928 6202 generic.go:356] "Generic (PLEG): container finished" podID="5caf3f97-2914-46bf-a72c-ca024714d5d4" containerID="d2e5c9e92f7b6f9ab44f0dd3111f09e1bea40207022203fad3c0e398a890e0a4" exitCode=0 Oct 30 17:01:58.023390 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:58.023017 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw" event={"ID":"5caf3f97-2914-46bf-a72c-ca024714d5d4","Type":"ContainerDied","Data":"d2e5c9e92f7b6f9ab44f0dd3111f09e1bea40207022203fad3c0e398a890e0a4"} Oct 30 17:01:58.412635 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:58.412601 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051d25d4-9b2f-482e-a9f7-1c630eec241c" path="/var/lib/kubelet/pods/051d25d4-9b2f-482e-a9f7-1c630eec241c/volumes" Oct 30 17:01:58.413138 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:58.413122 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="318e8454-0cf7-485e-b774-9ce1b368cac6" path="/var/lib/kubelet/pods/318e8454-0cf7-485e-b774-9ce1b368cac6/volumes" Oct 30 17:01:58.413562 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:58.413546 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33abe911-bb76-44ba-a61e-6c8c87c95bb8" path="/var/lib/kubelet/pods/33abe911-bb76-44ba-a61e-6c8c87c95bb8/volumes" Oct 30 17:01:58.414150 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:58.414132 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986"] Oct 30 17:01:58.424566 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:58.424544 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v"] Oct 30 17:01:58.530824 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:58.530791 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:58.654505 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:58.654448 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:58.795416 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:58.794855 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:59.335040 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:59.334979 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw" Oct 30 17:01:59.434101 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:59.434070 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5caf3f97-2914-46bf-a72c-ca024714d5d4-kube-api-access-ns2cn\" (UniqueName: \"kubernetes.io/projected/5caf3f97-2914-46bf-a72c-ca024714d5d4-kube-api-access-ns2cn\") pod \"5caf3f97-2914-46bf-a72c-ca024714d5d4\" (UID: \"5caf3f97-2914-46bf-a72c-ca024714d5d4\") " Oct 30 17:01:59.434305 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:59.434145 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5caf3f97-2914-46bf-a72c-ca024714d5d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5caf3f97-2914-46bf-a72c-ca024714d5d4-maas-api-xks-ca\") pod \"5caf3f97-2914-46bf-a72c-ca024714d5d4\" (UID: \"5caf3f97-2914-46bf-a72c-ca024714d5d4\") " Oct 30 17:01:59.434385 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:59.434366 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5caf3f97-2914-46bf-a72c-ca024714d5d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5caf3f97-2914-46bf-a72c-ca024714d5d4-maas-api-openshift-ca\") pod \"5caf3f97-2914-46bf-a72c-ca024714d5d4\" (UID: \"5caf3f97-2914-46bf-a72c-ca024714d5d4\") " Oct 30 17:01:59.434727 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:59.434696 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5caf3f97-2914-46bf-a72c-ca024714d5d4-maas-api-openshift-ca" pod "5caf3f97-2914-46bf-a72c-ca024714d5d4" (UID: "5caf3f97-2914-46bf-a72c-ca024714d5d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:59.436487 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:59.436461 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5caf3f97-2914-46bf-a72c-ca024714d5d4-kube-api-access-ns2cn" pod "5caf3f97-2914-46bf-a72c-ca024714d5d4" (UID: "5caf3f97-2914-46bf-a72c-ca024714d5d4"). InnerVolumeSpecName "kube-api-access-ns2cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:59.436587 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:59.436467 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5caf3f97-2914-46bf-a72c-ca024714d5d4-maas-api-xks-ca" pod "5caf3f97-2914-46bf-a72c-ca024714d5d4" (UID: "5caf3f97-2914-46bf-a72c-ca024714d5d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:59.536583 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:59.536549 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5caf3f97-2914-46bf-a72c-ca024714d5d4-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:59.536583 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:59.536580 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5caf3f97-2914-46bf-a72c-ca024714d5d4-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:01:59.536804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:01:59.536595 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns2cn\" (UniqueName: \"kubernetes.io/projected/5caf3f97-2914-46bf-a72c-ca024714d5d4-kube-api-access-ns2cn\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:02:00.081356 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:00.081323 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw" Oct 30 17:02:00.081533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:00.081363 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw" event={"ID":"5caf3f97-2914-46bf-a72c-ca024714d5d4","Type":"ContainerDied","Data":"e77327528ab328225d3159db38506a1f9d5f9a8454a6f76c9251a7d1a1b5375c"} Oct 30 17:02:00.081533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:00.081420 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e77327528ab328225d3159db38506a1f9d5f9a8454a6f76c9251a7d1a1b5375c" Oct 30 17:02:00.518110 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:00.518031 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:02.348584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:02.348523 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:03.414987 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:03.414819 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986" podUID="7d0e1b07-3b0f-4baf-bc98-005127e74f31" containerName="payload-processing" containerID="cri-o://d0eaef2c4f7f51f35e9c14194b41cbaa14965663bde83415f3fd2fd2e917156e" gracePeriod=25 Oct 30 17:02:03.430205 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:03.429823 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" podUID="4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e" containerName="payload-pre-processing" containerID="cri-o://c5e0a07769c0bbe69a12e96456ea392d9520705924ee1357c30206eff0195aca" gracePeriod=25 Oct 30 17:02:03.996573 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:03.988141 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986" Oct 30 17:02:04.139793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:04.127221 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d0e1b07-3b0f-4baf-bc98-005127e74f31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0e1b07-3b0f-4baf-bc98-005127e74f31-plugins-config-volume\") pod \"7d0e1b07-3b0f-4baf-bc98-005127e74f31\" (UID: \"7d0e1b07-3b0f-4baf-bc98-005127e74f31\") " Oct 30 17:02:04.139793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:04.127610 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0e1b07-3b0f-4baf-bc98-005127e74f31-kube-api-access-r24zx\" (UniqueName: \"kubernetes.io/projected/7d0e1b07-3b0f-4baf-bc98-005127e74f31-kube-api-access-r24zx\") pod \"7d0e1b07-3b0f-4baf-bc98-005127e74f31\" (UID: \"7d0e1b07-3b0f-4baf-bc98-005127e74f31\") " Oct 30 17:02:04.139793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:04.139649 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0e1b07-3b0f-4baf-bc98-005127e74f31-plugins-config-volume" pod "7d0e1b07-3b0f-4baf-bc98-005127e74f31" (UID: "7d0e1b07-3b0f-4baf-bc98-005127e74f31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:02:04.140157 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:04.140122 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0e1b07-3b0f-4baf-bc98-005127e74f31-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:02:04.190881 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:04.182359 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0e1b07-3b0f-4baf-bc98-005127e74f31-kube-api-access-r24zx" pod "7d0e1b07-3b0f-4baf-bc98-005127e74f31" (UID: "7d0e1b07-3b0f-4baf-bc98-005127e74f31"). InnerVolumeSpecName "kube-api-access-r24zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:02:04.216793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:04.215725 6202 generic.go:356] "Generic (PLEG): container finished" podID="7d0e1b07-3b0f-4baf-bc98-005127e74f31" containerID="d0eaef2c4f7f51f35e9c14194b41cbaa14965663bde83415f3fd2fd2e917156e" exitCode=0 Oct 30 17:02:04.216793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:04.215815 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986" event={"ID":"7d0e1b07-3b0f-4baf-bc98-005127e74f31","Type":"ContainerDied","Data":"d0eaef2c4f7f51f35e9c14194b41cbaa14965663bde83415f3fd2fd2e917156e"} Oct 30 17:02:04.216793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:04.215918 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986" event={"ID":"7d0e1b07-3b0f-4baf-bc98-005127e74f31","Type":"ContainerDied","Data":"7ba3b337529bbece690fd2bbdfcf160caaf2d125ee2f25c5f5fa550dd319434d"} Oct 30 17:02:04.216793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:04.215982 6202 scope.go:122] "RemoveContainer" containerID="d0eaef2c4f7f51f35e9c14194b41cbaa14965663bde83415f3fd2fd2e917156e" Oct 30 17:02:04.216793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:04.216194 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986" Oct 30 17:02:04.250804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:04.250291 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r24zx\" (UniqueName: \"kubernetes.io/projected/7d0e1b07-3b0f-4baf-bc98-005127e74f31-kube-api-access-r24zx\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:02:04.263989 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:04.263292 6202 scope.go:122] "RemoveContainer" containerID="d0eaef2c4f7f51f35e9c14194b41cbaa14965663bde83415f3fd2fd2e917156e" Oct 30 17:02:04.263989 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:02:04.263669 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0eaef2c4f7f51f35e9c14194b41cbaa14965663bde83415f3fd2fd2e917156e\": container with ID starting with d0eaef2c4f7f51f35e9c14194b41cbaa14965663bde83415f3fd2fd2e917156e not found: ID does not exist" containerID="d0eaef2c4f7f51f35e9c14194b41cbaa14965663bde83415f3fd2fd2e917156e" Oct 30 17:02:04.263989 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:04.263706 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0eaef2c4f7f51f35e9c14194b41cbaa14965663bde83415f3fd2fd2e917156e"} err="failed to get container status \"d0eaef2c4f7f51f35e9c14194b41cbaa14965663bde83415f3fd2fd2e917156e\": rpc error: code = NotFound desc = could not find container \"d0eaef2c4f7f51f35e9c14194b41cbaa14965663bde83415f3fd2fd2e917156e\": container with ID starting with d0eaef2c4f7f51f35e9c14194b41cbaa14965663bde83415f3fd2fd2e917156e not found: ID does not exist" Oct 30 17:02:04.322235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:04.319042 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986"] Oct 30 17:02:04.322235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:04.321031 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f92caf-77cfcd9448-6h986"] Oct 30 17:02:04.412759 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:04.412684 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d0e1b07-3b0f-4baf-bc98-005127e74f31" path="/var/lib/kubelet/pods/7d0e1b07-3b0f-4baf-bc98-005127e74f31/volumes" Oct 30 17:02:04.701314 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:04.701238 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:06.665846 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:06.665635 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:08.651619 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:08.651584 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:10.545090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:10.545006 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:10.662308 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:10.662236 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:12.220322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:12.220288 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:12.859886 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:12.859847 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" podUID="4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:02:13.381993 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.381956 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w"] Oct 30 17:02:13.383123 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383095 6202 cpu_manager.go:414] "containerMap: removing container" podUID="318e8454-0cf7-485e-b774-9ce1b368cac6" containerName="istio-proxy" Oct 30 17:02:13.383123 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383120 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="318e8454-0cf7-485e-b774-9ce1b368cac6" containerName="istio-proxy" Oct 30 17:02:13.383331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383139 6202 cpu_manager.go:414] "containerMap: removing container" podUID="318e8454-0cf7-485e-b774-9ce1b368cac6" containerName="istio-proxy" Oct 30 17:02:13.383331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383147 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="318e8454-0cf7-485e-b774-9ce1b368cac6" containerName="istio-proxy" Oct 30 17:02:13.383331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383171 6202 cpu_manager.go:414] "containerMap: removing container" podUID="051d25d4-9b2f-482e-a9f7-1c630eec241c" containerName="istio-proxy" Oct 30 17:02:13.383331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383178 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051d25d4-9b2f-482e-a9f7-1c630eec241c" containerName="istio-proxy" Oct 30 17:02:13.383331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383202 6202 cpu_manager.go:414] "containerMap: removing container" podUID="323c4d64-a380-4e62-9779-1a1a87ae3308" containerName="istio-proxy" Oct 30 17:02:13.383331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383209 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323c4d64-a380-4e62-9779-1a1a87ae3308" containerName="istio-proxy" Oct 30 17:02:13.383331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383222 6202 cpu_manager.go:414] "containerMap: removing container" podUID="33abe911-bb76-44ba-a61e-6c8c87c95bb8" containerName="istio-proxy" Oct 30 17:02:13.383331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383229 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33abe911-bb76-44ba-a61e-6c8c87c95bb8" containerName="istio-proxy" Oct 30 17:02:13.383331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383257 6202 cpu_manager.go:414] "containerMap: removing container" podUID="7d0e1b07-3b0f-4baf-bc98-005127e74f31" containerName="payload-processing" Oct 30 17:02:13.383331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383264 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d0e1b07-3b0f-4baf-bc98-005127e74f31" containerName="payload-processing" Oct 30 17:02:13.383331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383273 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5caf3f97-2914-46bf-a72c-ca024714d5d4" containerName="revoke-keys" Oct 30 17:02:13.383331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383281 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5caf3f97-2914-46bf-a72c-ca024714d5d4" containerName="revoke-keys" Oct 30 17:02:13.384171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383433 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="318e8454-0cf7-485e-b774-9ce1b368cac6" containerName="istio-proxy" Oct 30 17:02:13.384171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383448 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="318e8454-0cf7-485e-b774-9ce1b368cac6" containerName="istio-proxy" Oct 30 17:02:13.384171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383459 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="051d25d4-9b2f-482e-a9f7-1c630eec241c" containerName="istio-proxy" Oct 30 17:02:13.384171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383473 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="33abe911-bb76-44ba-a61e-6c8c87c95bb8" containerName="istio-proxy" Oct 30 17:02:13.384171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383487 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d0e1b07-3b0f-4baf-bc98-005127e74f31" containerName="payload-processing" Oct 30 17:02:13.384171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.383494 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5caf3f97-2914-46bf-a72c-ca024714d5d4" containerName="revoke-keys" Oct 30 17:02:13.405753 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.389960 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.409067 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.409033 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w"] Oct 30 17:02:13.413139 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.413109 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-e43400\"/\"kube-root-ca.crt\"" Oct 30 17:02:13.413300 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.413185 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-e43400\"/\"openshift-service-ca.crt\"" Oct 30 17:02:13.416891 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.416864 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-e43400\"/\"default-dockercfg-hxn94\"" Oct 30 17:02:13.417004 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.416861 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-e43400\"/\"auth-test-model-e43400-kserve-self-signed-certs\"" Oct 30 17:02:13.472362 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.472326 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:13.493052 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.493020 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-model-cache\") pod \"auth-test-model-e43400-kserve-5bf97847cc-wpd9w\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.493211 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.493081 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxg9s\" (UniqueName: \"kubernetes.io/projected/65d54ddf-f0f1-4271-8025-361a9f3e3507-kube-api-access-zxg9s\") pod \"auth-test-model-e43400-kserve-5bf97847cc-wpd9w\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.493573 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.493541 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-tmp-dir\") pod \"auth-test-model-e43400-kserve-5bf97847cc-wpd9w\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.493696 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.493625 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-dshm\") pod \"auth-test-model-e43400-kserve-5bf97847cc-wpd9w\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.493803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.493694 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-home\") pod \"auth-test-model-e43400-kserve-5bf97847cc-wpd9w\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.493803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.493737 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65d54ddf-f0f1-4271-8025-361a9f3e3507-tls-certs\") pod \"auth-test-model-e43400-kserve-5bf97847cc-wpd9w\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.595579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.595548 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-tmp-dir\") pod \"auth-test-model-e43400-kserve-5bf97847cc-wpd9w\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.595790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.595602 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-dshm\") pod \"auth-test-model-e43400-kserve-5bf97847cc-wpd9w\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.595790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.595643 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-home\") pod \"auth-test-model-e43400-kserve-5bf97847cc-wpd9w\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.595790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.595674 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65d54ddf-f0f1-4271-8025-361a9f3e3507-tls-certs\") pod \"auth-test-model-e43400-kserve-5bf97847cc-wpd9w\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.595790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.595755 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-model-cache\") pod \"auth-test-model-e43400-kserve-5bf97847cc-wpd9w\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.596025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.595821 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxg9s\" (UniqueName: \"kubernetes.io/projected/65d54ddf-f0f1-4271-8025-361a9f3e3507-kube-api-access-zxg9s\") pod \"auth-test-model-e43400-kserve-5bf97847cc-wpd9w\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.596337 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.596287 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-model-cache\") pod \"auth-test-model-e43400-kserve-5bf97847cc-wpd9w\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.596337 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.596325 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-home\") pod \"auth-test-model-e43400-kserve-5bf97847cc-wpd9w\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.596695 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.596359 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-tmp-dir\") pod \"auth-test-model-e43400-kserve-5bf97847cc-wpd9w\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.598232 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.598210 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-dshm\") pod \"auth-test-model-e43400-kserve-5bf97847cc-wpd9w\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.598391 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.598373 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65d54ddf-f0f1-4271-8025-361a9f3e3507-tls-certs\") pod \"auth-test-model-e43400-kserve-5bf97847cc-wpd9w\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.633836 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.632929 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxg9s\" (UniqueName: \"kubernetes.io/projected/65d54ddf-f0f1-4271-8025-361a9f3e3507-kube-api-access-zxg9s\") pod \"auth-test-model-e43400-kserve-5bf97847cc-wpd9w\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:13.725207 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:13.725170 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:14.031023 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:14.030947 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:14.041346 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:14.041317 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w"] Oct 30 17:02:14.116044 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:02:14.115983 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:02:14.332902 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:14.332761 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" event={"ID":"65d54ddf-f0f1-4271-8025-361a9f3e3507","Type":"ContainerStarted","Data":"8ebe2be664e77ccdfe91b213930b0273f6097284828029dbb5ad65948f05cd63"} Oct 30 17:02:14.333097 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:14.332917 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" event={"ID":"65d54ddf-f0f1-4271-8025-361a9f3e3507","Type":"ContainerStarted","Data":"ee68b579c3e7cdbf393e07165680c6e448a6e09185fcbeaba5fc65fceb7e1357"} Oct 30 17:02:14.375252 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:14.375204 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" podStartSLOduration=1.375190006 podStartE2EDuration="1.375190006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:14.374566231 +0000 UTC m=+2448.085020975" watchObservedRunningTime="2026-10-30 17:02:14.375190006 +0000 UTC m=+2448.085644750" Oct 30 17:02:14.630839 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:14.630136 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": read tcp 10.133.0.2:38776->10.133.1.9:15021: read: connection reset by peer" start-of-body= Oct 30 17:02:14.630839 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:14.630197 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" podUID="e7bcb077-0d1d-4046-9a5f-75fc61dd2886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": read tcp 10.133.0.2:38776->10.133.1.9:15021: read: connection reset by peer" Oct 30 17:02:14.726307 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:14.726274 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:15.343077 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:15.343037 6202 generic.go:356] "Generic (PLEG): container finished" podID="e7bcb077-0d1d-4046-9a5f-75fc61dd2886" containerID="ce8b40adbd789f536bf361e2b3ad0e0616e05d724036925874c9ffa350ed2c5d" exitCode=0 Oct 30 17:02:15.343262 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:15.343106 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" event={"ID":"e7bcb077-0d1d-4046-9a5f-75fc61dd2886","Type":"ContainerDied","Data":"ce8b40adbd789f536bf361e2b3ad0e0616e05d724036925874c9ffa350ed2c5d"} Oct 30 17:02:15.343544 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:15.343523 6202 scope.go:122] "RemoveContainer" containerID="ce8b40adbd789f536bf361e2b3ad0e0616e05d724036925874c9ffa350ed2c5d" Oct 30 17:02:15.345540 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:15.345506 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:02:15.345627 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:15.345600 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:02:15.345666 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:15.345647 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:02:15.818211 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:15.818113 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:16.376684 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:16.376638 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" event={"ID":"e7bcb077-0d1d-4046-9a5f-75fc61dd2886","Type":"ContainerStarted","Data":"16dfe3a2dd90e82eb79cac11fc402a181a1a6fe8fe6eca025395f9414e8cfd81"} Oct 30 17:02:17.307603 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:17.307566 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:02:17.308101 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:17.307710 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Oct 30 17:02:17.308101 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:17.307796 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" podUID="e7bcb077-0d1d-4046-9a5f-75fc61dd2886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Oct 30 17:02:17.568848 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:17.568248 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:18.308090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:18.308059 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Oct 30 17:02:18.308491 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:18.308114 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" podUID="e7bcb077-0d1d-4046-9a5f-75fc61dd2886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Oct 30 17:02:18.973756 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:18.973722 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g"] Oct 30 17:02:18.974007 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:18.973982 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" podUID="73af482c-6523-4268-9ddf-8c4eeefd5f32" containerName="maas-api" containerID="cri-o://0a86dd533395e5be7f50b6355cb92a01d747da5176c2d72ee44ccc42676f6f1e" gracePeriod=30 Oct 30 17:02:19.077792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.076320 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:19.314850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.313229 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Oct 30 17:02:19.314850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.313287 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" podUID="e7bcb077-0d1d-4046-9a5f-75fc61dd2886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Oct 30 17:02:19.593947 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.593901 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" Oct 30 17:02:19.597550 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.597449 6202 generic.go:356] "Generic (PLEG): container finished" podID="73af482c-6523-4268-9ddf-8c4eeefd5f32" containerID="0a86dd533395e5be7f50b6355cb92a01d747da5176c2d72ee44ccc42676f6f1e" exitCode=0 Oct 30 17:02:19.597550 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.597504 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" event={"ID":"73af482c-6523-4268-9ddf-8c4eeefd5f32","Type":"ContainerDied","Data":"0a86dd533395e5be7f50b6355cb92a01d747da5176c2d72ee44ccc42676f6f1e"} Oct 30 17:02:19.597550 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.597553 6202 scope.go:122] "RemoveContainer" containerID="0a86dd533395e5be7f50b6355cb92a01d747da5176c2d72ee44ccc42676f6f1e" Oct 30 17:02:19.597857 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.597714 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g" Oct 30 17:02:19.629844 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.629757 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:19.720324 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.720292 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73af482c-6523-4268-9ddf-8c4eeefd5f32-kube-api-access-nkj9r\" (UniqueName: \"kubernetes.io/projected/73af482c-6523-4268-9ddf-8c4eeefd5f32-kube-api-access-nkj9r\") pod \"73af482c-6523-4268-9ddf-8c4eeefd5f32\" (UID: \"73af482c-6523-4268-9ddf-8c4eeefd5f32\") " Oct 30 17:02:19.720324 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.720329 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73af482c-6523-4268-9ddf-8c4eeefd5f32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73af482c-6523-4268-9ddf-8c4eeefd5f32-metrics-certs\") pod \"73af482c-6523-4268-9ddf-8c4eeefd5f32\" (UID: \"73af482c-6523-4268-9ddf-8c4eeefd5f32\") " Oct 30 17:02:19.720553 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.720394 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73af482c-6523-4268-9ddf-8c4eeefd5f32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73af482c-6523-4268-9ddf-8c4eeefd5f32-openshift-service-ca\") pod \"73af482c-6523-4268-9ddf-8c4eeefd5f32\" (UID: \"73af482c-6523-4268-9ddf-8c4eeefd5f32\") " Oct 30 17:02:19.720553 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.720456 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73af482c-6523-4268-9ddf-8c4eeefd5f32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73af482c-6523-4268-9ddf-8c4eeefd5f32-maas-api-tls\") pod \"73af482c-6523-4268-9ddf-8c4eeefd5f32\" (UID: \"73af482c-6523-4268-9ddf-8c4eeefd5f32\") " Oct 30 17:02:19.720878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.720850 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73af482c-6523-4268-9ddf-8c4eeefd5f32-openshift-service-ca" pod "73af482c-6523-4268-9ddf-8c4eeefd5f32" (UID: "73af482c-6523-4268-9ddf-8c4eeefd5f32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:02:19.721187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.721166 6202 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73af482c-6523-4268-9ddf-8c4eeefd5f32-openshift-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:02:19.722607 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.722579 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73af482c-6523-4268-9ddf-8c4eeefd5f32-kube-api-access-nkj9r" pod "73af482c-6523-4268-9ddf-8c4eeefd5f32" (UID: "73af482c-6523-4268-9ddf-8c4eeefd5f32"). InnerVolumeSpecName "kube-api-access-nkj9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:02:19.722607 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.722596 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73af482c-6523-4268-9ddf-8c4eeefd5f32-metrics-certs" pod "73af482c-6523-4268-9ddf-8c4eeefd5f32" (UID: "73af482c-6523-4268-9ddf-8c4eeefd5f32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:02:19.722722 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.722612 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73af482c-6523-4268-9ddf-8c4eeefd5f32-maas-api-tls" pod "73af482c-6523-4268-9ddf-8c4eeefd5f32" (UID: "73af482c-6523-4268-9ddf-8c4eeefd5f32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:02:19.822404 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.822370 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkj9r\" (UniqueName: \"kubernetes.io/projected/73af482c-6523-4268-9ddf-8c4eeefd5f32-kube-api-access-nkj9r\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:02:19.822404 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.822405 6202 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73af482c-6523-4268-9ddf-8c4eeefd5f32-metrics-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:02:19.822634 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.822415 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73af482c-6523-4268-9ddf-8c4eeefd5f32-maas-api-tls\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:02:19.858848 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.856337 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:19.971581 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.971496 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6"] Oct 30 17:02:19.976013 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.975985 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g"] Oct 30 17:02:19.979216 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.979198 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f92caf-8x4t6"] Oct 30 17:02:19.982162 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:19.982140 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f92caf-68bf68464c-5899g"] Oct 30 17:02:20.067079 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:20.067047 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:20.307683 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:20.307602 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Oct 30 17:02:20.307875 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:20.307687 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" podUID="e7bcb077-0d1d-4046-9a5f-75fc61dd2886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Oct 30 17:02:20.414284 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:20.414242 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73af482c-6523-4268-9ddf-8c4eeefd5f32" path="/var/lib/kubelet/pods/73af482c-6523-4268-9ddf-8c4eeefd5f32/volumes" Oct 30 17:02:20.414953 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:20.414934 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bbe06bf-8e56-4399-9969-41ade4c450ef" path="/var/lib/kubelet/pods/9bbe06bf-8e56-4399-9969-41ade4c450ef/volumes" Oct 30 17:02:20.470993 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:20.470961 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:21.307287 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:21.307252 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Oct 30 17:02:21.307483 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:21.307304 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" podUID="e7bcb077-0d1d-4046-9a5f-75fc61dd2886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Oct 30 17:02:22.045898 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:22.045804 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:22.312135 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:22.312060 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:02:22.312573 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:22.312550 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:02:22.313386 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:22.313367 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:02:22.859196 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:22.859156 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" podUID="4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:02:23.238255 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:23.238157 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:23.756098 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:23.753457 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:23.765401 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:23.765344 6202 generic.go:356] "Generic (PLEG): container finished" podID="4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e" containerID="c5e0a07769c0bbe69a12e96456ea392d9520705924ee1357c30206eff0195aca" exitCode=0 Oct 30 17:02:23.767423 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:23.767359 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" event={"ID":"4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e","Type":"ContainerDied","Data":"c5e0a07769c0bbe69a12e96456ea392d9520705924ee1357c30206eff0195aca"} Oct 30 17:02:23.773542 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:23.773205 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:23.870103 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:23.870075 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" Oct 30 17:02:23.927725 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:23.927676 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e-kube-api-access-qkbb2\" (UniqueName: \"kubernetes.io/projected/4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e-kube-api-access-qkbb2\") pod \"4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e\" (UID: \"4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e\") " Oct 30 17:02:23.928419 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:23.928391 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e-plugins-config-volume\") pod \"4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e\" (UID: \"4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e\") " Oct 30 17:02:23.929489 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:23.929457 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e-plugins-config-volume" pod "4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e" (UID: "4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:02:23.937508 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:23.937462 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e-kube-api-access-qkbb2" pod "4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e" (UID: "4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e"). InnerVolumeSpecName "kube-api-access-qkbb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:02:24.030364 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:24.030274 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:02:24.030364 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:24.030313 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkbb2\" (UniqueName: \"kubernetes.io/projected/4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e-kube-api-access-qkbb2\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:02:24.050610 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:24.050575 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:24.737107 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:24.737025 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:02:24.775881 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:24.775845 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" Oct 30 17:02:24.775881 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:24.775870 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v" event={"ID":"4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e","Type":"ContainerDied","Data":"9af67f606229102b2f8add6d466080d75a1f3f75af9ea4fe551c2a4565c5d409"} Oct 30 17:02:24.776132 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:24.775963 6202 scope.go:122] "RemoveContainer" containerID="c5e0a07769c0bbe69a12e96456ea392d9520705924ee1357c30206eff0195aca" Oct 30 17:02:24.819816 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:24.819785 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v"] Oct 30 17:02:24.823897 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:24.823867 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f92caf-6db754b875-f5m6v"] Oct 30 17:02:25.898034 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:25.898000 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:26.413321 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:26.413291 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e" path="/var/lib/kubelet/pods/4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e/volumes" Oct 30 17:02:27.109418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:27.109379 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:27.232723 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:27.232687 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:27.496044 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:27.495962 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25"] Oct 30 17:02:27.496231 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:27.496201 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" podUID="26ca8cc6-0f79-47f4-807a-df2ac126efdc" containerName="istio-proxy" containerID="cri-o://2b3c6776af50f271b30c268b58a0052bebb7116e2be7f854a27f869fdbc4aa7f" gracePeriod=30 Oct 30 17:02:27.539387 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:27.539353 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:27.716618 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:27.716583 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:28.116393 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.116337 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:28.571121 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.571097 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j"] Oct 30 17:02:28.571865 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.571843 6202 cpu_manager.go:414] "containerMap: removing container" podUID="4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e" containerName="payload-pre-processing" Oct 30 17:02:28.571865 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.571860 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e" containerName="payload-pre-processing" Oct 30 17:02:28.572072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.571877 6202 cpu_manager.go:414] "containerMap: removing container" podUID="73af482c-6523-4268-9ddf-8c4eeefd5f32" containerName="maas-api" Oct 30 17:02:28.572072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.571882 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73af482c-6523-4268-9ddf-8c4eeefd5f32" containerName="maas-api" Oct 30 17:02:28.572072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.572023 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca8d8fd-c151-4db5-9d26-30e3f51d1e3e" containerName="payload-pre-processing" Oct 30 17:02:28.572072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.572033 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="73af482c-6523-4268-9ddf-8c4eeefd5f32" containerName="maas-api" Oct 30 17:02:28.575720 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.575701 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.595377 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.595354 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-ae5afd-openshift-default-dockercfg-mqb49\"" Oct 30 17:02:28.595588 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.595563 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5018b303-b054-4954-a781-39523acd950f-istio-token\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.595865 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.595848 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-credential-socket\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.595925 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.595911 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-workload-socket\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.595981 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.595953 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5018b303-b054-4954-a781-39523acd950f-istiod-ca-cert\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.596093 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.596076 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-workload-certs\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.596156 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.596106 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5018b303-b054-4954-a781-39523acd950f-istio-podinfo\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.596232 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.596208 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-istio-envoy\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.599729 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.596247 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hpxv\" (UniqueName: \"kubernetes.io/projected/5018b303-b054-4954-a781-39523acd950f-kube-api-access-4hpxv\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.599729 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.596286 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-istio-data\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.613511 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.613480 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j"] Oct 30 17:02:28.628336 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.628308 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:28.715959 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.697615 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5018b303-b054-4954-a781-39523acd950f-istio-podinfo\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.773964 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.773920 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-istio-envoy\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.774245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.774222 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hpxv\" (UniqueName: \"kubernetes.io/projected/5018b303-b054-4954-a781-39523acd950f-kube-api-access-4hpxv\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.774400 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.774383 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-istio-data\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.774658 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.774631 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5018b303-b054-4954-a781-39523acd950f-istio-token\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.775030 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.775010 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-credential-socket\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.775201 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.775181 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-workload-socket\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.775365 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.775349 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5018b303-b054-4954-a781-39523acd950f-istiod-ca-cert\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.775507 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.775491 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-workload-certs\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.776147 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.776124 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-workload-certs\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.786914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.782070 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-istio-data\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.787078 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.731528 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5018b303-b054-4954-a781-39523acd950f-istio-podinfo\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.787184 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.783855 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-credential-socket\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.787271 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.784404 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-workload-socket\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.787360 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.785030 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5018b303-b054-4954-a781-39523acd950f-istiod-ca-cert\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.799329 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.799302 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-istio-envoy\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.816598 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.816569 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5018b303-b054-4954-a781-39523acd950f-istio-token\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.828739 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.828671 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hpxv\" (UniqueName: \"kubernetes.io/projected/5018b303-b054-4954-a781-39523acd950f-kube-api-access-4hpxv\") pod \"e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.894164 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.894131 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:28.929308 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:28.929279 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:29.243490 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:29.243465 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j"] Oct 30 17:02:29.244344 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:02:29.244269 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5018b303_b054_4954_a781_39523acd950f.slice/crio-5eb212cbaa11adfc28c72e813d7977d0c10125e09b4e1943da647863ddd6052f WatchSource:0}: Error finding container 5eb212cbaa11adfc28c72e813d7977d0c10125e09b4e1943da647863ddd6052f: Status 404 returned error can't find the container with id 5eb212cbaa11adfc28c72e813d7977d0c10125e09b4e1943da647863ddd6052f Oct 30 17:02:29.247724 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:29.247688 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:02:29.247860 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:29.247755 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:02:29.247860 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:29.247813 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:02:29.472352 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:29.472313 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:29.608503 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:29.608448 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:29.835266 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:29.835225 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" event={"ID":"5018b303-b054-4954-a781-39523acd950f","Type":"ContainerStarted","Data":"a9b9a6bc6f76cc0355d6d312416ef6df123cf4decec8bbfd6d84590129cbd548"} Oct 30 17:02:29.835543 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:29.835286 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" event={"ID":"5018b303-b054-4954-a781-39523acd950f","Type":"ContainerStarted","Data":"5eb212cbaa11adfc28c72e813d7977d0c10125e09b4e1943da647863ddd6052f"} Oct 30 17:02:29.884281 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:29.884173 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" podStartSLOduration=1.8841577250000001 podStartE2EDuration="1.884157725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:29.880619236 +0000 UTC m=+2463.591073978" watchObservedRunningTime="2026-10-30 17:02:29.884157725 +0000 UTC m=+2463.594612468" Oct 30 17:02:29.894404 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:29.894372 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:30.901139 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:30.901110 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:31.415643 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:31.415605 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:31.853066 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:31.853035 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:31.854166 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:31.854147 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:02:32.421058 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.420617 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:32.546049 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.546017 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:32.591760 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.591730 6202 scope.go:122] "RemoveContainer" containerID="695b9484dd3f46853901e1ee9a29504e46b2022bafbde711458e566d9481e5f4" Oct 30 17:02:32.610645 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.610619 6202 scope.go:122] "RemoveContainer" containerID="ea65e5b21d07716e4fd88f9ad9bee9e524c74b2ca7fb2d9abaa648bb0fef951d" Oct 30 17:02:32.621919 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.621892 6202 scope.go:122] "RemoveContainer" containerID="00d9dfec16320a461356bac36b742f8624ab9805223e9447e20990b817ec0118" Oct 30 17:02:32.633866 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.633845 6202 scope.go:122] "RemoveContainer" containerID="1228a3596b5c9528aea24640cc1982cca57f71ebb6f6ad3a4fa1a7b3734f8018" Oct 30 17:02:32.815431 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.815408 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 17:02:32.865639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.865605 6202 generic.go:356] "Generic (PLEG): container finished" podID="26ca8cc6-0f79-47f4-807a-df2ac126efdc" containerID="2b3c6776af50f271b30c268b58a0052bebb7116e2be7f854a27f869fdbc4aa7f" exitCode=0 Oct 30 17:02:32.865876 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.865688 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" Oct 30 17:02:32.865876 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.865680 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" event={"ID":"26ca8cc6-0f79-47f4-807a-df2ac126efdc","Type":"ContainerDied","Data":"2b3c6776af50f271b30c268b58a0052bebb7116e2be7f854a27f869fdbc4aa7f"} Oct 30 17:02:32.865876 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.865839 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25" event={"ID":"26ca8cc6-0f79-47f4-807a-df2ac126efdc","Type":"ContainerDied","Data":"19020ecf74691ae284c007cffb334dd58cc00b3056ee5f555570b703b1f97714"} Oct 30 17:02:32.866153 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.865880 6202 scope.go:122] "RemoveContainer" containerID="2b3c6776af50f271b30c268b58a0052bebb7116e2be7f854a27f869fdbc4aa7f" Oct 30 17:02:32.880322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.880037 6202 scope.go:122] "RemoveContainer" containerID="2b3c6776af50f271b30c268b58a0052bebb7116e2be7f854a27f869fdbc4aa7f" Oct 30 17:02:32.880503 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:02:32.880466 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3c6776af50f271b30c268b58a0052bebb7116e2be7f854a27f869fdbc4aa7f\": container with ID starting with 2b3c6776af50f271b30c268b58a0052bebb7116e2be7f854a27f869fdbc4aa7f not found: ID does not exist" containerID="2b3c6776af50f271b30c268b58a0052bebb7116e2be7f854a27f869fdbc4aa7f" Oct 30 17:02:32.880661 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.880514 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3c6776af50f271b30c268b58a0052bebb7116e2be7f854a27f869fdbc4aa7f"} err="failed to get container status \"2b3c6776af50f271b30c268b58a0052bebb7116e2be7f854a27f869fdbc4aa7f\": rpc error: code = NotFound desc = could not find container \"2b3c6776af50f271b30c268b58a0052bebb7116e2be7f854a27f869fdbc4aa7f\": container with ID starting with 2b3c6776af50f271b30c268b58a0052bebb7116e2be7f854a27f869fdbc4aa7f not found: ID does not exist" Oct 30 17:02:32.943474 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.943389 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-podinfo\") pod \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " Oct 30 17:02:32.943474 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.943429 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-token\" (UniqueName: \"kubernetes.io/projected/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-token\") pod \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " Oct 30 17:02:32.943474 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.943474 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-envoy\") pod \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " Oct 30 17:02:32.943787 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.943540 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-workload-socket\") pod \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " Oct 30 17:02:32.943787 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.943622 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ca8cc6-0f79-47f4-807a-df2ac126efdc-kube-api-access-dqj7v\" (UniqueName: \"kubernetes.io/projected/26ca8cc6-0f79-47f4-807a-df2ac126efdc-kube-api-access-dqj7v\") pod \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " Oct 30 17:02:32.943787 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.943664 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istiod-ca-cert\") pod \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " Oct 30 17:02:32.943787 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.943730 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-credential-socket\") pod \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " Oct 30 17:02:32.944045 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.943847 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-workload-certs\") pod \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " Oct 30 17:02:32.944045 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.943876 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-data\") pod \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\" (UID: \"26ca8cc6-0f79-47f4-807a-df2ac126efdc\") " Oct 30 17:02:32.944699 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.944655 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-data" pod "26ca8cc6-0f79-47f4-807a-df2ac126efdc" (UID: "26ca8cc6-0f79-47f4-807a-df2ac126efdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:02:32.944699 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.944668 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-workload-certs" pod "26ca8cc6-0f79-47f4-807a-df2ac126efdc" (UID: "26ca8cc6-0f79-47f4-807a-df2ac126efdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:02:32.944699 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.944697 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-workload-socket" pod "26ca8cc6-0f79-47f4-807a-df2ac126efdc" (UID: "26ca8cc6-0f79-47f4-807a-df2ac126efdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:02:32.944988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.944731 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-credential-socket" pod "26ca8cc6-0f79-47f4-807a-df2ac126efdc" (UID: "26ca8cc6-0f79-47f4-807a-df2ac126efdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:02:32.944988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.944757 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istiod-ca-cert" pod "26ca8cc6-0f79-47f4-807a-df2ac126efdc" (UID: "26ca8cc6-0f79-47f4-807a-df2ac126efdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:02:32.946186 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.946164 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-podinfo" pod "26ca8cc6-0f79-47f4-807a-df2ac126efdc" (UID: "26ca8cc6-0f79-47f4-807a-df2ac126efdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:02:32.947321 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.947293 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ca8cc6-0f79-47f4-807a-df2ac126efdc-kube-api-access-dqj7v" pod "26ca8cc6-0f79-47f4-807a-df2ac126efdc" (UID: "26ca8cc6-0f79-47f4-807a-df2ac126efdc"). InnerVolumeSpecName "kube-api-access-dqj7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:02:32.947426 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.947365 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-envoy" pod "26ca8cc6-0f79-47f4-807a-df2ac126efdc" (UID: "26ca8cc6-0f79-47f4-807a-df2ac126efdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:02:32.948113 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:32.948090 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-token" pod "26ca8cc6-0f79-47f4-807a-df2ac126efdc" (UID: "26ca8cc6-0f79-47f4-807a-df2ac126efdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:02:33.046654 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:33.046617 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:02:33.046654 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:33.046650 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:02:33.046918 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:33.046667 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:02:33.046918 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:33.046680 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:02:33.046918 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:33.046693 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqj7v\" (UniqueName: \"kubernetes.io/projected/26ca8cc6-0f79-47f4-807a-df2ac126efdc-kube-api-access-dqj7v\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:02:33.046918 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:33.046708 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:02:33.046918 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:33.046721 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:02:33.046918 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:33.046734 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:02:33.046918 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:33.046749 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ca8cc6-0f79-47f4-807a-df2ac126efdc-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:02:33.214205 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:33.214136 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25"] Oct 30 17:02:33.226051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:33.226028 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f92caf-openshift-default-56d7b858bc-q7h25"] Oct 30 17:02:33.397030 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:33.396991 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:34.105324 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.105232 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:34.230061 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.230025 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:34.273759 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.273714 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx"] Oct 30 17:02:34.274897 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.274871 6202 cpu_manager.go:414] "containerMap: removing container" podUID="26ca8cc6-0f79-47f4-807a-df2ac126efdc" containerName="istio-proxy" Oct 30 17:02:34.274897 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.274896 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ca8cc6-0f79-47f4-807a-df2ac126efdc" containerName="istio-proxy" Oct 30 17:02:34.275133 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.275115 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ca8cc6-0f79-47f4-807a-df2ac126efdc" containerName="istio-proxy" Oct 30 17:02:34.279723 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.279703 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" Oct 30 17:02:34.297614 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.297589 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx"] Oct 30 17:02:34.305251 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.305229 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-ae5afd\"" Oct 30 17:02:34.305369 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.305334 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-ae5afd-dockercfg-6tnd2\"" Oct 30 17:02:34.503882 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.493459 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w82nc\" (UniqueName: \"kubernetes.io/projected/b123429e-9af8-402a-b91a-1a0c2fdc7697-kube-api-access-w82nc\") pod \"payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx\" (UID: \"b123429e-9af8-402a-b91a-1a0c2fdc7697\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" Oct 30 17:02:34.503882 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.493688 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b123429e-9af8-402a-b91a-1a0c2fdc7697-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx\" (UID: \"b123429e-9af8-402a-b91a-1a0c2fdc7697\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" Oct 30 17:02:34.508583 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.508529 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ca8cc6-0f79-47f4-807a-df2ac126efdc" path="/var/lib/kubelet/pods/26ca8cc6-0f79-47f4-807a-df2ac126efdc/volumes" Oct 30 17:02:34.509206 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.509181 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28"] Oct 30 17:02:34.522845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.522745 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28"] Oct 30 17:02:34.523057 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.522879 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" Oct 30 17:02:34.594818 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.594784 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w82nc\" (UniqueName: \"kubernetes.io/projected/b123429e-9af8-402a-b91a-1a0c2fdc7697-kube-api-access-w82nc\") pod \"payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx\" (UID: \"b123429e-9af8-402a-b91a-1a0c2fdc7697\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" Oct 30 17:02:34.594971 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.594888 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b123429e-9af8-402a-b91a-1a0c2fdc7697-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx\" (UID: \"b123429e-9af8-402a-b91a-1a0c2fdc7697\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" Oct 30 17:02:34.595511 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.595492 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b123429e-9af8-402a-b91a-1a0c2fdc7697-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx\" (UID: \"b123429e-9af8-402a-b91a-1a0c2fdc7697\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" Oct 30 17:02:34.621577 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.621549 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w82nc\" (UniqueName: \"kubernetes.io/projected/b123429e-9af8-402a-b91a-1a0c2fdc7697-kube-api-access-w82nc\") pod \"payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx\" (UID: \"b123429e-9af8-402a-b91a-1a0c2fdc7697\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" Oct 30 17:02:34.696309 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.696281 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbddf\" (UniqueName: \"kubernetes.io/projected/3807a7b3-8948-416d-97fe-e94528e239d2-kube-api-access-qbddf\") pod \"payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28\" (UID: \"3807a7b3-8948-416d-97fe-e94528e239d2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" Oct 30 17:02:34.696492 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.696438 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3807a7b3-8948-416d-97fe-e94528e239d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28\" (UID: \"3807a7b3-8948-416d-97fe-e94528e239d2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" Oct 30 17:02:34.797580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.797482 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3807a7b3-8948-416d-97fe-e94528e239d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28\" (UID: \"3807a7b3-8948-416d-97fe-e94528e239d2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" Oct 30 17:02:34.797759 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.797639 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbddf\" (UniqueName: \"kubernetes.io/projected/3807a7b3-8948-416d-97fe-e94528e239d2-kube-api-access-qbddf\") pod \"payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28\" (UID: \"3807a7b3-8948-416d-97fe-e94528e239d2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" Oct 30 17:02:34.798247 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.798227 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3807a7b3-8948-416d-97fe-e94528e239d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28\" (UID: \"3807a7b3-8948-416d-97fe-e94528e239d2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" Oct 30 17:02:34.821136 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.821111 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbddf\" (UniqueName: \"kubernetes.io/projected/3807a7b3-8948-416d-97fe-e94528e239d2-kube-api-access-qbddf\") pod \"payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28\" (UID: \"3807a7b3-8948-416d-97fe-e94528e239d2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" Oct 30 17:02:34.849377 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.849347 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" Oct 30 17:02:34.893796 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:34.893738 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" Oct 30 17:02:35.155608 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:35.155578 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28"] Oct 30 17:02:35.156076 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:02:35.156022 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3807a7b3_8948_416d_97fe_e94528e239d2.slice/crio-3c325d24a4edd01e8151fd6b218ff1a6459fb317d7c936438d5ee6abf60272ed WatchSource:0}: Error finding container 3c325d24a4edd01e8151fd6b218ff1a6459fb317d7c936438d5ee6abf60272ed: Status 404 returned error can't find the container with id 3c325d24a4edd01e8151fd6b218ff1a6459fb317d7c936438d5ee6abf60272ed Oct 30 17:02:35.215901 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:35.215856 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx"] Oct 30 17:02:35.222309 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:02:35.222283 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb123429e_9af8_402a_b91a_1a0c2fdc7697.slice/crio-a4ffbeb4ab72ad6d96ab4c85d41eeb1592bd11cb54cc5f7454a7ef19528d6930 WatchSource:0}: Error finding container a4ffbeb4ab72ad6d96ab4c85d41eeb1592bd11cb54cc5f7454a7ef19528d6930: Status 404 returned error can't find the container with id a4ffbeb4ab72ad6d96ab4c85d41eeb1592bd11cb54cc5f7454a7ef19528d6930 Oct 30 17:02:35.295308 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:35.295275 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:35.904210 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:35.904174 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" event={"ID":"b123429e-9af8-402a-b91a-1a0c2fdc7697","Type":"ContainerStarted","Data":"267285a07aaec76c93ac695ab23734819cd5035e61889d38d0c8a760d990e767"} Oct 30 17:02:35.904402 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:35.904266 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" event={"ID":"b123429e-9af8-402a-b91a-1a0c2fdc7697","Type":"ContainerStarted","Data":"a4ffbeb4ab72ad6d96ab4c85d41eeb1592bd11cb54cc5f7454a7ef19528d6930"} Oct 30 17:02:35.904449 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:35.904401 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" Oct 30 17:02:35.905956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:35.905931 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" event={"ID":"3807a7b3-8948-416d-97fe-e94528e239d2","Type":"ContainerStarted","Data":"771bb3e055ce6ba9ef32a1134ff9a5894652e6924b717421b4cb5e7ea4242003"} Oct 30 17:02:35.906082 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:35.905970 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" event={"ID":"3807a7b3-8948-416d-97fe-e94528e239d2","Type":"ContainerStarted","Data":"3c325d24a4edd01e8151fd6b218ff1a6459fb317d7c936438d5ee6abf60272ed"} Oct 30 17:02:35.906082 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:35.906014 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" Oct 30 17:02:35.946802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:35.946727 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" podStartSLOduration=1.946711901 podStartE2EDuration="1.946711901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:35.945275906 +0000 UTC m=+2469.655730649" watchObservedRunningTime="2026-10-30 17:02:35.946711901 +0000 UTC m=+2469.657166646" Oct 30 17:02:35.994475 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:35.994428 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" podStartSLOduration=1.994416174 podStartE2EDuration="1.994416174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:35.991259169 +0000 UTC m=+2469.701713913" watchObservedRunningTime="2026-10-30 17:02:35.994416174 +0000 UTC m=+2469.704870942" Oct 30 17:02:36.954031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:36.953997 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:38.620198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:38.620164 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:39.779651 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:39.779617 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:40.408394 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:40.408351 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:41.062849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:41.062807 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:41.365365 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:41.360204 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:42.151328 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:42.151294 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:43.869128 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:43.869095 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:44.070476 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:02:44.070434 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:02:45.565167 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:45.565125 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:45.799631 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:45.799591 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:46.917122 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:46.917089 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" Oct 30 17:02:46.917511 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:46.917288 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" Oct 30 17:02:47.282230 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:47.282156 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:49.020257 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:49.020214 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:49.032391 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:49.032364 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb"] Oct 30 17:02:49.045881 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:49.045849 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa2c7f7-crqbb"] Oct 30 17:02:49.753801 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:49.753710 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:49.894115 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:49.892491 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:50.416455 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:50.416419 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3638dac-29cc-4e5e-bc9b-3de82be10a96" path="/var/lib/kubelet/pods/f3638dac-29cc-4e5e-bc9b-3de82be10a96/volumes" Oct 30 17:02:51.194274 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:51.194231 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:52.936535 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:52.936502 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:54.640892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:54.640859 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:56.479412 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:56.479355 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:58.190988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:58.190953 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:59.885567 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:02:59.885535 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:01.784095 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:01.784055 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:03.489027 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:03.488998 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:05.118648 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:05.118603 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:06.886164 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:06.886123 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:08.080237 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.080203 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:08.363688 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.363617 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw"] Oct 30 17:03:08.368256 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.368235 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.385596 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.385567 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw"] Oct 30 17:03:08.393380 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.393354 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-1b7bb3\"/\"kube-root-ca.crt\"" Oct 30 17:03:08.397754 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.397734 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-1b7bb3\"/\"default-dockercfg-p8dsf\"" Oct 30 17:03:08.397884 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.397792 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-1b7bb3\"/\"openshift-service-ca.crt\"" Oct 30 17:03:08.397932 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.397902 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-1b7bb3\"/\"auth-test-model-1b7bb3-kserve-self-signed-certs\"" Oct 30 17:03:08.400431 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.400410 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:08.525743 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.525709 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hsr9\" (UniqueName: \"kubernetes.io/projected/53372428-2070-44b7-89b4-5218dfd08ebf-kube-api-access-6hsr9\") pod \"auth-test-model-1b7bb3-kserve-54d7956885-dq4mw\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.525975 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.525753 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-tmp-dir\") pod \"auth-test-model-1b7bb3-kserve-54d7956885-dq4mw\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.526075 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.526055 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-home\") pod \"auth-test-model-1b7bb3-kserve-54d7956885-dq4mw\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.526159 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.526136 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-dshm\") pod \"auth-test-model-1b7bb3-kserve-54d7956885-dq4mw\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.526336 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.526318 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53372428-2070-44b7-89b4-5218dfd08ebf-tls-certs\") pod \"auth-test-model-1b7bb3-kserve-54d7956885-dq4mw\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.526416 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.526357 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-model-cache\") pod \"auth-test-model-1b7bb3-kserve-54d7956885-dq4mw\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.632068 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.631972 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53372428-2070-44b7-89b4-5218dfd08ebf-tls-certs\") pod \"auth-test-model-1b7bb3-kserve-54d7956885-dq4mw\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.632391 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.632318 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-model-cache\") pod \"auth-test-model-1b7bb3-kserve-54d7956885-dq4mw\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.632632 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.632612 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hsr9\" (UniqueName: \"kubernetes.io/projected/53372428-2070-44b7-89b4-5218dfd08ebf-kube-api-access-6hsr9\") pod \"auth-test-model-1b7bb3-kserve-54d7956885-dq4mw\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.632832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.632760 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-tmp-dir\") pod \"auth-test-model-1b7bb3-kserve-54d7956885-dq4mw\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.634753 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.634731 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-home\") pod \"auth-test-model-1b7bb3-kserve-54d7956885-dq4mw\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.635270 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.635250 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-dshm\") pod \"auth-test-model-1b7bb3-kserve-54d7956885-dq4mw\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.635615 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.634054 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-model-cache\") pod \"auth-test-model-1b7bb3-kserve-54d7956885-dq4mw\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.635727 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.635177 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-home\") pod \"auth-test-model-1b7bb3-kserve-54d7956885-dq4mw\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.635838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.634564 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-tmp-dir\") pod \"auth-test-model-1b7bb3-kserve-54d7956885-dq4mw\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.673443 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.673408 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53372428-2070-44b7-89b4-5218dfd08ebf-tls-certs\") pod \"auth-test-model-1b7bb3-kserve-54d7956885-dq4mw\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.674196 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.674168 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hsr9\" (UniqueName: \"kubernetes.io/projected/53372428-2070-44b7-89b4-5218dfd08ebf-kube-api-access-6hsr9\") pod \"auth-test-model-1b7bb3-kserve-54d7956885-dq4mw\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.676323 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.676294 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-dshm\") pod \"auth-test-model-1b7bb3-kserve-54d7956885-dq4mw\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.684065 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.684042 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:08.694328 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.694298 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:08.826245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.826197 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:08.943013 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.942900 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:08.993729 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:03:08.993699 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53372428_2070_44b7_89b4_5218dfd08ebf.slice/crio-c3041ec130b1c09e81fd3dd4514202fcf19d2b142fcd15da305864c639e900ee WatchSource:0}: Error finding container c3041ec130b1c09e81fd3dd4514202fcf19d2b142fcd15da305864c639e900ee: Status 404 returned error can't find the container with id c3041ec130b1c09e81fd3dd4514202fcf19d2b142fcd15da305864c639e900ee Oct 30 17:03:08.994401 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:08.994380 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw"] Oct 30 17:03:09.058392 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:09.058361 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:09.243654 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:09.243574 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" event={"ID":"53372428-2070-44b7-89b4-5218dfd08ebf","Type":"ContainerStarted","Data":"81bd7c5bdae6827ef0226b4771acebf931c4d2ebf3416a2225267b25f3a37231"} Oct 30 17:03:09.243654 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:09.243615 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" event={"ID":"53372428-2070-44b7-89b4-5218dfd08ebf","Type":"ContainerStarted","Data":"c3041ec130b1c09e81fd3dd4514202fcf19d2b142fcd15da305864c639e900ee"} Oct 30 17:03:09.286057 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:09.286010 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" podStartSLOduration=1.285995462 podStartE2EDuration="1.285995462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:09.284155668 +0000 UTC m=+2502.994610410" watchObservedRunningTime="2026-10-30 17:03:09.285995462 +0000 UTC m=+2502.996450204" Oct 30 17:03:09.685698 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:09.685646 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:10.472396 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:10.472361 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:12.295039 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:12.295001 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:14.072493 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:03:14.072455 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:03:14.073585 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:14.073555 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:15.818928 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:15.818889 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:17.517420 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:17.517385 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:17.684787 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:17.684732 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:18.684988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:18.684962 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:18.697851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:18.697822 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:19.336851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:19.336822 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:03:19.443829 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:19.443800 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:20.899552 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:20.899517 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:21.210529 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:21.210449 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:23.164785 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:23.164727 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:24.875327 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:24.875283 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:26.623716 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.623641 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:26.627474 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.627443 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q"] Oct 30 17:03:26.634437 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.634405 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.662533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.662504 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-9e533f-openshift-default-dockercfg-x62c6\"" Oct 30 17:03:26.668357 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.668327 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q"] Oct 30 17:03:26.719885 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.719842 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-podinfo\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.720080 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.719907 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-token\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.720080 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.719981 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320b55f1-a255-4f32-8b97-26cd86755a2d-istiod-ca-cert\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.720169 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.720111 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz44k\" (UniqueName: \"kubernetes.io/projected/320b55f1-a255-4f32-8b97-26cd86755a2d-kube-api-access-gz44k\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.720169 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.720134 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-workload-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.720253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.720171 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-credential-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.720344 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.720324 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-envoy\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.720396 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.720368 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-data\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.720455 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.720410 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-workload-certs\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.821990 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.821953 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-envoy\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.822171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.822004 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-data\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.822171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.822067 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-workload-certs\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.822171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.822143 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-podinfo\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.822336 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.822184 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-token\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.822336 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.822205 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320b55f1-a255-4f32-8b97-26cd86755a2d-istiod-ca-cert\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.822336 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.822321 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz44k\" (UniqueName: \"kubernetes.io/projected/320b55f1-a255-4f32-8b97-26cd86755a2d-kube-api-access-gz44k\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.822505 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.822351 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-workload-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.822505 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.822407 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-credential-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.822505 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.822405 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-data\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.822505 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.822478 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-workload-certs\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.822998 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.822976 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-workload-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.823210 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.823190 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320b55f1-a255-4f32-8b97-26cd86755a2d-istiod-ca-cert\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.823251 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.823217 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-credential-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.824563 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.824543 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-envoy\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.824696 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.824675 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-podinfo\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.850340 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.850307 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-token\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.850594 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.850568 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz44k\" (UniqueName: \"kubernetes.io/projected/320b55f1-a255-4f32-8b97-26cd86755a2d-kube-api-access-gz44k\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:26.951403 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:26.951293 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:27.266700 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:27.266666 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q"] Oct 30 17:03:27.267128 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:03:27.267102 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320b55f1_a255_4f32_8b97_26cd86755a2d.slice/crio-2c1ebf7390ed91334d94aab7072d611e205a437eece005b12911e79b55e2a61c WatchSource:0}: Error finding container 2c1ebf7390ed91334d94aab7072d611e205a437eece005b12911e79b55e2a61c: Status 404 returned error can't find the container with id 2c1ebf7390ed91334d94aab7072d611e205a437eece005b12911e79b55e2a61c Oct 30 17:03:27.270149 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:27.270113 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:03:27.270255 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:27.270179 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:03:27.270255 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:27.270208 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:03:27.402414 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:27.402370 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" event={"ID":"320b55f1-a255-4f32-8b97-26cd86755a2d","Type":"ContainerStarted","Data":"2351ddff425996d75d55bc9dd9a5452efcb8987562784bc7a02a50d68b535e1f"} Oct 30 17:03:27.402612 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:27.402455 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" event={"ID":"320b55f1-a255-4f32-8b97-26cd86755a2d","Type":"ContainerStarted","Data":"2c1ebf7390ed91334d94aab7072d611e205a437eece005b12911e79b55e2a61c"} Oct 30 17:03:27.448599 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:27.448543 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" podStartSLOduration=1.448529291 podStartE2EDuration="1.448529291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:27.448046191 +0000 UTC m=+2521.158500968" watchObservedRunningTime="2026-10-30 17:03:27.448529291 +0000 UTC m=+2521.158984054" Oct 30 17:03:27.954791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:27.953864 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:28.221603 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:28.221516 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:28.968793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:28.954670 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Oct 30 17:03:28.968793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:28.954734 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" podUID="320b55f1-a255-4f32-8b97-26cd86755a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Oct 30 17:03:29.962530 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:29.962162 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Oct 30 17:03:29.962530 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:29.962242 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" podUID="320b55f1-a255-4f32-8b97-26cd86755a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Oct 30 17:03:29.966789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:29.963056 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:30.964792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:30.953206 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Oct 30 17:03:30.964792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:30.953268 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" podUID="320b55f1-a255-4f32-8b97-26cd86755a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Oct 30 17:03:31.816077 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:31.816044 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:31.958789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:31.952327 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Oct 30 17:03:31.958789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:31.952393 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" podUID="320b55f1-a255-4f32-8b97-26cd86755a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Oct 30 17:03:32.785316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:32.785268 6202 scope.go:122] "RemoveContainer" containerID="eb56478b60ff9cba2b4689b35781e709766a58830b9fd95ed7ba4a14348b62ab" Oct 30 17:03:32.953794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:32.952387 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Oct 30 17:03:32.953794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:32.952449 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" podUID="320b55f1-a255-4f32-8b97-26cd86755a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Oct 30 17:03:34.009119 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:34.009088 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:34.120431 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:34.120395 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:34.121180 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:34.121147 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:34.122025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:34.122004 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:03:35.691809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:35.691712 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:37.419331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:37.419287 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:38.033688 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:38.033642 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf"] Oct 30 17:03:38.049012 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:38.048971 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36323b68-5hvgf"] Oct 30 17:03:38.412820 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:38.412789 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc229aa-6e25-47b4-939d-23d34a39cd30" path="/var/lib/kubelet/pods/6bc229aa-6e25-47b4-939d-23d34a39cd30/volumes" Oct 30 17:03:39.189992 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:39.189963 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:40.887089 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:40.887054 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:41.227281 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:41.227195 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:42.743114 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:42.743070 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:43.030849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:43.030750 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k"] Oct 30 17:03:43.041986 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:43.041959 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbd1d14b-jdb2k"] Oct 30 17:03:44.072946 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:03:44.072905 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:03:44.412294 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:44.412217 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2363fccd-b8eb-4896-a0e4-61af066ae754" path="/var/lib/kubelet/pods/2363fccd-b8eb-4896-a0e4-61af066ae754/volumes" Oct 30 17:03:44.450000 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:44.449968 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:46.374019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:46.373988 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:48.104912 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:48.104869 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:49.077147 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:49.077114 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w"] Oct 30 17:03:49.113041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:49.112994 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:49.829908 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:49.829871 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:51.524336 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:51.524295 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:53.256791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:53.256345 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:54.569791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:54.569428 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:54.961811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:54.961174 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:56.666308 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:56.666252 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:58.528907 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:58.528871 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:59.176038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:59.176009 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:59.288191 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:59.288135 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:59.764760 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:59.764727 6202 generic.go:356] "Generic (PLEG): container finished" podID="5018b303-b054-4954-a781-39523acd950f" containerID="a9b9a6bc6f76cc0355d6d312416ef6df123cf4decec8bbfd6d84590129cbd548" exitCode=0 Oct 30 17:03:59.765290 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:59.764814 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" event={"ID":"5018b303-b054-4954-a781-39523acd950f","Type":"ContainerDied","Data":"a9b9a6bc6f76cc0355d6d312416ef6df123cf4decec8bbfd6d84590129cbd548"} Oct 30 17:03:59.765393 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:59.765376 6202 scope.go:122] "RemoveContainer" containerID="a9b9a6bc6f76cc0355d6d312416ef6df123cf4decec8bbfd6d84590129cbd548" Oct 30 17:03:59.767122 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:59.767095 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:03:59.767208 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:59.767162 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:03:59.767246 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:03:59.767218 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:04:00.393572 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:00.393538 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:00.776661 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:00.776577 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" event={"ID":"5018b303-b054-4954-a781-39523acd950f","Type":"ContainerStarted","Data":"f0d0b3293d58a510c5c3d9b40d5b680f266ba358017621f167787089cac14d82"} Oct 30 17:04:00.894759 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:00.894731 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:04:00.895941 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:00.895894 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body= Oct 30 17:04:00.896093 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:00.895955 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" podUID="5018b303-b054-4954-a781-39523acd950f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" Oct 30 17:04:01.858380 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:01.857943 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:04:01.895041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:01.895008 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body= Oct 30 17:04:01.895234 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:01.895070 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" podUID="5018b303-b054-4954-a781-39523acd950f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" Oct 30 17:04:02.214976 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:02.214891 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:02.894625 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:02.894587 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body= Oct 30 17:04:02.895077 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:02.894779 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" podUID="5018b303-b054-4954-a781-39523acd950f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" Oct 30 17:04:03.895101 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:03.895062 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body= Oct 30 17:04:03.895598 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:03.895127 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" podUID="5018b303-b054-4954-a781-39523acd950f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" Oct 30 17:04:04.041725 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.041683 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt"] Oct 30 17:04:04.076296 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.076265 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bf8bcf7-nwxrt"] Oct 30 17:04:04.105861 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.105792 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" podUID="65d54ddf-f0f1-4271-8025-361a9f3e3507" containerName="main" containerID="cri-o://8ebe2be664e77ccdfe91b213930b0273f6097284828029dbb5ad65948f05cd63" gracePeriod=45 Oct 30 17:04:04.132242 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.132218 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:04.413574 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.413540 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="250d2276-7b40-4aab-87dd-2a342eba32fc" path="/var/lib/kubelet/pods/250d2276-7b40-4aab-87dd-2a342eba32fc/volumes" Oct 30 17:04:04.483014 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.482934 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:04:04.607277 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.607242 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-model-cache\") pod \"65d54ddf-f0f1-4271-8025-361a9f3e3507\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " Oct 30 17:04:04.607510 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.607487 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-dshm\") pod \"65d54ddf-f0f1-4271-8025-361a9f3e3507\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " Oct 30 17:04:04.607601 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.607503 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-model-cache" pod "65d54ddf-f0f1-4271-8025-361a9f3e3507" (UID: "65d54ddf-f0f1-4271-8025-361a9f3e3507"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:04:04.607601 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.607551 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d54ddf-f0f1-4271-8025-361a9f3e3507-kube-api-access-zxg9s\" (UniqueName: \"kubernetes.io/projected/65d54ddf-f0f1-4271-8025-361a9f3e3507-kube-api-access-zxg9s\") pod \"65d54ddf-f0f1-4271-8025-361a9f3e3507\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " Oct 30 17:04:04.607723 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.607651 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65d54ddf-f0f1-4271-8025-361a9f3e3507-tls-certs\" (UniqueName: \"kubernetes.io/secret/65d54ddf-f0f1-4271-8025-361a9f3e3507-tls-certs\") pod \"65d54ddf-f0f1-4271-8025-361a9f3e3507\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " Oct 30 17:04:04.607723 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.607708 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-tmp-dir\") pod \"65d54ddf-f0f1-4271-8025-361a9f3e3507\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " Oct 30 17:04:04.608143 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.608119 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-home\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-home\") pod \"65d54ddf-f0f1-4271-8025-361a9f3e3507\" (UID: \"65d54ddf-f0f1-4271-8025-361a9f3e3507\") " Oct 30 17:04:04.608892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.608867 6202 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-model-cache\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:04.609152 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.609129 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-home" pod "65d54ddf-f0f1-4271-8025-361a9f3e3507" (UID: "65d54ddf-f0f1-4271-8025-361a9f3e3507"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:04:04.609336 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.609316 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-tmp-dir" pod "65d54ddf-f0f1-4271-8025-361a9f3e3507" (UID: "65d54ddf-f0f1-4271-8025-361a9f3e3507"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:04:04.612189 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.612166 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-dshm" pod "65d54ddf-f0f1-4271-8025-361a9f3e3507" (UID: "65d54ddf-f0f1-4271-8025-361a9f3e3507"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:04:04.617225 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.616917 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d54ddf-f0f1-4271-8025-361a9f3e3507-kube-api-access-zxg9s" pod "65d54ddf-f0f1-4271-8025-361a9f3e3507" (UID: "65d54ddf-f0f1-4271-8025-361a9f3e3507"). InnerVolumeSpecName "kube-api-access-zxg9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:04.618450 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.618424 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65d54ddf-f0f1-4271-8025-361a9f3e3507-tls-certs" pod "65d54ddf-f0f1-4271-8025-361a9f3e3507" (UID: "65d54ddf-f0f1-4271-8025-361a9f3e3507"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:04:04.710106 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.710074 6202 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-dshm\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:04.710106 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.710103 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxg9s\" (UniqueName: \"kubernetes.io/projected/65d54ddf-f0f1-4271-8025-361a9f3e3507-kube-api-access-zxg9s\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:04.710106 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.710113 6202 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65d54ddf-f0f1-4271-8025-361a9f3e3507-tls-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:04.710348 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.710122 6202 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-tmp-dir\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:04.710348 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.710130 6202 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65d54ddf-f0f1-4271-8025-361a9f3e3507-home\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:04.810052 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.809963 6202 generic.go:356] "Generic (PLEG): container finished" podID="65d54ddf-f0f1-4271-8025-361a9f3e3507" containerID="8ebe2be664e77ccdfe91b213930b0273f6097284828029dbb5ad65948f05cd63" exitCode=0 Oct 30 17:04:04.810052 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.810013 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" event={"ID":"65d54ddf-f0f1-4271-8025-361a9f3e3507","Type":"ContainerDied","Data":"8ebe2be664e77ccdfe91b213930b0273f6097284828029dbb5ad65948f05cd63"} Oct 30 17:04:04.810275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.810063 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" event={"ID":"65d54ddf-f0f1-4271-8025-361a9f3e3507","Type":"ContainerDied","Data":"ee68b579c3e7cdbf393e07165680c6e448a6e09185fcbeaba5fc65fceb7e1357"} Oct 30 17:04:04.810275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.810080 6202 scope.go:122] "RemoveContainer" containerID="8ebe2be664e77ccdfe91b213930b0273f6097284828029dbb5ad65948f05cd63" Oct 30 17:04:04.810275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.810023 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w" Oct 30 17:04:04.821792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.821742 6202 scope.go:122] "RemoveContainer" containerID="8ebe2be664e77ccdfe91b213930b0273f6097284828029dbb5ad65948f05cd63" Oct 30 17:04:04.822067 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:04:04.822049 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebe2be664e77ccdfe91b213930b0273f6097284828029dbb5ad65948f05cd63\": container with ID starting with 8ebe2be664e77ccdfe91b213930b0273f6097284828029dbb5ad65948f05cd63 not found: ID does not exist" containerID="8ebe2be664e77ccdfe91b213930b0273f6097284828029dbb5ad65948f05cd63" Oct 30 17:04:04.822119 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.822075 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebe2be664e77ccdfe91b213930b0273f6097284828029dbb5ad65948f05cd63"} err="failed to get container status \"8ebe2be664e77ccdfe91b213930b0273f6097284828029dbb5ad65948f05cd63\": rpc error: code = NotFound desc = could not find container \"8ebe2be664e77ccdfe91b213930b0273f6097284828029dbb5ad65948f05cd63\": container with ID starting with 8ebe2be664e77ccdfe91b213930b0273f6097284828029dbb5ad65948f05cd63 not found: ID does not exist" Oct 30 17:04:04.883845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.883341 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w"] Oct 30 17:04:04.894580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.894553 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-e43400/auth-test-model-e43400-kserve-5bf97847cc-wpd9w"] Oct 30 17:04:04.894816 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.894791 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body= Oct 30 17:04:04.894933 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:04.894856 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" podUID="5018b303-b054-4954-a781-39523acd950f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" Oct 30 17:04:05.823932 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:05.823881 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:05.912617 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:05.912583 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:04:05.914539 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:05.914510 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:04:06.412503 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:06.412464 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d54ddf-f0f1-4271-8025-361a9f3e3507" path="/var/lib/kubelet/pods/65d54ddf-f0f1-4271-8025-361a9f3e3507/volumes" Oct 30 17:04:07.535703 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:07.535669 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:09.236574 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:09.236538 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:10.871759 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:10.871723 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:12.494963 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:12.494922 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:14.076433 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:04:14.076391 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:04:14.267553 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:14.267521 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:16.085301 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:16.085264 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:17.898745 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:17.898703 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:20.027806 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.027744 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:20.527197 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.527163 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9"] Oct 30 17:04:20.528094 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.528062 6202 cpu_manager.go:414] "containerMap: removing container" podUID="65d54ddf-f0f1-4271-8025-361a9f3e3507" containerName="main" Oct 30 17:04:20.528094 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.528092 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d54ddf-f0f1-4271-8025-361a9f3e3507" containerName="main" Oct 30 17:04:20.528406 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.528386 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d54ddf-f0f1-4271-8025-361a9f3e3507" containerName="main" Oct 30 17:04:20.534257 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.534233 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.560277 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.560244 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-661509\"/\"default-dockercfg-z2f8h\"" Oct 30 17:04:20.560487 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.560393 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-661509\"/\"rate-test-model-661509-kserve-self-signed-certs\"" Oct 30 17:04:20.569675 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.569640 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9"] Oct 30 17:04:20.572075 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.572051 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-661509\"/\"openshift-service-ca.crt\"" Oct 30 17:04:20.572264 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.572098 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-661509\"/\"kube-root-ca.crt\"" Oct 30 17:04:20.593635 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.593590 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:20.664184 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.664140 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-model-cache\") pod \"rate-test-model-661509-kserve-56db4b6ccc-b22x9\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.664408 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.664217 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e660a917-1db1-4c82-b938-92f1ba909939-tls-certs\") pod \"rate-test-model-661509-kserve-56db4b6ccc-b22x9\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.664481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.664408 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-tmp-dir\") pod \"rate-test-model-661509-kserve-56db4b6ccc-b22x9\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.664481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.664456 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh7j5\" (UniqueName: \"kubernetes.io/projected/e660a917-1db1-4c82-b938-92f1ba909939-kube-api-access-rh7j5\") pod \"rate-test-model-661509-kserve-56db4b6ccc-b22x9\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.664619 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.664605 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-dshm\") pod \"rate-test-model-661509-kserve-56db4b6ccc-b22x9\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.664678 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.664640 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-home\") pod \"rate-test-model-661509-kserve-56db4b6ccc-b22x9\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.783844 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.781697 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-tmp-dir\") pod \"rate-test-model-661509-kserve-56db4b6ccc-b22x9\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.783844 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.781797 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh7j5\" (UniqueName: \"kubernetes.io/projected/e660a917-1db1-4c82-b938-92f1ba909939-kube-api-access-rh7j5\") pod \"rate-test-model-661509-kserve-56db4b6ccc-b22x9\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.783844 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.781931 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-dshm\") pod \"rate-test-model-661509-kserve-56db4b6ccc-b22x9\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.783844 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.781975 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-home\") pod \"rate-test-model-661509-kserve-56db4b6ccc-b22x9\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.783844 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.782025 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-model-cache\") pod \"rate-test-model-661509-kserve-56db4b6ccc-b22x9\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.783844 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.782066 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e660a917-1db1-4c82-b938-92f1ba909939-tls-certs\") pod \"rate-test-model-661509-kserve-56db4b6ccc-b22x9\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.785816 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.785022 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-home\") pod \"rate-test-model-661509-kserve-56db4b6ccc-b22x9\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.785816 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.785260 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-model-cache\") pod \"rate-test-model-661509-kserve-56db4b6ccc-b22x9\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.785816 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.785479 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-tmp-dir\") pod \"rate-test-model-661509-kserve-56db4b6ccc-b22x9\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.810789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.804686 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e660a917-1db1-4c82-b938-92f1ba909939-tls-certs\") pod \"rate-test-model-661509-kserve-56db4b6ccc-b22x9\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.819745 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.819711 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-dshm\") pod \"rate-test-model-661509-kserve-56db4b6ccc-b22x9\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.848643 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.848609 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh7j5\" (UniqueName: \"kubernetes.io/projected/e660a917-1db1-4c82-b938-92f1ba909939-kube-api-access-rh7j5\") pod \"rate-test-model-661509-kserve-56db4b6ccc-b22x9\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:20.867109 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:20.867073 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:21.178655 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:04:21.178609 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode660a917_1db1_4c82_b938_92f1ba909939.slice/crio-9fd5030ba2d40e972cd8229aaa010fb353c9ec7e688d189dc396ac7c8825f35e WatchSource:0}: Error finding container 9fd5030ba2d40e972cd8229aaa010fb353c9ec7e688d189dc396ac7c8825f35e: Status 404 returned error can't find the container with id 9fd5030ba2d40e972cd8229aaa010fb353c9ec7e688d189dc396ac7c8825f35e Oct 30 17:04:21.179133 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:21.178732 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9"] Oct 30 17:04:21.732314 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:21.732280 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6"] Oct 30 17:04:21.737179 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:21.737157 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6" Oct 30 17:04:21.763359 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:21.763309 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:04:21.763990 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:21.763971 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:04:21.764107 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:21.764004 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:04:21.764107 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:21.764024 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:04:21.764881 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:21.764860 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6"] Oct 30 17:04:21.796045 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:21.795995 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrfc9\" (UniqueName: \"kubernetes.io/projected/24dd6920-1fd0-4644-892c-5c3d04d09c14-kube-api-access-nrfc9\") pod \"maas-api-revoke-sub-580be196-rpdk6\" (UID: \"24dd6920-1fd0-4644-892c-5c3d04d09c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6" Oct 30 17:04:21.796247 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:21.796128 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24dd6920-1fd0-4644-892c-5c3d04d09c14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-580be196-rpdk6\" (UID: \"24dd6920-1fd0-4644-892c-5c3d04d09c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6" Oct 30 17:04:21.797056 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:21.797023 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24dd6920-1fd0-4644-892c-5c3d04d09c14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-580be196-rpdk6\" (UID: \"24dd6920-1fd0-4644-892c-5c3d04d09c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6" Oct 30 17:04:21.827186 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:21.827153 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:21.898828 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:21.898730 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrfc9\" (UniqueName: \"kubernetes.io/projected/24dd6920-1fd0-4644-892c-5c3d04d09c14-kube-api-access-nrfc9\") pod \"maas-api-revoke-sub-580be196-rpdk6\" (UID: \"24dd6920-1fd0-4644-892c-5c3d04d09c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6" Oct 30 17:04:21.899043 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:21.898830 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24dd6920-1fd0-4644-892c-5c3d04d09c14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-580be196-rpdk6\" (UID: \"24dd6920-1fd0-4644-892c-5c3d04d09c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6" Oct 30 17:04:21.899125 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:21.899078 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24dd6920-1fd0-4644-892c-5c3d04d09c14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-580be196-rpdk6\" (UID: \"24dd6920-1fd0-4644-892c-5c3d04d09c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6" Oct 30 17:04:21.899658 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:21.899632 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24dd6920-1fd0-4644-892c-5c3d04d09c14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-580be196-rpdk6\" (UID: \"24dd6920-1fd0-4644-892c-5c3d04d09c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6" Oct 30 17:04:21.902130 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:21.902099 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24dd6920-1fd0-4644-892c-5c3d04d09c14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-580be196-rpdk6\" (UID: \"24dd6920-1fd0-4644-892c-5c3d04d09c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6" Oct 30 17:04:21.924041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:21.924013 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrfc9\" (UniqueName: \"kubernetes.io/projected/24dd6920-1fd0-4644-892c-5c3d04d09c14-kube-api-access-nrfc9\") pod \"maas-api-revoke-sub-580be196-rpdk6\" (UID: \"24dd6920-1fd0-4644-892c-5c3d04d09c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6" Oct 30 17:04:22.037883 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:22.037785 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" event={"ID":"e660a917-1db1-4c82-b938-92f1ba909939","Type":"ContainerStarted","Data":"f5a405694d525cc93c58e26ce6639a50e8d777ebc71e04b7108256bacd01f99e"} Oct 30 17:04:22.037883 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:22.037838 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" event={"ID":"e660a917-1db1-4c82-b938-92f1ba909939","Type":"ContainerStarted","Data":"9fd5030ba2d40e972cd8229aaa010fb353c9ec7e688d189dc396ac7c8825f35e"} Oct 30 17:04:22.052564 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:22.052532 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6" Oct 30 17:04:22.092368 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:22.092315 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" podStartSLOduration=2.092298436 podStartE2EDuration="2.092298436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:22.086829027 +0000 UTC m=+2575.797283779" watchObservedRunningTime="2026-10-30 17:04:22.092298436 +0000 UTC m=+2575.802753186" Oct 30 17:04:22.372166 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:04:22.372129 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24dd6920_1fd0_4644_892c_5c3d04d09c14.slice/crio-e788dcbd3851718196a7b92a9722a94610bf341f2338c2d8c002a3317b28cd00 WatchSource:0}: Error finding container e788dcbd3851718196a7b92a9722a94610bf341f2338c2d8c002a3317b28cd00: Status 404 returned error can't find the container with id e788dcbd3851718196a7b92a9722a94610bf341f2338c2d8c002a3317b28cd00 Oct 30 17:04:22.373587 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:22.373557 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6"] Oct 30 17:04:22.867187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:22.867149 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:23.052386 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:23.052290 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6" event={"ID":"24dd6920-1fd0-4644-892c-5c3d04d09c14","Type":"ContainerStarted","Data":"ffe2cd96c4bd7c1c39410cb08cc3f44b67dadf697448f1e9e01e3181041054c9"} Oct 30 17:04:23.052558 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:23.052388 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6" event={"ID":"24dd6920-1fd0-4644-892c-5c3d04d09c14","Type":"ContainerStarted","Data":"e788dcbd3851718196a7b92a9722a94610bf341f2338c2d8c002a3317b28cd00"} Oct 30 17:04:23.099306 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:23.099250 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6" podStartSLOduration=2.099234342 podStartE2EDuration="2.099234342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:23.098565382 +0000 UTC m=+2576.809020144" watchObservedRunningTime="2026-10-30 17:04:23.099234342 +0000 UTC m=+2576.809689085" Oct 30 17:04:23.502110 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:23.502076 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:24.063833 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:24.063788 6202 generic.go:356] "Generic (PLEG): container finished" podID="24dd6920-1fd0-4644-892c-5c3d04d09c14" containerID="ffe2cd96c4bd7c1c39410cb08cc3f44b67dadf697448f1e9e01e3181041054c9" exitCode=0 Oct 30 17:04:24.064064 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:24.063912 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6" event={"ID":"24dd6920-1fd0-4644-892c-5c3d04d09c14","Type":"ContainerDied","Data":"ffe2cd96c4bd7c1c39410cb08cc3f44b67dadf697448f1e9e01e3181041054c9"} Oct 30 17:04:25.251710 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:25.251672 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:25.274457 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:25.274424 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6" Oct 30 17:04:25.368285 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:25.368125 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:25.381417 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:25.381385 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24dd6920-1fd0-4644-892c-5c3d04d09c14-kube-api-access-nrfc9\" (UniqueName: \"kubernetes.io/projected/24dd6920-1fd0-4644-892c-5c3d04d09c14-kube-api-access-nrfc9\") pod \"24dd6920-1fd0-4644-892c-5c3d04d09c14\" (UID: \"24dd6920-1fd0-4644-892c-5c3d04d09c14\") " Oct 30 17:04:25.381652 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:25.381459 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24dd6920-1fd0-4644-892c-5c3d04d09c14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24dd6920-1fd0-4644-892c-5c3d04d09c14-maas-api-openshift-ca\") pod \"24dd6920-1fd0-4644-892c-5c3d04d09c14\" (UID: \"24dd6920-1fd0-4644-892c-5c3d04d09c14\") " Oct 30 17:04:25.381652 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:25.381518 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24dd6920-1fd0-4644-892c-5c3d04d09c14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24dd6920-1fd0-4644-892c-5c3d04d09c14-maas-api-xks-ca\") pod \"24dd6920-1fd0-4644-892c-5c3d04d09c14\" (UID: \"24dd6920-1fd0-4644-892c-5c3d04d09c14\") " Oct 30 17:04:25.381909 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:25.381879 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24dd6920-1fd0-4644-892c-5c3d04d09c14-maas-api-openshift-ca" pod "24dd6920-1fd0-4644-892c-5c3d04d09c14" (UID: "24dd6920-1fd0-4644-892c-5c3d04d09c14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:25.382204 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:25.382186 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24dd6920-1fd0-4644-892c-5c3d04d09c14-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:25.383842 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:25.383817 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24dd6920-1fd0-4644-892c-5c3d04d09c14-maas-api-xks-ca" pod "24dd6920-1fd0-4644-892c-5c3d04d09c14" (UID: "24dd6920-1fd0-4644-892c-5c3d04d09c14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:04:25.384236 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:25.384196 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24dd6920-1fd0-4644-892c-5c3d04d09c14-kube-api-access-nrfc9" pod "24dd6920-1fd0-4644-892c-5c3d04d09c14" (UID: "24dd6920-1fd0-4644-892c-5c3d04d09c14"). InnerVolumeSpecName "kube-api-access-nrfc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:25.486250 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:25.486179 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrfc9\" (UniqueName: \"kubernetes.io/projected/24dd6920-1fd0-4644-892c-5c3d04d09c14-kube-api-access-nrfc9\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:25.500039 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:25.500009 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24dd6920-1fd0-4644-892c-5c3d04d09c14-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:26.083861 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:26.083830 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6" Oct 30 17:04:26.084061 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:26.083827 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6" event={"ID":"24dd6920-1fd0-4644-892c-5c3d04d09c14","Type":"ContainerDied","Data":"e788dcbd3851718196a7b92a9722a94610bf341f2338c2d8c002a3317b28cd00"} Oct 30 17:04:26.084061 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:26.083971 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e788dcbd3851718196a7b92a9722a94610bf341f2338c2d8c002a3317b28cd00" Oct 30 17:04:27.010334 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:27.010299 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:27.536728 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:27.536686 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw"] Oct 30 17:04:27.579497 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:27.579450 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:28.255386 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.255351 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:28.733279 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.733239 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:28.773945 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.773908 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t"] Oct 30 17:04:28.775288 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.775264 6202 cpu_manager.go:414] "containerMap: removing container" podUID="24dd6920-1fd0-4644-892c-5c3d04d09c14" containerName="revoke-keys" Oct 30 17:04:28.775373 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.775295 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24dd6920-1fd0-4644-892c-5c3d04d09c14" containerName="revoke-keys" Oct 30 17:04:28.775515 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.775500 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="24dd6920-1fd0-4644-892c-5c3d04d09c14" containerName="revoke-keys" Oct 30 17:04:28.780397 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.780373 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.795607 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.795585 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b16279-openshift-default-dockercfg-k9kvc\"" Oct 30 17:04:28.812004 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.811966 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t"] Oct 30 17:04:28.857408 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.857360 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-credential-socket\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.857716 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.857506 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-workload-socket\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.857716 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.857616 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-token\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.857716 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.857677 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-567xt\" (UniqueName: \"kubernetes.io/projected/cd39addc-aef3-41d3-aae4-6358ffbdf53e-kube-api-access-567xt\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.857716 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.857713 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istiod-ca-cert\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.858064 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.857760 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-podinfo\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.858064 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.857878 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-data\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.858064 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.857941 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-envoy\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.858064 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.857976 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-workload-certs\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.938840 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.938797 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:28.967833 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.967725 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-data\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.968207 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.968176 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-envoy\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.968381 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.968362 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-workload-certs\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.968543 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.968526 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-credential-socket\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.968742 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.968721 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-workload-socket\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.968971 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.968949 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-token\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.969146 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.969128 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-567xt\" (UniqueName: \"kubernetes.io/projected/cd39addc-aef3-41d3-aae4-6358ffbdf53e-kube-api-access-567xt\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.969300 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.969283 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istiod-ca-cert\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.969460 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.969444 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-podinfo\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.987851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.978917 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-credential-socket\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.988127 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.985550 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-data\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.991795 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.985789 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-workload-certs\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.992004 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.986325 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-workload-socket\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.992107 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.987356 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istiod-ca-cert\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:28.999202 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:28.999166 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-envoy\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:29.013958 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:29.013927 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-567xt\" (UniqueName: \"kubernetes.io/projected/cd39addc-aef3-41d3-aae4-6358ffbdf53e-kube-api-access-567xt\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:29.014695 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:29.014669 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-token\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:29.023531 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:29.022636 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-podinfo\") pod \"e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:29.096983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:29.095562 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:29.432324 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:04:29.432288 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd39addc_aef3_41d3_aae4_6358ffbdf53e.slice/crio-568c15a2165eab1089093886731459dff70484c1370a4c36312eb26d8a03e946 WatchSource:0}: Error finding container 568c15a2165eab1089093886731459dff70484c1370a4c36312eb26d8a03e946: Status 404 returned error can't find the container with id 568c15a2165eab1089093886731459dff70484c1370a4c36312eb26d8a03e946 Oct 30 17:04:29.432654 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:29.432515 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t"] Oct 30 17:04:29.435159 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:29.435126 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:04:29.435250 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:29.435190 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:04:29.435250 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:29.435219 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:04:30.131225 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:30.131188 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" event={"ID":"cd39addc-aef3-41d3-aae4-6358ffbdf53e","Type":"ContainerStarted","Data":"2c0a61cf7c8464598a3b322eade405e2b9d16d04dcef15d59d6977914c279a8b"} Oct 30 17:04:30.131456 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:30.131243 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" event={"ID":"cd39addc-aef3-41d3-aae4-6358ffbdf53e","Type":"ContainerStarted","Data":"568c15a2165eab1089093886731459dff70484c1370a4c36312eb26d8a03e946"} Oct 30 17:04:30.196587 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:30.196257 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" podStartSLOduration=2.196238268 podStartE2EDuration="2.196238268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:30.19395458 +0000 UTC m=+2583.904409324" watchObservedRunningTime="2026-10-30 17:04:30.196238268 +0000 UTC m=+2583.906693012" Oct 30 17:04:30.603994 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:30.603959 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:30.867641 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:30.867546 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:30.883925 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:30.883892 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:31.096235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:31.096201 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:31.103598 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:31.103563 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:31.140151 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:31.140004 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:31.141040 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:31.141014 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:04:31.152880 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:31.152838 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:04:32.089222 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.089173 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749"] Oct 30 17:04:32.094143 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.094117 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749" Oct 30 17:04:32.117325 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.117288 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749"] Oct 30 17:04:32.123564 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.123512 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:04:32.123564 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.123514 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:04:32.123803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.123602 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:04:32.123803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.123648 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:04:32.238254 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.238210 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvlf5\" (UniqueName: \"kubernetes.io/projected/53a30ace-ae20-4a45-a8a2-1ba3e0008486-kube-api-access-bvlf5\") pod \"maas-api-revoke-sub-3e33204e-zx749\" (UID: \"53a30ace-ae20-4a45-a8a2-1ba3e0008486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749" Oct 30 17:04:32.238503 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.238471 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53a30ace-ae20-4a45-a8a2-1ba3e0008486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e33204e-zx749\" (UID: \"53a30ace-ae20-4a45-a8a2-1ba3e0008486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749" Oct 30 17:04:32.238734 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.238706 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53a30ace-ae20-4a45-a8a2-1ba3e0008486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e33204e-zx749\" (UID: \"53a30ace-ae20-4a45-a8a2-1ba3e0008486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749" Oct 30 17:04:32.340541 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.340456 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvlf5\" (UniqueName: \"kubernetes.io/projected/53a30ace-ae20-4a45-a8a2-1ba3e0008486-kube-api-access-bvlf5\") pod \"maas-api-revoke-sub-3e33204e-zx749\" (UID: \"53a30ace-ae20-4a45-a8a2-1ba3e0008486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749" Oct 30 17:04:32.340731 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.340557 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53a30ace-ae20-4a45-a8a2-1ba3e0008486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e33204e-zx749\" (UID: \"53a30ace-ae20-4a45-a8a2-1ba3e0008486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749" Oct 30 17:04:32.340731 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.340621 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53a30ace-ae20-4a45-a8a2-1ba3e0008486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e33204e-zx749\" (UID: \"53a30ace-ae20-4a45-a8a2-1ba3e0008486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749" Oct 30 17:04:32.341316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.341292 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53a30ace-ae20-4a45-a8a2-1ba3e0008486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e33204e-zx749\" (UID: \"53a30ace-ae20-4a45-a8a2-1ba3e0008486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749" Oct 30 17:04:32.343292 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.343266 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53a30ace-ae20-4a45-a8a2-1ba3e0008486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e33204e-zx749\" (UID: \"53a30ace-ae20-4a45-a8a2-1ba3e0008486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749" Oct 30 17:04:32.364578 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.364552 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvlf5\" (UniqueName: \"kubernetes.io/projected/53a30ace-ae20-4a45-a8a2-1ba3e0008486-kube-api-access-bvlf5\") pod \"maas-api-revoke-sub-3e33204e-zx749\" (UID: \"53a30ace-ae20-4a45-a8a2-1ba3e0008486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749" Oct 30 17:04:32.415116 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.415085 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749" Oct 30 17:04:32.654200 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.654103 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:32.730200 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.730156 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749"] Oct 30 17:04:32.737967 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:04:32.737932 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a30ace_ae20_4a45_a8a2_1ba3e0008486.slice/crio-c9b2ea8f9cb3db4d470f1de602102499bdf375a8c73e83f3848e9c14fb3e7c12 WatchSource:0}: Error finding container c9b2ea8f9cb3db4d470f1de602102499bdf375a8c73e83f3848e9c14fb3e7c12: Status 404 returned error can't find the container with id c9b2ea8f9cb3db4d470f1de602102499bdf375a8c73e83f3848e9c14fb3e7c12 Oct 30 17:04:32.781846 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.781815 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:32.914019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:32.913930 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:33.033468 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:33.033432 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx"] Oct 30 17:04:33.039504 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:33.039474 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx" Oct 30 17:04:33.059642 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:33.059607 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx"] Oct 30 17:04:33.077814 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:33.077781 6202 scope.go:122] "RemoveContainer" containerID="4132baddde47a724fc3ece7d40a9e318dda0ec3dbcc511b8636d9aa0867521f3" Oct 30 17:04:33.153861 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:33.153823 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ded757e-4257-4ac4-9b06-15b34822ae38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0992b63f-dtdmx\" (UID: \"2ded757e-4257-4ac4-9b06-15b34822ae38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx" Oct 30 17:04:33.154422 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:33.153914 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ded757e-4257-4ac4-9b06-15b34822ae38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0992b63f-dtdmx\" (UID: \"2ded757e-4257-4ac4-9b06-15b34822ae38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx" Oct 30 17:04:33.154422 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:33.154175 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6kbl\" (UniqueName: \"kubernetes.io/projected/2ded757e-4257-4ac4-9b06-15b34822ae38-kube-api-access-d6kbl\") pod \"maas-api-revoke-sub-0992b63f-dtdmx\" (UID: \"2ded757e-4257-4ac4-9b06-15b34822ae38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx" Oct 30 17:04:33.169411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:33.169312 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749" event={"ID":"53a30ace-ae20-4a45-a8a2-1ba3e0008486","Type":"ContainerStarted","Data":"c9b2ea8f9cb3db4d470f1de602102499bdf375a8c73e83f3848e9c14fb3e7c12"} Oct 30 17:04:33.212212 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:33.212150 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749" podStartSLOduration=1.212134767 podStartE2EDuration="1.212134767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:33.208692423 +0000 UTC m=+2586.919147145" watchObservedRunningTime="2026-10-30 17:04:33.212134767 +0000 UTC m=+2586.922589510" Oct 30 17:04:33.256111 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:33.256072 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6kbl\" (UniqueName: \"kubernetes.io/projected/2ded757e-4257-4ac4-9b06-15b34822ae38-kube-api-access-d6kbl\") pod \"maas-api-revoke-sub-0992b63f-dtdmx\" (UID: \"2ded757e-4257-4ac4-9b06-15b34822ae38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx" Oct 30 17:04:33.256293 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:33.256215 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ded757e-4257-4ac4-9b06-15b34822ae38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0992b63f-dtdmx\" (UID: \"2ded757e-4257-4ac4-9b06-15b34822ae38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx" Oct 30 17:04:33.256492 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:33.256472 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ded757e-4257-4ac4-9b06-15b34822ae38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0992b63f-dtdmx\" (UID: \"2ded757e-4257-4ac4-9b06-15b34822ae38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx" Oct 30 17:04:33.257136 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:33.257107 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ded757e-4257-4ac4-9b06-15b34822ae38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0992b63f-dtdmx\" (UID: \"2ded757e-4257-4ac4-9b06-15b34822ae38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx" Oct 30 17:04:33.259098 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:33.259081 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ded757e-4257-4ac4-9b06-15b34822ae38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0992b63f-dtdmx\" (UID: \"2ded757e-4257-4ac4-9b06-15b34822ae38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx" Oct 30 17:04:33.279968 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:33.279939 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6kbl\" (UniqueName: \"kubernetes.io/projected/2ded757e-4257-4ac4-9b06-15b34822ae38-kube-api-access-d6kbl\") pod \"maas-api-revoke-sub-0992b63f-dtdmx\" (UID: \"2ded757e-4257-4ac4-9b06-15b34822ae38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx" Oct 30 17:04:33.374443 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:33.374407 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx" Oct 30 17:04:33.680757 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:33.680720 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx"] Oct 30 17:04:33.683214 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:04:33.683135 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ded757e_4257_4ac4_9b06_15b34822ae38.slice/crio-ac8755e79d4fccb4f05acfa86cc1093c95cd30891c933a02cb1f82d40264885f WatchSource:0}: Error finding container ac8755e79d4fccb4f05acfa86cc1093c95cd30891c933a02cb1f82d40264885f: Status 404 returned error can't find the container with id ac8755e79d4fccb4f05acfa86cc1093c95cd30891c933a02cb1f82d40264885f Oct 30 17:04:34.181105 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:34.181061 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx" event={"ID":"2ded757e-4257-4ac4-9b06-15b34822ae38","Type":"ContainerStarted","Data":"81197c09ff1c6d38b3b5d2211d7dce5a89939be1ca4572532f114bcce1ea9db7"} Oct 30 17:04:34.181723 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:34.181128 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx" event={"ID":"2ded757e-4257-4ac4-9b06-15b34822ae38","Type":"ContainerStarted","Data":"ac8755e79d4fccb4f05acfa86cc1093c95cd30891c933a02cb1f82d40264885f"} Oct 30 17:04:34.183502 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:34.183474 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749" event={"ID":"53a30ace-ae20-4a45-a8a2-1ba3e0008486","Type":"ContainerStarted","Data":"36e0c77db599f86dd23a18f17d3958812e1929e67bc9d457ec9f4ef9600beeb1"} Oct 30 17:04:34.237358 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:34.237250 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx" podStartSLOduration=1.237233032 podStartE2EDuration="1.237233032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:34.222671026 +0000 UTC m=+2587.933125771" watchObservedRunningTime="2026-10-30 17:04:34.237233032 +0000 UTC m=+2587.947687778" Oct 30 17:04:34.434554 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:34.434514 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:34.562311 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:34.562220 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:34.617399 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:34.617360 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98"] Oct 30 17:04:34.622099 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:34.622078 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98" Oct 30 17:04:34.636757 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:34.636729 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98"] Oct 30 17:04:34.697175 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:34.697132 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b16279-dockercfg-2hncv\"" Oct 30 17:04:34.707820 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:34.703456 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b16279\"" Oct 30 17:04:34.817367 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:34.817274 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7598e1-39ec-436a-a92e-371400e6580f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98\" (UID: \"3f7598e1-39ec-436a-a92e-371400e6580f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98" Oct 30 17:04:34.817558 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:34.817532 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs9p7\" (UniqueName: \"kubernetes.io/projected/3f7598e1-39ec-436a-a92e-371400e6580f-kube-api-access-bs9p7\") pod \"payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98\" (UID: \"3f7598e1-39ec-436a-a92e-371400e6580f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98" Oct 30 17:04:34.919198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:34.919157 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs9p7\" (UniqueName: \"kubernetes.io/projected/3f7598e1-39ec-436a-a92e-371400e6580f-kube-api-access-bs9p7\") pod \"payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98\" (UID: \"3f7598e1-39ec-436a-a92e-371400e6580f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98" Oct 30 17:04:34.919434 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:34.919415 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7598e1-39ec-436a-a92e-371400e6580f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98\" (UID: \"3f7598e1-39ec-436a-a92e-371400e6580f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98" Oct 30 17:04:34.920187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:34.920159 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7598e1-39ec-436a-a92e-371400e6580f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98\" (UID: \"3f7598e1-39ec-436a-a92e-371400e6580f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98" Oct 30 17:04:34.942863 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:34.942821 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs9p7\" (UniqueName: \"kubernetes.io/projected/3f7598e1-39ec-436a-a92e-371400e6580f-kube-api-access-bs9p7\") pod \"payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98\" (UID: \"3f7598e1-39ec-436a-a92e-371400e6580f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98" Oct 30 17:04:35.194058 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:35.194015 6202 generic.go:356] "Generic (PLEG): container finished" podID="2ded757e-4257-4ac4-9b06-15b34822ae38" containerID="81197c09ff1c6d38b3b5d2211d7dce5a89939be1ca4572532f114bcce1ea9db7" exitCode=0 Oct 30 17:04:35.194563 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:35.194055 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx" event={"ID":"2ded757e-4257-4ac4-9b06-15b34822ae38","Type":"ContainerDied","Data":"81197c09ff1c6d38b3b5d2211d7dce5a89939be1ca4572532f114bcce1ea9db7"} Oct 30 17:04:35.195931 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:35.195907 6202 generic.go:356] "Generic (PLEG): container finished" podID="53a30ace-ae20-4a45-a8a2-1ba3e0008486" containerID="36e0c77db599f86dd23a18f17d3958812e1929e67bc9d457ec9f4ef9600beeb1" exitCode=0 Oct 30 17:04:35.196067 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:35.195937 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749" event={"ID":"53a30ace-ae20-4a45-a8a2-1ba3e0008486","Type":"ContainerDied","Data":"36e0c77db599f86dd23a18f17d3958812e1929e67bc9d457ec9f4ef9600beeb1"} Oct 30 17:04:35.237139 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:35.237100 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98" Oct 30 17:04:35.559887 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:35.559852 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98"] Oct 30 17:04:35.573364 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:04:35.562641 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7598e1_39ec_436a_a92e_371400e6580f.slice/crio-89f24c2d463bcb1870cbd93a642d71a3cf28a311840c552d0ec50992dcbc0cc2 WatchSource:0}: Error finding container 89f24c2d463bcb1870cbd93a642d71a3cf28a311840c552d0ec50992dcbc0cc2: Status 404 returned error can't find the container with id 89f24c2d463bcb1870cbd93a642d71a3cf28a311840c552d0ec50992dcbc0cc2 Oct 30 17:04:36.200509 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.200473 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:36.206184 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.206149 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98" event={"ID":"3f7598e1-39ec-436a-a92e-371400e6580f","Type":"ContainerStarted","Data":"90fac49b2674ef30c862b79bc3b4f1e6cd7cae0fa75eb63ac3a3640111f9cf70"} Oct 30 17:04:36.206368 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.206203 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98" event={"ID":"3f7598e1-39ec-436a-a92e-371400e6580f","Type":"ContainerStarted","Data":"89f24c2d463bcb1870cbd93a642d71a3cf28a311840c552d0ec50992dcbc0cc2"} Oct 30 17:04:36.206472 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.206448 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98" Oct 30 17:04:36.280099 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.280024 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98" podStartSLOduration=2.280001681 podStartE2EDuration="2.280001681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:36.269901983 +0000 UTC m=+2589.980356729" watchObservedRunningTime="2026-10-30 17:04:36.280001681 +0000 UTC m=+2589.990456426" Oct 30 17:04:36.476344 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.476261 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749" Oct 30 17:04:36.516567 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.516540 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx" Oct 30 17:04:36.573784 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.573719 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ded757e-4257-4ac4-9b06-15b34822ae38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ded757e-4257-4ac4-9b06-15b34822ae38-maas-api-openshift-ca\") pod \"2ded757e-4257-4ac4-9b06-15b34822ae38\" (UID: \"2ded757e-4257-4ac4-9b06-15b34822ae38\") " Oct 30 17:04:36.573973 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.573950 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a30ace-ae20-4a45-a8a2-1ba3e0008486-kube-api-access-bvlf5\" (UniqueName: \"kubernetes.io/projected/53a30ace-ae20-4a45-a8a2-1ba3e0008486-kube-api-access-bvlf5\") pod \"53a30ace-ae20-4a45-a8a2-1ba3e0008486\" (UID: \"53a30ace-ae20-4a45-a8a2-1ba3e0008486\") " Oct 30 17:04:36.574088 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.574038 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53a30ace-ae20-4a45-a8a2-1ba3e0008486-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53a30ace-ae20-4a45-a8a2-1ba3e0008486-maas-api-openshift-ca\") pod \"53a30ace-ae20-4a45-a8a2-1ba3e0008486\" (UID: \"53a30ace-ae20-4a45-a8a2-1ba3e0008486\") " Oct 30 17:04:36.574159 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.574115 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53a30ace-ae20-4a45-a8a2-1ba3e0008486-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53a30ace-ae20-4a45-a8a2-1ba3e0008486-maas-api-xks-ca\") pod \"53a30ace-ae20-4a45-a8a2-1ba3e0008486\" (UID: \"53a30ace-ae20-4a45-a8a2-1ba3e0008486\") " Oct 30 17:04:36.574159 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.574137 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ded757e-4257-4ac4-9b06-15b34822ae38-maas-api-openshift-ca" pod "2ded757e-4257-4ac4-9b06-15b34822ae38" (UID: "2ded757e-4257-4ac4-9b06-15b34822ae38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:36.574277 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.574159 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ded757e-4257-4ac4-9b06-15b34822ae38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ded757e-4257-4ac4-9b06-15b34822ae38-maas-api-xks-ca\") pod \"2ded757e-4257-4ac4-9b06-15b34822ae38\" (UID: \"2ded757e-4257-4ac4-9b06-15b34822ae38\") " Oct 30 17:04:36.574332 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.574298 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ded757e-4257-4ac4-9b06-15b34822ae38-kube-api-access-d6kbl\" (UniqueName: \"kubernetes.io/projected/2ded757e-4257-4ac4-9b06-15b34822ae38-kube-api-access-d6kbl\") pod \"2ded757e-4257-4ac4-9b06-15b34822ae38\" (UID: \"2ded757e-4257-4ac4-9b06-15b34822ae38\") " Oct 30 17:04:36.575158 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.574982 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ded757e-4257-4ac4-9b06-15b34822ae38-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:36.575158 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.574977 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53a30ace-ae20-4a45-a8a2-1ba3e0008486-maas-api-openshift-ca" pod "53a30ace-ae20-4a45-a8a2-1ba3e0008486" (UID: "53a30ace-ae20-4a45-a8a2-1ba3e0008486"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:36.578226 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.578195 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ded757e-4257-4ac4-9b06-15b34822ae38-kube-api-access-d6kbl" pod "2ded757e-4257-4ac4-9b06-15b34822ae38" (UID: "2ded757e-4257-4ac4-9b06-15b34822ae38"). InnerVolumeSpecName "kube-api-access-d6kbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:36.583751 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.583700 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a30ace-ae20-4a45-a8a2-1ba3e0008486-kube-api-access-bvlf5" pod "53a30ace-ae20-4a45-a8a2-1ba3e0008486" (UID: "53a30ace-ae20-4a45-a8a2-1ba3e0008486"). InnerVolumeSpecName "kube-api-access-bvlf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:36.587114 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.585707 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ded757e-4257-4ac4-9b06-15b34822ae38-maas-api-xks-ca" pod "2ded757e-4257-4ac4-9b06-15b34822ae38" (UID: "2ded757e-4257-4ac4-9b06-15b34822ae38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:04:36.587260 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.587133 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53a30ace-ae20-4a45-a8a2-1ba3e0008486-maas-api-xks-ca" pod "53a30ace-ae20-4a45-a8a2-1ba3e0008486" (UID: "53a30ace-ae20-4a45-a8a2-1ba3e0008486"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:04:36.676959 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.676913 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvlf5\" (UniqueName: \"kubernetes.io/projected/53a30ace-ae20-4a45-a8a2-1ba3e0008486-kube-api-access-bvlf5\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:36.676959 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.676957 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53a30ace-ae20-4a45-a8a2-1ba3e0008486-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:36.677184 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.676974 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53a30ace-ae20-4a45-a8a2-1ba3e0008486-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:36.677184 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.676987 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ded757e-4257-4ac4-9b06-15b34822ae38-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:36.677184 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:36.677003 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6kbl\" (UniqueName: \"kubernetes.io/projected/2ded757e-4257-4ac4-9b06-15b34822ae38-kube-api-access-d6kbl\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:37.216035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:37.216004 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749" Oct 30 17:04:37.216550 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:37.216045 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749" event={"ID":"53a30ace-ae20-4a45-a8a2-1ba3e0008486","Type":"ContainerDied","Data":"c9b2ea8f9cb3db4d470f1de602102499bdf375a8c73e83f3848e9c14fb3e7c12"} Oct 30 17:04:37.216550 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:37.216100 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9b2ea8f9cb3db4d470f1de602102499bdf375a8c73e83f3848e9c14fb3e7c12" Oct 30 17:04:37.217708 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:37.217684 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx" Oct 30 17:04:37.217838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:37.217708 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx" event={"ID":"2ded757e-4257-4ac4-9b06-15b34822ae38","Type":"ContainerDied","Data":"ac8755e79d4fccb4f05acfa86cc1093c95cd30891c933a02cb1f82d40264885f"} Oct 30 17:04:37.217838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:37.217749 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac8755e79d4fccb4f05acfa86cc1093c95cd30891c933a02cb1f82d40264885f" Oct 30 17:04:37.901404 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:37.901366 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:38.158824 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.158703 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4"] Oct 30 17:04:38.159517 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.159500 6202 cpu_manager.go:414] "containerMap: removing container" podUID="53a30ace-ae20-4a45-a8a2-1ba3e0008486" containerName="revoke-keys" Oct 30 17:04:38.159561 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.159519 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a30ace-ae20-4a45-a8a2-1ba3e0008486" containerName="revoke-keys" Oct 30 17:04:38.159561 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.159529 6202 cpu_manager.go:414] "containerMap: removing container" podUID="2ded757e-4257-4ac4-9b06-15b34822ae38" containerName="revoke-keys" Oct 30 17:04:38.159561 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.159534 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ded757e-4257-4ac4-9b06-15b34822ae38" containerName="revoke-keys" Oct 30 17:04:38.159666 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.159657 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a30ace-ae20-4a45-a8a2-1ba3e0008486" containerName="revoke-keys" Oct 30 17:04:38.159666 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.159665 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ded757e-4257-4ac4-9b06-15b34822ae38" containerName="revoke-keys" Oct 30 17:04:38.164216 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.164191 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.199488 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.199456 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4"] Oct 30 17:04:38.201520 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.201489 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxpn2\" (UniqueName: \"kubernetes.io/projected/a227113a-3241-42d2-98d8-4c67c1ca8f16-kube-api-access-xxpn2\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.201712 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.201542 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.201889 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.201802 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.201956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.201892 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.201956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.201941 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.202065 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.201968 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a227113a-3241-42d2-98d8-4c67c1ca8f16-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.202065 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.202047 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.202193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.202075 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.202193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.202149 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.303452 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.303407 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.304019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.303497 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.304019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.303549 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.304019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.303578 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a227113a-3241-42d2-98d8-4c67c1ca8f16-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.304019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.303657 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.304019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.303686 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.304019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.303757 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.304019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.303828 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxpn2\" (UniqueName: \"kubernetes.io/projected/a227113a-3241-42d2-98d8-4c67c1ca8f16-kube-api-access-xxpn2\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.304019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.303865 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.304019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.304011 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.304481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.304111 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.304481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.304249 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.304481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.304304 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.304481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.304388 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a227113a-3241-42d2-98d8-4c67c1ca8f16-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.306158 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.306130 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.306298 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.306283 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.327005 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.326975 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.327176 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.327129 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxpn2\" (UniqueName: \"kubernetes.io/projected/a227113a-3241-42d2-98d8-4c67c1ca8f16-kube-api-access-xxpn2\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.481239 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.481133 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:38.801395 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:04:38.801362 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda227113a_3241_42d2_98d8_4c67c1ca8f16.slice/crio-d1a84471a87f20e289911c858abfd0a9b10ea4b24472164ac4c5e494de638f73 WatchSource:0}: Error finding container d1a84471a87f20e289911c858abfd0a9b10ea4b24472164ac4c5e494de638f73: Status 404 returned error can't find the container with id d1a84471a87f20e289911c858abfd0a9b10ea4b24472164ac4c5e494de638f73 Oct 30 17:04:38.802134 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.802109 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4"] Oct 30 17:04:38.804366 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.804327 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:04:38.804450 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.804400 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:04:38.804450 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:38.804431 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:04:39.032382 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.032340 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx"] Oct 30 17:04:39.036985 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.036946 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0fd799-xcvfx"] Oct 30 17:04:39.236867 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.236829 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" event={"ID":"a227113a-3241-42d2-98d8-4c67c1ca8f16","Type":"ContainerStarted","Data":"c41e712a6e5049a241eb41fe82f4c2e8804f6f65fa256b460a01568c9a7bdd88"} Oct 30 17:04:39.237046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.236897 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" event={"ID":"a227113a-3241-42d2-98d8-4c67c1ca8f16","Type":"ContainerStarted","Data":"d1a84471a87f20e289911c858abfd0a9b10ea4b24472164ac4c5e494de638f73"} Oct 30 17:04:39.282401 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.282351 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" podStartSLOduration=1.28233675 podStartE2EDuration="1.28233675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:39.280527299 +0000 UTC m=+2592.990982045" watchObservedRunningTime="2026-10-30 17:04:39.28233675 +0000 UTC m=+2592.992791493" Oct 30 17:04:39.482289 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.482249 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:39.484486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.484458 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 17:04:39.484617 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.484507 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" podUID="a227113a-3241-42d2-98d8-4c67c1ca8f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 17:04:39.686286 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.686244 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:39.704651 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.704595 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7"] Oct 30 17:04:39.716789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.711000 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.733723 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.733672 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-dc74d5-openshift-default-dockercfg-7dmmg\"" Oct 30 17:04:39.785750 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.784977 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7"] Oct 30 17:04:39.830863 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.830751 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-workload-certs\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.831083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.830960 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-workload-socket\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.831083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.831040 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-credential-socket\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.831083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.831060 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-data\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.831083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.831082 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-podinfo\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.831280 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.831139 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istiod-ca-cert\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.831280 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.831163 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhgwt\" (UniqueName: \"kubernetes.io/projected/d3e18c75-a227-4a6b-a998-3d3982ad16b2-kube-api-access-lhgwt\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.831280 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.831183 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-envoy\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.831280 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.831233 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-token\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.932919 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.932887 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-token\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.933110 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.932963 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-workload-certs\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.933110 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.933028 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-workload-socket\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.933110 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.933088 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-credential-socket\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.933110 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.933105 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-data\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.933330 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.933121 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-podinfo\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.933330 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.933172 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istiod-ca-cert\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.933330 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.933194 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhgwt\" (UniqueName: \"kubernetes.io/projected/d3e18c75-a227-4a6b-a998-3d3982ad16b2-kube-api-access-lhgwt\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.933330 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.933214 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-envoy\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.933551 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.933497 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-workload-certs\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.933843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.933802 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-data\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.933843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.933834 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-workload-socket\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.934124 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.934103 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-credential-socket\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.934444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.934422 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istiod-ca-cert\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.935760 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.935737 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-envoy\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.935877 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.935828 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-podinfo\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.971804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.971555 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhgwt\" (UniqueName: \"kubernetes.io/projected/d3e18c75-a227-4a6b-a998-3d3982ad16b2-kube-api-access-lhgwt\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:39.972559 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:39.972531 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-token\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:40.070794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:40.070064 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:40.403334 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:04:40.403304 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e18c75_a227_4a6b_a998_3d3982ad16b2.slice/crio-ce51480ce10f733420b4934e6bc76ce6d4e388395476a570a52822a16f3b48d9 WatchSource:0}: Error finding container ce51480ce10f733420b4934e6bc76ce6d4e388395476a570a52822a16f3b48d9: Status 404 returned error can't find the container with id ce51480ce10f733420b4934e6bc76ce6d4e388395476a570a52822a16f3b48d9 Oct 30 17:04:40.406645 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:40.406609 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:04:40.406745 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:40.406679 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:04:40.406745 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:40.406711 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:04:40.414035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:40.414005 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ab7d97a-1e00-4022-8828-25abdd86d43b" path="/var/lib/kubelet/pods/3ab7d97a-1e00-4022-8828-25abdd86d43b/volumes" Oct 30 17:04:40.414374 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:40.414359 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7"] Oct 30 17:04:40.482013 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:40.481981 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 17:04:40.482173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:40.482034 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" podUID="a227113a-3241-42d2-98d8-4c67c1ca8f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 17:04:41.278175 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:41.278126 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:41.297574 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:41.297524 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" event={"ID":"d3e18c75-a227-4a6b-a998-3d3982ad16b2","Type":"ContainerStarted","Data":"cf867d002a6d035c46d839f83ebc7ccf5e160de175fae6e6a25066bb2e35a05a"} Oct 30 17:04:41.297803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:41.297632 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" event={"ID":"d3e18c75-a227-4a6b-a998-3d3982ad16b2","Type":"ContainerStarted","Data":"ce51480ce10f733420b4934e6bc76ce6d4e388395476a570a52822a16f3b48d9"} Oct 30 17:04:41.383944 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:41.383884 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" podStartSLOduration=2.383862178 podStartE2EDuration="2.383862178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:41.369935988 +0000 UTC m=+2595.080390738" watchObservedRunningTime="2026-10-30 17:04:41.383862178 +0000 UTC m=+2595.094316989" Oct 30 17:04:41.495790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:41.485279 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 17:04:41.495790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:41.485353 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" podUID="a227113a-3241-42d2-98d8-4c67c1ca8f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 17:04:41.513343 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:41.513311 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:42.073322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:42.073286 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:42.096962 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:42.096919 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 30 17:04:42.097278 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:42.097244 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" podUID="d3e18c75-a227-4a6b-a998-3d3982ad16b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 30 17:04:42.486502 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:42.486390 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 17:04:42.487194 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:42.487158 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" podUID="a227113a-3241-42d2-98d8-4c67c1ca8f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 17:04:42.586102 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:42.586056 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" podUID="53372428-2070-44b7-89b4-5218dfd08ebf" containerName="main" containerID="cri-o://81bd7c5bdae6827ef0226b4771acebf931c4d2ebf3416a2225267b25f3a37231" gracePeriod=45 Oct 30 17:04:43.076815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.073349 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 30 17:04:43.076815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.073458 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" podUID="d3e18c75-a227-4a6b-a998-3d3982ad16b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 30 17:04:43.122022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.121931 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:04:43.136141 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.136108 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-tmp-dir\") pod \"53372428-2070-44b7-89b4-5218dfd08ebf\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " Oct 30 17:04:43.136357 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.136187 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-dshm\") pod \"53372428-2070-44b7-89b4-5218dfd08ebf\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " Oct 30 17:04:43.136357 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.136245 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-home\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-home\") pod \"53372428-2070-44b7-89b4-5218dfd08ebf\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " Oct 30 17:04:43.136488 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.136405 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53372428-2070-44b7-89b4-5218dfd08ebf-kube-api-access-6hsr9\" (UniqueName: \"kubernetes.io/projected/53372428-2070-44b7-89b4-5218dfd08ebf-kube-api-access-6hsr9\") pod \"53372428-2070-44b7-89b4-5218dfd08ebf\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " Oct 30 17:04:43.136488 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.136441 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-model-cache\") pod \"53372428-2070-44b7-89b4-5218dfd08ebf\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " Oct 30 17:04:43.136594 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.136574 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53372428-2070-44b7-89b4-5218dfd08ebf-tls-certs\" (UniqueName: \"kubernetes.io/secret/53372428-2070-44b7-89b4-5218dfd08ebf-tls-certs\") pod \"53372428-2070-44b7-89b4-5218dfd08ebf\" (UID: \"53372428-2070-44b7-89b4-5218dfd08ebf\") " Oct 30 17:04:43.137031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.136941 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-home" pod "53372428-2070-44b7-89b4-5218dfd08ebf" (UID: "53372428-2070-44b7-89b4-5218dfd08ebf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:04:43.137031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.136988 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-model-cache" pod "53372428-2070-44b7-89b4-5218dfd08ebf" (UID: "53372428-2070-44b7-89b4-5218dfd08ebf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:04:43.137251 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.137178 6202 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-home\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:43.137251 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.137198 6202 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-model-cache\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:43.137364 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.137343 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-tmp-dir" pod "53372428-2070-44b7-89b4-5218dfd08ebf" (UID: "53372428-2070-44b7-89b4-5218dfd08ebf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:04:43.167333 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.167268 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-dshm" pod "53372428-2070-44b7-89b4-5218dfd08ebf" (UID: "53372428-2070-44b7-89b4-5218dfd08ebf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:04:43.168128 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.168083 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53372428-2070-44b7-89b4-5218dfd08ebf-tls-certs" pod "53372428-2070-44b7-89b4-5218dfd08ebf" (UID: "53372428-2070-44b7-89b4-5218dfd08ebf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:04:43.185029 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.184967 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53372428-2070-44b7-89b4-5218dfd08ebf-kube-api-access-6hsr9" pod "53372428-2070-44b7-89b4-5218dfd08ebf" (UID: "53372428-2070-44b7-89b4-5218dfd08ebf"). InnerVolumeSpecName "kube-api-access-6hsr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:43.270806 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.256104 6202 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-dshm\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:43.290341 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.290307 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hsr9\" (UniqueName: \"kubernetes.io/projected/53372428-2070-44b7-89b4-5218dfd08ebf-kube-api-access-6hsr9\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:43.294797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.290587 6202 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53372428-2070-44b7-89b4-5218dfd08ebf-tls-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:43.298801 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.295036 6202 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53372428-2070-44b7-89b4-5218dfd08ebf-tmp-dir\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:43.328815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.327674 6202 generic.go:356] "Generic (PLEG): container finished" podID="53372428-2070-44b7-89b4-5218dfd08ebf" containerID="81bd7c5bdae6827ef0226b4771acebf931c4d2ebf3416a2225267b25f3a37231" exitCode=0 Oct 30 17:04:43.328815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.328305 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" Oct 30 17:04:43.328815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.328756 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" event={"ID":"53372428-2070-44b7-89b4-5218dfd08ebf","Type":"ContainerDied","Data":"81bd7c5bdae6827ef0226b4771acebf931c4d2ebf3416a2225267b25f3a37231"} Oct 30 17:04:43.329148 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.328825 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw" event={"ID":"53372428-2070-44b7-89b4-5218dfd08ebf","Type":"ContainerDied","Data":"c3041ec130b1c09e81fd3dd4514202fcf19d2b142fcd15da305864c639e900ee"} Oct 30 17:04:43.329148 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.328848 6202 scope.go:122] "RemoveContainer" containerID="81bd7c5bdae6827ef0226b4771acebf931c4d2ebf3416a2225267b25f3a37231" Oct 30 17:04:43.399438 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.399338 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:43.407346 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.407318 6202 scope.go:122] "RemoveContainer" containerID="81bd7c5bdae6827ef0226b4771acebf931c4d2ebf3416a2225267b25f3a37231" Oct 30 17:04:43.407736 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:04:43.407713 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81bd7c5bdae6827ef0226b4771acebf931c4d2ebf3416a2225267b25f3a37231\": container with ID starting with 81bd7c5bdae6827ef0226b4771acebf931c4d2ebf3416a2225267b25f3a37231 not found: ID does not exist" containerID="81bd7c5bdae6827ef0226b4771acebf931c4d2ebf3416a2225267b25f3a37231" Oct 30 17:04:43.407813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.407747 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81bd7c5bdae6827ef0226b4771acebf931c4d2ebf3416a2225267b25f3a37231"} err="failed to get container status \"81bd7c5bdae6827ef0226b4771acebf931c4d2ebf3416a2225267b25f3a37231\": rpc error: code = NotFound desc = could not find container \"81bd7c5bdae6827ef0226b4771acebf931c4d2ebf3416a2225267b25f3a37231\": container with ID starting with 81bd7c5bdae6827ef0226b4771acebf931c4d2ebf3416a2225267b25f3a37231 not found: ID does not exist" Oct 30 17:04:43.436968 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.436932 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw"] Oct 30 17:04:43.499220 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.485069 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 17:04:43.499220 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.485126 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" podUID="a227113a-3241-42d2-98d8-4c67c1ca8f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 17:04:43.556941 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.556851 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-1b7bb3/auth-test-model-1b7bb3-kserve-54d7956885-dq4mw"] Oct 30 17:04:43.618806 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.615028 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht"] Oct 30 17:04:43.618806 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.616324 6202 cpu_manager.go:414] "containerMap: removing container" podUID="53372428-2070-44b7-89b4-5218dfd08ebf" containerName="main" Oct 30 17:04:43.618806 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.616348 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53372428-2070-44b7-89b4-5218dfd08ebf" containerName="main" Oct 30 17:04:43.618806 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.616518 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="53372428-2070-44b7-89b4-5218dfd08ebf" containerName="main" Oct 30 17:04:43.623804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.623780 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht"] Oct 30 17:04:43.624000 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.623878 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht" Oct 30 17:04:43.676849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.665277 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:04:43.676849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.665467 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:04:43.676849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.672151 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:04:43.676849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.672220 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:04:43.716377 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.716342 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf16d38-2b07-4378-bc4e-4a91a978506f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc69f624-7l8ht\" (UID: \"eaf16d38-2b07-4378-bc4e-4a91a978506f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht" Oct 30 17:04:43.716597 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.716414 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf16d38-2b07-4378-bc4e-4a91a978506f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc69f624-7l8ht\" (UID: \"eaf16d38-2b07-4378-bc4e-4a91a978506f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht" Oct 30 17:04:43.716702 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.716647 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98d7m\" (UniqueName: \"kubernetes.io/projected/eaf16d38-2b07-4378-bc4e-4a91a978506f-kube-api-access-98d7m\") pod \"maas-api-revoke-sub-cc69f624-7l8ht\" (UID: \"eaf16d38-2b07-4378-bc4e-4a91a978506f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht" Oct 30 17:04:43.818218 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.818173 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf16d38-2b07-4378-bc4e-4a91a978506f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc69f624-7l8ht\" (UID: \"eaf16d38-2b07-4378-bc4e-4a91a978506f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht" Oct 30 17:04:43.818459 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.818286 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf16d38-2b07-4378-bc4e-4a91a978506f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc69f624-7l8ht\" (UID: \"eaf16d38-2b07-4378-bc4e-4a91a978506f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht" Oct 30 17:04:43.818459 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.818381 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98d7m\" (UniqueName: \"kubernetes.io/projected/eaf16d38-2b07-4378-bc4e-4a91a978506f-kube-api-access-98d7m\") pod \"maas-api-revoke-sub-cc69f624-7l8ht\" (UID: \"eaf16d38-2b07-4378-bc4e-4a91a978506f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht" Oct 30 17:04:43.819019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.818994 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf16d38-2b07-4378-bc4e-4a91a978506f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc69f624-7l8ht\" (UID: \"eaf16d38-2b07-4378-bc4e-4a91a978506f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht" Oct 30 17:04:43.820978 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.820956 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf16d38-2b07-4378-bc4e-4a91a978506f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc69f624-7l8ht\" (UID: \"eaf16d38-2b07-4378-bc4e-4a91a978506f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht" Oct 30 17:04:43.861791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:43.849872 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98d7m\" (UniqueName: \"kubernetes.io/projected/eaf16d38-2b07-4378-bc4e-4a91a978506f-kube-api-access-98d7m\") pod \"maas-api-revoke-sub-cc69f624-7l8ht\" (UID: \"eaf16d38-2b07-4378-bc4e-4a91a978506f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht" Oct 30 17:04:44.006256 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:44.006146 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht" Oct 30 17:04:44.072797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:44.071302 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 30 17:04:44.072797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:44.071377 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" podUID="d3e18c75-a227-4a6b-a998-3d3982ad16b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 30 17:04:44.228017 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:04:44.208796 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:04:44.422714 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:44.422677 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53372428-2070-44b7-89b4-5218dfd08ebf" path="/var/lib/kubelet/pods/53372428-2070-44b7-89b4-5218dfd08ebf/volumes" Oct 30 17:04:44.423077 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:44.423057 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht"] Oct 30 17:04:44.502811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:44.483284 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 17:04:44.502811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:44.483366 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" podUID="a227113a-3241-42d2-98d8-4c67c1ca8f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 17:04:45.075538 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:45.075496 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 30 17:04:45.076255 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:45.075844 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" podUID="d3e18c75-a227-4a6b-a998-3d3982ad16b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 30 17:04:45.328696 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:45.328583 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:45.480789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:45.480687 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht" event={"ID":"eaf16d38-2b07-4378-bc4e-4a91a978506f","Type":"ContainerStarted","Data":"923e6fdbcf9aa00d7b67a06dd8d2940d25936bcf9dc3e604da014314c925b7c7"} Oct 30 17:04:45.480789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:45.480790 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht" event={"ID":"eaf16d38-2b07-4378-bc4e-4a91a978506f","Type":"ContainerStarted","Data":"2d6d0e4d449f429e9ef8053f8a473821272b15ad7d3495eda6e5cd99f0103b0b"} Oct 30 17:04:45.483003 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:45.482968 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 17:04:45.483152 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:45.483040 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" podUID="a227113a-3241-42d2-98d8-4c67c1ca8f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 17:04:45.594850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:45.593006 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:46.079793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:46.078908 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 30 17:04:46.079793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:46.078992 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" podUID="d3e18c75-a227-4a6b-a998-3d3982ad16b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 30 17:04:46.488197 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:46.488124 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:46.489318 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:46.489301 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:46.490155 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:46.490136 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:04:46.494748 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:46.494722 6202 generic.go:356] "Generic (PLEG): container finished" podID="eaf16d38-2b07-4378-bc4e-4a91a978506f" containerID="923e6fdbcf9aa00d7b67a06dd8d2940d25936bcf9dc3e604da014314c925b7c7" exitCode=0 Oct 30 17:04:46.494870 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:46.494802 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht" event={"ID":"eaf16d38-2b07-4378-bc4e-4a91a978506f","Type":"ContainerDied","Data":"923e6fdbcf9aa00d7b67a06dd8d2940d25936bcf9dc3e604da014314c925b7c7"} Oct 30 17:04:46.558794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:46.558324 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht" podStartSLOduration=3.558305001 podStartE2EDuration="3.558305001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:45.535201527 +0000 UTC m=+2599.245656271" watchObservedRunningTime="2026-10-30 17:04:46.558305001 +0000 UTC m=+2600.268759741" Oct 30 17:04:47.071574 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:47.071532 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 30 17:04:47.072045 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:47.071603 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" podUID="d3e18c75-a227-4a6b-a998-3d3982ad16b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 30 17:04:47.198079 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:47.198041 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:47.220834 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:47.220806 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98" Oct 30 17:04:47.744651 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:47.744619 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht" Oct 30 17:04:47.814614 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:47.814531 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf16d38-2b07-4378-bc4e-4a91a978506f-kube-api-access-98d7m\" (UniqueName: \"kubernetes.io/projected/eaf16d38-2b07-4378-bc4e-4a91a978506f-kube-api-access-98d7m\") pod \"eaf16d38-2b07-4378-bc4e-4a91a978506f\" (UID: \"eaf16d38-2b07-4378-bc4e-4a91a978506f\") " Oct 30 17:04:47.814822 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:47.814620 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaf16d38-2b07-4378-bc4e-4a91a978506f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf16d38-2b07-4378-bc4e-4a91a978506f-maas-api-xks-ca\") pod \"eaf16d38-2b07-4378-bc4e-4a91a978506f\" (UID: \"eaf16d38-2b07-4378-bc4e-4a91a978506f\") " Oct 30 17:04:47.814822 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:47.814732 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf16d38-2b07-4378-bc4e-4a91a978506f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf16d38-2b07-4378-bc4e-4a91a978506f-maas-api-openshift-ca\") pod \"eaf16d38-2b07-4378-bc4e-4a91a978506f\" (UID: \"eaf16d38-2b07-4378-bc4e-4a91a978506f\") " Oct 30 17:04:47.815196 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:47.815161 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf16d38-2b07-4378-bc4e-4a91a978506f-maas-api-openshift-ca" pod "eaf16d38-2b07-4378-bc4e-4a91a978506f" (UID: "eaf16d38-2b07-4378-bc4e-4a91a978506f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:47.815551 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:47.815525 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf16d38-2b07-4378-bc4e-4a91a978506f-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:47.817529 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:47.817500 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf16d38-2b07-4378-bc4e-4a91a978506f-maas-api-xks-ca" pod "eaf16d38-2b07-4378-bc4e-4a91a978506f" (UID: "eaf16d38-2b07-4378-bc4e-4a91a978506f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:04:47.817669 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:47.817546 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf16d38-2b07-4378-bc4e-4a91a978506f-kube-api-access-98d7m" pod "eaf16d38-2b07-4378-bc4e-4a91a978506f" (UID: "eaf16d38-2b07-4378-bc4e-4a91a978506f"). InnerVolumeSpecName "kube-api-access-98d7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:47.916890 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:47.916856 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98d7m\" (UniqueName: \"kubernetes.io/projected/eaf16d38-2b07-4378-bc4e-4a91a978506f-kube-api-access-98d7m\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:47.916890 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:47.916884 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf16d38-2b07-4378-bc4e-4a91a978506f-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:48.080339 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:48.080302 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:48.081424 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:48.081361 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:48.082002 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:48.081976 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:04:48.521258 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:48.521227 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht" Oct 30 17:04:48.521463 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:48.521261 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht" event={"ID":"eaf16d38-2b07-4378-bc4e-4a91a978506f","Type":"ContainerDied","Data":"2d6d0e4d449f429e9ef8053f8a473821272b15ad7d3495eda6e5cd99f0103b0b"} Oct 30 17:04:48.521463 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:48.521325 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d6d0e4d449f429e9ef8053f8a473821272b15ad7d3495eda6e5cd99f0103b0b" Oct 30 17:04:49.173952 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:49.173711 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:50.032023 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:50.031992 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh"] Oct 30 17:04:50.045976 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:50.045939 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c19327a7-xwbwh"] Oct 30 17:04:50.414747 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:50.414704 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e87fa350-f777-4857-8d44-7d19efbbd689" path="/var/lib/kubelet/pods/e87fa350-f777-4857-8d44-7d19efbbd689/volumes" Oct 30 17:04:50.970712 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:50.970665 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:52.804114 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:52.804075 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:54.383596 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.383555 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4"] Oct 30 17:04:54.384802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.384760 6202 cpu_manager.go:414] "containerMap: removing container" podUID="eaf16d38-2b07-4378-bc4e-4a91a978506f" containerName="revoke-keys" Oct 30 17:04:54.384868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.384805 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf16d38-2b07-4378-bc4e-4a91a978506f" containerName="revoke-keys" Oct 30 17:04:54.385051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.385038 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf16d38-2b07-4378-bc4e-4a91a978506f" containerName="revoke-keys" Oct 30 17:04:54.389962 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.389940 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4" Oct 30 17:04:54.412984 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.412957 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:04:54.412984 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.412983 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:04:54.415351 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.415321 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4"] Oct 30 17:04:54.416332 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.416315 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:04:54.416419 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.416386 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:04:54.542847 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.542811 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551943de-3741-4bb0-82d3-5bb8cd23a8ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba4aaf11-r2zl4\" (UID: \"551943de-3741-4bb0-82d3-5bb8cd23a8ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4" Oct 30 17:04:54.543049 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.542961 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551943de-3741-4bb0-82d3-5bb8cd23a8ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba4aaf11-r2zl4\" (UID: \"551943de-3741-4bb0-82d3-5bb8cd23a8ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4" Oct 30 17:04:54.543126 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.543050 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5qzj\" (UniqueName: \"kubernetes.io/projected/551943de-3741-4bb0-82d3-5bb8cd23a8ef-kube-api-access-q5qzj\") pod \"maas-api-revoke-sub-ba4aaf11-r2zl4\" (UID: \"551943de-3741-4bb0-82d3-5bb8cd23a8ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4" Oct 30 17:04:54.644729 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.644639 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551943de-3741-4bb0-82d3-5bb8cd23a8ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba4aaf11-r2zl4\" (UID: \"551943de-3741-4bb0-82d3-5bb8cd23a8ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4" Oct 30 17:04:54.644729 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.644690 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5qzj\" (UniqueName: \"kubernetes.io/projected/551943de-3741-4bb0-82d3-5bb8cd23a8ef-kube-api-access-q5qzj\") pod \"maas-api-revoke-sub-ba4aaf11-r2zl4\" (UID: \"551943de-3741-4bb0-82d3-5bb8cd23a8ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4" Oct 30 17:04:54.645053 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.645030 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551943de-3741-4bb0-82d3-5bb8cd23a8ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba4aaf11-r2zl4\" (UID: \"551943de-3741-4bb0-82d3-5bb8cd23a8ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4" Oct 30 17:04:54.645810 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.645787 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551943de-3741-4bb0-82d3-5bb8cd23a8ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba4aaf11-r2zl4\" (UID: \"551943de-3741-4bb0-82d3-5bb8cd23a8ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4" Oct 30 17:04:54.647487 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.647464 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551943de-3741-4bb0-82d3-5bb8cd23a8ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba4aaf11-r2zl4\" (UID: \"551943de-3741-4bb0-82d3-5bb8cd23a8ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4" Oct 30 17:04:54.668230 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.668197 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5qzj\" (UniqueName: \"kubernetes.io/projected/551943de-3741-4bb0-82d3-5bb8cd23a8ef-kube-api-access-q5qzj\") pod \"maas-api-revoke-sub-ba4aaf11-r2zl4\" (UID: \"551943de-3741-4bb0-82d3-5bb8cd23a8ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4" Oct 30 17:04:54.690730 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.690692 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:54.705308 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:54.705271 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4" Oct 30 17:04:55.047827 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:55.047795 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4"] Oct 30 17:04:55.048933 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:04:55.048904 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551943de_3741_4bb0_82d3_5bb8cd23a8ef.slice/crio-0cfb95794d57febb14b5527f58df3c219ff5c9140b8f37f5e9dfda9110f63644 WatchSource:0}: Error finding container 0cfb95794d57febb14b5527f58df3c219ff5c9140b8f37f5e9dfda9110f63644: Status 404 returned error can't find the container with id 0cfb95794d57febb14b5527f58df3c219ff5c9140b8f37f5e9dfda9110f63644 Oct 30 17:04:55.591604 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:55.591568 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4" event={"ID":"551943de-3741-4bb0-82d3-5bb8cd23a8ef","Type":"ContainerStarted","Data":"e6df92ad4e31a156c30f6137263ef65f0d0093cc616ee831f61371c177ca4afb"} Oct 30 17:04:55.592092 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:55.591625 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4" event={"ID":"551943de-3741-4bb0-82d3-5bb8cd23a8ef","Type":"ContainerStarted","Data":"0cfb95794d57febb14b5527f58df3c219ff5c9140b8f37f5e9dfda9110f63644"} Oct 30 17:04:55.635567 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:55.635468 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4" podStartSLOduration=1.635451542 podStartE2EDuration="1.635451542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:55.632206552 +0000 UTC m=+2609.342661297" watchObservedRunningTime="2026-10-30 17:04:55.635451542 +0000 UTC m=+2609.345906288" Oct 30 17:04:56.429252 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:56.429220 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:56.601984 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:56.601944 6202 generic.go:356] "Generic (PLEG): container finished" podID="551943de-3741-4bb0-82d3-5bb8cd23a8ef" containerID="e6df92ad4e31a156c30f6137263ef65f0d0093cc616ee831f61371c177ca4afb" exitCode=0 Oct 30 17:04:56.602365 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:56.602027 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4" event={"ID":"551943de-3741-4bb0-82d3-5bb8cd23a8ef","Type":"ContainerDied","Data":"e6df92ad4e31a156c30f6137263ef65f0d0093cc616ee831f61371c177ca4afb"} Oct 30 17:04:57.797378 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:57.797351 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4" Oct 30 17:04:57.902519 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:57.902430 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551943de-3741-4bb0-82d3-5bb8cd23a8ef-kube-api-access-q5qzj\" (UniqueName: \"kubernetes.io/projected/551943de-3741-4bb0-82d3-5bb8cd23a8ef-kube-api-access-q5qzj\") pod \"551943de-3741-4bb0-82d3-5bb8cd23a8ef\" (UID: \"551943de-3741-4bb0-82d3-5bb8cd23a8ef\") " Oct 30 17:04:57.902519 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:57.902490 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/551943de-3741-4bb0-82d3-5bb8cd23a8ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551943de-3741-4bb0-82d3-5bb8cd23a8ef-maas-api-xks-ca\") pod \"551943de-3741-4bb0-82d3-5bb8cd23a8ef\" (UID: \"551943de-3741-4bb0-82d3-5bb8cd23a8ef\") " Oct 30 17:04:57.902796 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:57.902611 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551943de-3741-4bb0-82d3-5bb8cd23a8ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551943de-3741-4bb0-82d3-5bb8cd23a8ef-maas-api-openshift-ca\") pod \"551943de-3741-4bb0-82d3-5bb8cd23a8ef\" (UID: \"551943de-3741-4bb0-82d3-5bb8cd23a8ef\") " Oct 30 17:04:57.903160 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:57.903093 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551943de-3741-4bb0-82d3-5bb8cd23a8ef-maas-api-openshift-ca" pod "551943de-3741-4bb0-82d3-5bb8cd23a8ef" (UID: "551943de-3741-4bb0-82d3-5bb8cd23a8ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:57.903492 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:57.903462 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551943de-3741-4bb0-82d3-5bb8cd23a8ef-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:57.905453 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:57.905425 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/551943de-3741-4bb0-82d3-5bb8cd23a8ef-maas-api-xks-ca" pod "551943de-3741-4bb0-82d3-5bb8cd23a8ef" (UID: "551943de-3741-4bb0-82d3-5bb8cd23a8ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:04:57.905453 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:57.905439 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551943de-3741-4bb0-82d3-5bb8cd23a8ef-kube-api-access-q5qzj" pod "551943de-3741-4bb0-82d3-5bb8cd23a8ef" (UID: "551943de-3741-4bb0-82d3-5bb8cd23a8ef"). InnerVolumeSpecName "kube-api-access-q5qzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:58.005085 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:58.005045 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5qzj\" (UniqueName: \"kubernetes.io/projected/551943de-3741-4bb0-82d3-5bb8cd23a8ef-kube-api-access-q5qzj\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:58.005085 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:58.005077 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551943de-3741-4bb0-82d3-5bb8cd23a8ef-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:04:58.204494 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:58.204407 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:58.619984 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:58.619950 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4" Oct 30 17:04:58.620200 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:58.619980 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4" event={"ID":"551943de-3741-4bb0-82d3-5bb8cd23a8ef","Type":"ContainerDied","Data":"0cfb95794d57febb14b5527f58df3c219ff5c9140b8f37f5e9dfda9110f63644"} Oct 30 17:04:58.620200 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:04:58.620128 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cfb95794d57febb14b5527f58df3c219ff5c9140b8f37f5e9dfda9110f63644" Oct 30 17:05:00.107378 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:00.107345 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:01.557392 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.557353 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc"] Oct 30 17:05:01.561045 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.561011 6202 cpu_manager.go:414] "containerMap: removing container" podUID="551943de-3741-4bb0-82d3-5bb8cd23a8ef" containerName="revoke-keys" Oct 30 17:05:01.561045 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.561044 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551943de-3741-4bb0-82d3-5bb8cd23a8ef" containerName="revoke-keys" Oct 30 17:05:01.561964 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.561936 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="551943de-3741-4bb0-82d3-5bb8cd23a8ef" containerName="revoke-keys" Oct 30 17:05:01.574845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.574815 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc" Oct 30 17:05:01.585066 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.585034 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc"] Oct 30 17:05:01.592813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.590912 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:05:01.602363 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.602149 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:05:01.602363 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.602183 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:05:01.602363 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.602206 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:05:01.665227 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.665192 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt262\" (UniqueName: \"kubernetes.io/projected/28dd80f8-8d21-4d64-9327-104671fbbe0d-kube-api-access-vt262\") pod \"maas-api-revoke-sub-805ee993-vq8xc\" (UID: \"28dd80f8-8d21-4d64-9327-104671fbbe0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc" Oct 30 17:05:01.665415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.665344 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28dd80f8-8d21-4d64-9327-104671fbbe0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805ee993-vq8xc\" (UID: \"28dd80f8-8d21-4d64-9327-104671fbbe0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc" Oct 30 17:05:01.665472 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.665427 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28dd80f8-8d21-4d64-9327-104671fbbe0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805ee993-vq8xc\" (UID: \"28dd80f8-8d21-4d64-9327-104671fbbe0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc" Oct 30 17:05:01.767334 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.767288 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28dd80f8-8d21-4d64-9327-104671fbbe0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805ee993-vq8xc\" (UID: \"28dd80f8-8d21-4d64-9327-104671fbbe0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc" Oct 30 17:05:01.767578 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.767562 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt262\" (UniqueName: \"kubernetes.io/projected/28dd80f8-8d21-4d64-9327-104671fbbe0d-kube-api-access-vt262\") pod \"maas-api-revoke-sub-805ee993-vq8xc\" (UID: \"28dd80f8-8d21-4d64-9327-104671fbbe0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc" Oct 30 17:05:01.767714 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.767588 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:01.767714 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.767693 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28dd80f8-8d21-4d64-9327-104671fbbe0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805ee993-vq8xc\" (UID: \"28dd80f8-8d21-4d64-9327-104671fbbe0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc" Oct 30 17:05:01.768135 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.768115 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28dd80f8-8d21-4d64-9327-104671fbbe0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805ee993-vq8xc\" (UID: \"28dd80f8-8d21-4d64-9327-104671fbbe0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc" Oct 30 17:05:01.770338 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.770307 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28dd80f8-8d21-4d64-9327-104671fbbe0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805ee993-vq8xc\" (UID: \"28dd80f8-8d21-4d64-9327-104671fbbe0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc" Oct 30 17:05:01.791105 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.791079 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt262\" (UniqueName: \"kubernetes.io/projected/28dd80f8-8d21-4d64-9327-104671fbbe0d-kube-api-access-vt262\") pod \"maas-api-revoke-sub-805ee993-vq8xc\" (UID: \"28dd80f8-8d21-4d64-9327-104671fbbe0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc" Oct 30 17:05:01.897411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:01.897378 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc" Oct 30 17:05:02.206294 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:05:02.206268 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28dd80f8_8d21_4d64_9327_104671fbbe0d.slice/crio-6467e6b70a486994be19a827256063594dc2e7b6bc9aba4929cca1e1b23c93e3 WatchSource:0}: Error finding container 6467e6b70a486994be19a827256063594dc2e7b6bc9aba4929cca1e1b23c93e3: Status 404 returned error can't find the container with id 6467e6b70a486994be19a827256063594dc2e7b6bc9aba4929cca1e1b23c93e3 Oct 30 17:05:02.208028 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:02.207998 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc"] Oct 30 17:05:02.656268 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:02.656229 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc" event={"ID":"28dd80f8-8d21-4d64-9327-104671fbbe0d","Type":"ContainerStarted","Data":"6467e6b70a486994be19a827256063594dc2e7b6bc9aba4929cca1e1b23c93e3"} Oct 30 17:05:03.533519 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:03.533442 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:03.667746 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:03.667710 6202 generic.go:356] "Generic (PLEG): container finished" podID="28dd80f8-8d21-4d64-9327-104671fbbe0d" containerID="d6e1aff3172193db7cdeb855f86cdfb66a10f205b34a967a0434f8b475cc823f" exitCode=0 Oct 30 17:05:03.668310 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:03.667799 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc" event={"ID":"28dd80f8-8d21-4d64-9327-104671fbbe0d","Type":"ContainerDied","Data":"d6e1aff3172193db7cdeb855f86cdfb66a10f205b34a967a0434f8b475cc823f"} Oct 30 17:05:04.856818 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:04.856785 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc" Oct 30 17:05:04.925508 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:04.925469 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28dd80f8-8d21-4d64-9327-104671fbbe0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28dd80f8-8d21-4d64-9327-104671fbbe0d-maas-api-openshift-ca\") pod \"28dd80f8-8d21-4d64-9327-104671fbbe0d\" (UID: \"28dd80f8-8d21-4d64-9327-104671fbbe0d\") " Oct 30 17:05:04.925686 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:04.925526 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28dd80f8-8d21-4d64-9327-104671fbbe0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28dd80f8-8d21-4d64-9327-104671fbbe0d-maas-api-xks-ca\") pod \"28dd80f8-8d21-4d64-9327-104671fbbe0d\" (UID: \"28dd80f8-8d21-4d64-9327-104671fbbe0d\") " Oct 30 17:05:04.925749 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:04.925699 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28dd80f8-8d21-4d64-9327-104671fbbe0d-kube-api-access-vt262\" (UniqueName: \"kubernetes.io/projected/28dd80f8-8d21-4d64-9327-104671fbbe0d-kube-api-access-vt262\") pod \"28dd80f8-8d21-4d64-9327-104671fbbe0d\" (UID: \"28dd80f8-8d21-4d64-9327-104671fbbe0d\") " Oct 30 17:05:04.926807 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:04.926694 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28dd80f8-8d21-4d64-9327-104671fbbe0d-maas-api-openshift-ca" pod "28dd80f8-8d21-4d64-9327-104671fbbe0d" (UID: "28dd80f8-8d21-4d64-9327-104671fbbe0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:04.930235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:04.930208 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28dd80f8-8d21-4d64-9327-104671fbbe0d-maas-api-xks-ca" pod "28dd80f8-8d21-4d64-9327-104671fbbe0d" (UID: "28dd80f8-8d21-4d64-9327-104671fbbe0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:05:04.931268 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:04.931235 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28dd80f8-8d21-4d64-9327-104671fbbe0d-kube-api-access-vt262" pod "28dd80f8-8d21-4d64-9327-104671fbbe0d" (UID: "28dd80f8-8d21-4d64-9327-104671fbbe0d"). InnerVolumeSpecName "kube-api-access-vt262". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:05.027484 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:05.027442 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28dd80f8-8d21-4d64-9327-104671fbbe0d-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:05:05.027484 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:05.027477 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28dd80f8-8d21-4d64-9327-104671fbbe0d-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:05:05.027484 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:05.027487 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt262\" (UniqueName: \"kubernetes.io/projected/28dd80f8-8d21-4d64-9327-104671fbbe0d-kube-api-access-vt262\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:05:05.208564 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:05.208481 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:05.687852 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:05.687817 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc" event={"ID":"28dd80f8-8d21-4d64-9327-104671fbbe0d","Type":"ContainerDied","Data":"6467e6b70a486994be19a827256063594dc2e7b6bc9aba4929cca1e1b23c93e3"} Oct 30 17:05:05.688057 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:05.687828 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc" Oct 30 17:05:05.688057 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:05.687913 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6467e6b70a486994be19a827256063594dc2e7b6bc9aba4929cca1e1b23c93e3" Oct 30 17:05:06.903707 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:06.903664 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:08.031136 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:08.031095 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl"] Oct 30 17:05:08.044700 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:08.044662 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f0ec459-kxwhl"] Oct 30 17:05:08.204597 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:08.204563 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:08.413842 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:08.413806 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abac9b81-c250-4d4d-b1ed-f0615a7f2dd9" path="/var/lib/kubelet/pods/abac9b81-c250-4d4d-b1ed-f0615a7f2dd9/volumes" Oct 30 17:05:08.776486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:08.776365 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:10.402418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:10.402382 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:12.027234 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:12.027186 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:12.167047 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:12.166915 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:14.065725 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:14.065678 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:14.071224 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:05:14.071190 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:05:15.001021 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.000983 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq"] Oct 30 17:05:15.001809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.001759 6202 cpu_manager.go:414] "containerMap: removing container" podUID="28dd80f8-8d21-4d64-9327-104671fbbe0d" containerName="revoke-keys" Oct 30 17:05:15.001809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.001797 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28dd80f8-8d21-4d64-9327-104671fbbe0d" containerName="revoke-keys" Oct 30 17:05:15.002028 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.001998 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="28dd80f8-8d21-4d64-9327-104671fbbe0d" containerName="revoke-keys" Oct 30 17:05:15.006039 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.006018 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq" Oct 30 17:05:15.022535 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.022507 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:05:15.026445 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.026412 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq"] Oct 30 17:05:15.033428 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.033401 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:05:15.033599 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.033495 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:05:15.033599 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.033514 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:05:15.129520 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.129480 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff05222-a66b-4f7a-994e-4df8c8f2cac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c58226f4-64tvq\" (UID: \"fff05222-a66b-4f7a-994e-4df8c8f2cac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq" Oct 30 17:05:15.129988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.129595 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff05222-a66b-4f7a-994e-4df8c8f2cac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c58226f4-64tvq\" (UID: \"fff05222-a66b-4f7a-994e-4df8c8f2cac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq" Oct 30 17:05:15.129988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.129666 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl6vk\" (UniqueName: \"kubernetes.io/projected/fff05222-a66b-4f7a-994e-4df8c8f2cac5-kube-api-access-nl6vk\") pod \"maas-api-revoke-sub-c58226f4-64tvq\" (UID: \"fff05222-a66b-4f7a-994e-4df8c8f2cac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq" Oct 30 17:05:15.231560 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.231519 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff05222-a66b-4f7a-994e-4df8c8f2cac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c58226f4-64tvq\" (UID: \"fff05222-a66b-4f7a-994e-4df8c8f2cac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq" Oct 30 17:05:15.231714 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.231587 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl6vk\" (UniqueName: \"kubernetes.io/projected/fff05222-a66b-4f7a-994e-4df8c8f2cac5-kube-api-access-nl6vk\") pod \"maas-api-revoke-sub-c58226f4-64tvq\" (UID: \"fff05222-a66b-4f7a-994e-4df8c8f2cac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq" Oct 30 17:05:15.231810 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.231797 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff05222-a66b-4f7a-994e-4df8c8f2cac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c58226f4-64tvq\" (UID: \"fff05222-a66b-4f7a-994e-4df8c8f2cac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq" Oct 30 17:05:15.232267 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.232244 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff05222-a66b-4f7a-994e-4df8c8f2cac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c58226f4-64tvq\" (UID: \"fff05222-a66b-4f7a-994e-4df8c8f2cac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq" Oct 30 17:05:15.234308 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.234290 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff05222-a66b-4f7a-994e-4df8c8f2cac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c58226f4-64tvq\" (UID: \"fff05222-a66b-4f7a-994e-4df8c8f2cac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq" Oct 30 17:05:15.263491 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.263410 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl6vk\" (UniqueName: \"kubernetes.io/projected/fff05222-a66b-4f7a-994e-4df8c8f2cac5-kube-api-access-nl6vk\") pod \"maas-api-revoke-sub-c58226f4-64tvq\" (UID: \"fff05222-a66b-4f7a-994e-4df8c8f2cac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq" Oct 30 17:05:15.335035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.334996 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq" Oct 30 17:05:15.642222 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.642190 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq"] Oct 30 17:05:15.777675 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.777628 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq" event={"ID":"fff05222-a66b-4f7a-994e-4df8c8f2cac5","Type":"ContainerStarted","Data":"dc8c5ca6155759feee8ab8779966e21e66623f64ccf95b781d32fe547a940dbb"} Oct 30 17:05:15.820350 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:15.820311 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:16.176721 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:16.176676 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:16.788286 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:16.788254 6202 generic.go:356] "Generic (PLEG): container finished" podID="fff05222-a66b-4f7a-994e-4df8c8f2cac5" containerID="f49f4ce7186d5e83ea906a1da32107dbe1d9757ac9dae7a812d07078ecbbf7b7" exitCode=0 Oct 30 17:05:16.788438 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:16.788309 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq" event={"ID":"fff05222-a66b-4f7a-994e-4df8c8f2cac5","Type":"ContainerDied","Data":"f49f4ce7186d5e83ea906a1da32107dbe1d9757ac9dae7a812d07078ecbbf7b7"} Oct 30 17:05:17.031727 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:17.031636 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn"] Oct 30 17:05:17.036292 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:17.036255 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376c46dc-nqkpn"] Oct 30 17:05:17.576103 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:17.576058 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:17.981566 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:17.981535 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq" Oct 30 17:05:18.080692 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:18.080654 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff05222-a66b-4f7a-994e-4df8c8f2cac5-kube-api-access-nl6vk\" (UniqueName: \"kubernetes.io/projected/fff05222-a66b-4f7a-994e-4df8c8f2cac5-kube-api-access-nl6vk\") pod \"fff05222-a66b-4f7a-994e-4df8c8f2cac5\" (UID: \"fff05222-a66b-4f7a-994e-4df8c8f2cac5\") " Oct 30 17:05:18.080944 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:18.080796 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff05222-a66b-4f7a-994e-4df8c8f2cac5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff05222-a66b-4f7a-994e-4df8c8f2cac5-maas-api-xks-ca\") pod \"fff05222-a66b-4f7a-994e-4df8c8f2cac5\" (UID: \"fff05222-a66b-4f7a-994e-4df8c8f2cac5\") " Oct 30 17:05:18.081023 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:18.081002 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff05222-a66b-4f7a-994e-4df8c8f2cac5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff05222-a66b-4f7a-994e-4df8c8f2cac5-maas-api-openshift-ca\") pod \"fff05222-a66b-4f7a-994e-4df8c8f2cac5\" (UID: \"fff05222-a66b-4f7a-994e-4df8c8f2cac5\") " Oct 30 17:05:18.081502 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:18.081473 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff05222-a66b-4f7a-994e-4df8c8f2cac5-maas-api-openshift-ca" pod "fff05222-a66b-4f7a-994e-4df8c8f2cac5" (UID: "fff05222-a66b-4f7a-994e-4df8c8f2cac5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:18.081704 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:18.081686 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff05222-a66b-4f7a-994e-4df8c8f2cac5-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:05:18.083654 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:18.083632 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff05222-a66b-4f7a-994e-4df8c8f2cac5-maas-api-xks-ca" pod "fff05222-a66b-4f7a-994e-4df8c8f2cac5" (UID: "fff05222-a66b-4f7a-994e-4df8c8f2cac5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:05:18.083740 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:18.083693 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff05222-a66b-4f7a-994e-4df8c8f2cac5-kube-api-access-nl6vk" pod "fff05222-a66b-4f7a-994e-4df8c8f2cac5" (UID: "fff05222-a66b-4f7a-994e-4df8c8f2cac5"). InnerVolumeSpecName "kube-api-access-nl6vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:18.115132 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:18.115051 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:18.183119 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:18.183077 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff05222-a66b-4f7a-994e-4df8c8f2cac5-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:05:18.183119 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:18.183113 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl6vk\" (UniqueName: \"kubernetes.io/projected/fff05222-a66b-4f7a-994e-4df8c8f2cac5-kube-api-access-nl6vk\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:05:18.413959 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:18.413870 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b6497d9-6147-4773-bc38-5e0f6bf7eb2f" path="/var/lib/kubelet/pods/9b6497d9-6147-4773-bc38-5e0f6bf7eb2f/volumes" Oct 30 17:05:18.808347 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:18.808263 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq" Oct 30 17:05:18.808347 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:18.808304 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq" event={"ID":"fff05222-a66b-4f7a-994e-4df8c8f2cac5","Type":"ContainerDied","Data":"dc8c5ca6155759feee8ab8779966e21e66623f64ccf95b781d32fe547a940dbb"} Oct 30 17:05:18.808944 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:18.808353 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc8c5ca6155759feee8ab8779966e21e66623f64ccf95b781d32fe547a940dbb" Oct 30 17:05:18.876557 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:18.876524 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:19.294375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:19.294334 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:21.123517 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:21.123482 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:21.877591 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:21.877560 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q"] Oct 30 17:05:21.878384 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:21.878366 6202 cpu_manager.go:414] "containerMap: removing container" podUID="fff05222-a66b-4f7a-994e-4df8c8f2cac5" containerName="revoke-keys" Oct 30 17:05:21.878384 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:21.878385 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff05222-a66b-4f7a-994e-4df8c8f2cac5" containerName="revoke-keys" Oct 30 17:05:21.878570 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:21.878559 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff05222-a66b-4f7a-994e-4df8c8f2cac5" containerName="revoke-keys" Oct 30 17:05:21.882372 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:21.882355 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q" Oct 30 17:05:21.898007 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:21.897976 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:05:21.902836 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:21.902809 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q"] Oct 30 17:05:21.909989 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:21.909965 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:05:21.910154 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:21.910054 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:05:21.910154 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:21.910068 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:05:22.039353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:22.039321 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f76c6767-7x94q\" (UID: \"6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q" Oct 30 17:05:22.039534 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:22.039464 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f76c6767-7x94q\" (UID: \"6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q" Oct 30 17:05:22.039684 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:22.039657 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb52j\" (UniqueName: \"kubernetes.io/projected/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-kube-api-access-vb52j\") pod \"maas-api-revoke-sub-f76c6767-7x94q\" (UID: \"6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q" Oct 30 17:05:22.141517 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:22.141423 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f76c6767-7x94q\" (UID: \"6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q" Oct 30 17:05:22.142012 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:22.141565 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f76c6767-7x94q\" (UID: \"6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q" Oct 30 17:05:22.142012 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:22.141686 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb52j\" (UniqueName: \"kubernetes.io/projected/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-kube-api-access-vb52j\") pod \"maas-api-revoke-sub-f76c6767-7x94q\" (UID: \"6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q" Oct 30 17:05:22.142237 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:22.142216 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f76c6767-7x94q\" (UID: \"6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q" Oct 30 17:05:22.144269 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:22.144243 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f76c6767-7x94q\" (UID: \"6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q" Oct 30 17:05:22.165338 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:22.165308 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb52j\" (UniqueName: \"kubernetes.io/projected/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-kube-api-access-vb52j\") pod \"maas-api-revoke-sub-f76c6767-7x94q\" (UID: \"6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q" Oct 30 17:05:22.196986 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:22.196952 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q" Oct 30 17:05:22.504185 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:22.504148 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q"] Oct 30 17:05:22.504363 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:05:22.504339 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3964f6_35bd_4cbb_a70f_76b7eeff0d6a.slice/crio-893f7b7aa3455edac73356fb36cb40a0ab36c6fa8c1c4de09b7c9a8d4bcbe336 WatchSource:0}: Error finding container 893f7b7aa3455edac73356fb36cb40a0ab36c6fa8c1c4de09b7c9a8d4bcbe336: Status 404 returned error can't find the container with id 893f7b7aa3455edac73356fb36cb40a0ab36c6fa8c1c4de09b7c9a8d4bcbe336 Oct 30 17:05:22.760695 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:22.760618 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:22.870234 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:22.870191 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q" event={"ID":"6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a","Type":"ContainerStarted","Data":"893f7b7aa3455edac73356fb36cb40a0ab36c6fa8c1c4de09b7c9a8d4bcbe336"} Oct 30 17:05:23.881351 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:23.881311 6202 generic.go:356] "Generic (PLEG): container finished" podID="6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a" containerID="52737995839a0d2e3a8a8e67d7edff7fdabc314931d2d34f6daa2ffc907e1364" exitCode=0 Oct 30 17:05:23.881893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:23.881396 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q" event={"ID":"6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a","Type":"ContainerDied","Data":"52737995839a0d2e3a8a8e67d7edff7fdabc314931d2d34f6daa2ffc907e1364"} Oct 30 17:05:24.534134 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:24.534099 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:25.081879 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:25.081847 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q" Oct 30 17:05:25.227619 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:25.227540 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-maas-api-openshift-ca\") pod \"6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a\" (UID: \"6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a\") " Oct 30 17:05:25.227619 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:25.227583 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-maas-api-xks-ca\") pod \"6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a\" (UID: \"6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a\") " Oct 30 17:05:25.227872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:25.227825 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-kube-api-access-vb52j\" (UniqueName: \"kubernetes.io/projected/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-kube-api-access-vb52j\") pod \"6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a\" (UID: \"6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a\") " Oct 30 17:05:25.227993 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:25.227967 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-maas-api-openshift-ca" pod "6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a" (UID: "6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:25.228532 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:25.228509 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:05:25.229923 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:25.229900 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-maas-api-xks-ca" pod "6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a" (UID: "6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:05:25.230036 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:25.229989 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-kube-api-access-vb52j" pod "6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a" (UID: "6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a"). InnerVolumeSpecName "kube-api-access-vb52j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:25.330418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:25.330382 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:05:25.330418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:25.330412 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb52j\" (UniqueName: \"kubernetes.io/projected/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a-kube-api-access-vb52j\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:05:25.903438 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:25.903410 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q" Oct 30 17:05:25.903604 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:25.903442 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q" event={"ID":"6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a","Type":"ContainerDied","Data":"893f7b7aa3455edac73356fb36cb40a0ab36c6fa8c1c4de09b7c9a8d4bcbe336"} Oct 30 17:05:25.903604 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:25.903483 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="893f7b7aa3455edac73356fb36cb40a0ab36c6fa8c1c4de09b7c9a8d4bcbe336" Oct 30 17:05:26.438941 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:26.438861 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:28.306066 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:28.306029 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:29.564848 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:29.564795 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:29.957093 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:29.957043 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:30.231373 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:30.231284 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:31.030154 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:31.030110 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk"] Oct 30 17:05:31.043901 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:31.043859 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-795ac4c1-95jmk"] Oct 30 17:05:31.751372 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:31.751332 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:32.412721 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:32.412679 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c680bca6-5420-4bcc-8ef5-5bfecb85218e" path="/var/lib/kubelet/pods/c680bca6-5420-4bcc-8ef5-5bfecb85218e/volumes" Oct 30 17:05:33.126959 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:33.126929 6202 scope.go:122] "RemoveContainer" containerID="d338bb26291bb6467fa95f4da28630c67148c5225f466ab85976bb59615ca15e" Oct 30 17:05:33.138271 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:33.138244 6202 scope.go:122] "RemoveContainer" containerID="d92f23068a207b8e53014819a18447680664b70a89e32c415db6e8aac7b9c11d" Oct 30 17:05:33.152345 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:33.152322 6202 scope.go:122] "RemoveContainer" containerID="4eea44602423055b25679bc409203dbaa23cf941d3977646674b22bc0619b61e" Oct 30 17:05:33.165395 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:33.165373 6202 scope.go:122] "RemoveContainer" containerID="2f02ac66ba72a04c57015d0dc0e8432683f6c37ff2dd5ace9ccd9c6c8b50a9d9" Oct 30 17:05:33.175312 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:33.175288 6202 scope.go:122] "RemoveContainer" containerID="6b61dc69ce24c878600b3dd3f2abcbaa510b191381753a64bc54c930c70aa686" Oct 30 17:05:33.594642 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:33.594600 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:35.259160 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:35.259125 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:37.069720 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:37.069678 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:38.893042 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:38.893003 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:39.243982 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:39.243894 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:40.808004 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:40.807962 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:41.485796 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:41.485253 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:41.901008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:41.894577 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:42.510816 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:42.510755 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:43.219795 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:43.218483 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:44.097049 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:05:44.097011 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:05:44.890266 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:44.890228 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:45.200332 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:45.200249 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:45.570728 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:45.570636 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:45.984018 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:45.983980 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj"] Oct 30 17:05:45.985071 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:45.985046 6202 cpu_manager.go:414] "containerMap: removing container" podUID="6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a" containerName="revoke-keys" Oct 30 17:05:45.985071 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:45.985070 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a" containerName="revoke-keys" Oct 30 17:05:45.985309 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:45.985294 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a" containerName="revoke-keys" Oct 30 17:05:45.991972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:45.991947 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.015300 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.015269 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj"] Oct 30 17:05:46.020928 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.020898 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-55e83e-openshift-default-dockercfg-w5hdx\"" Oct 30 17:05:46.082020 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.081983 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-workload-certs\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.083300 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.083133 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww9rz\" (UniqueName: \"kubernetes.io/projected/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-kube-api-access-ww9rz\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.083594 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.083574 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-podinfo\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.083817 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.082616 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:46.083973 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.083951 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istiod-ca-cert\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.084103 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.084089 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-credential-socket\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.084262 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.084247 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-envoy\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.084413 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.084400 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-token\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.084521 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.084508 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-data\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.084632 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.084619 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-workload-socket\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.272342 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.272237 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istiod-ca-cert\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.300340 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.300305 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-credential-socket\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.300724 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.300703 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-envoy\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.300963 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.300944 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-token\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.301084 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.301070 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-data\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.301208 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.301193 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-workload-socket\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.301347 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.301335 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-workload-certs\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.301541 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.301528 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww9rz\" (UniqueName: \"kubernetes.io/projected/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-kube-api-access-ww9rz\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.301657 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.301644 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-podinfo\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.302434 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.298960 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istiod-ca-cert\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.302434 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.302355 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-credential-socket\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.302613 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.302491 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-workload-socket\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.326790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.321316 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-data\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.326790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.321639 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-workload-certs\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.331814 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.331723 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-envoy\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.344904 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.335104 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-podinfo\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.344904 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.342269 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-token\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.357791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.354291 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww9rz\" (UniqueName: \"kubernetes.io/projected/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-kube-api-access-ww9rz\") pod \"e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.419910 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.419879 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:46.613627 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.613592 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:46.982719 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.982690 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj"] Oct 30 17:05:46.982860 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:05:46.982755 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d43f09c_faf8_49d7_8c86_0cfd033cff9d.slice/crio-edeb5c862b4b27f9ccfb1191be3e5bdceb0a432c82e2a405163a486ccee525c9 WatchSource:0}: Error finding container edeb5c862b4b27f9ccfb1191be3e5bdceb0a432c82e2a405163a486ccee525c9: Status 404 returned error can't find the container with id edeb5c862b4b27f9ccfb1191be3e5bdceb0a432c82e2a405163a486ccee525c9 Oct 30 17:05:46.985122 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.985092 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:46.985864 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.985834 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:05:46.985956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.985939 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:05:46.986011 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:46.985978 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:05:47.098799 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:47.093936 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb"] Oct 30 17:05:47.106788 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:47.106078 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24f73874-ks6lb"] Oct 30 17:05:47.319450 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:47.319360 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" event={"ID":"6d43f09c-faf8-49d7-8c86-0cfd033cff9d","Type":"ContainerStarted","Data":"5e54e4c068085e8c968eacc72829d9c0169a8d4335b73138062df9664f8148ae"} Oct 30 17:05:47.319450 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:47.319415 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" event={"ID":"6d43f09c-faf8-49d7-8c86-0cfd033cff9d","Type":"ContainerStarted","Data":"edeb5c862b4b27f9ccfb1191be3e5bdceb0a432c82e2a405163a486ccee525c9"} Oct 30 17:05:47.377622 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:47.377553 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" podStartSLOduration=2.377532305 podStartE2EDuration="2.377532305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:47.377066498 +0000 UTC m=+2661.087521243" watchObservedRunningTime="2026-10-30 17:05:47.377532305 +0000 UTC m=+2661.087987050" Oct 30 17:05:47.613880 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:47.613844 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:48.413116 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:48.413085 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77cf4478-e8b2-4a37-afe9-caee2bfae0c6" path="/var/lib/kubelet/pods/77cf4478-e8b2-4a37-afe9-caee2bfae0c6/volumes" Oct 30 17:05:48.620092 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:48.620063 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:48.821400 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:48.821366 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:49.335470 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:49.335435 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:49.336431 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:49.336413 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:05:49.798638 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:49.798556 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:49.923344 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:49.923301 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:50.899676 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:50.899567 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:51.478121 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:51.477973 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:51.613319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:51.613271 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:52.707874 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:52.707830 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:53.381149 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:53.381053 6202 generic.go:356] "Generic (PLEG): container finished" podID="cd39addc-aef3-41d3-aae4-6358ffbdf53e" containerID="2c0a61cf7c8464598a3b322eade405e2b9d16d04dcef15d59d6977914c279a8b" exitCode=0 Oct 30 17:05:53.381149 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:53.381079 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" event={"ID":"cd39addc-aef3-41d3-aae4-6358ffbdf53e","Type":"ContainerDied","Data":"2c0a61cf7c8464598a3b322eade405e2b9d16d04dcef15d59d6977914c279a8b"} Oct 30 17:05:53.381595 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:53.381579 6202 scope.go:122] "RemoveContainer" containerID="2c0a61cf7c8464598a3b322eade405e2b9d16d04dcef15d59d6977914c279a8b" Oct 30 17:05:53.383539 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:53.383508 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:05:53.383667 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:53.383578 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:05:53.383667 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:53.383610 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:05:54.363790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.355720 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx"] Oct 30 17:05:54.375114 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.375059 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx" Oct 30 17:05:54.380887 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.380833 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx"] Oct 30 17:05:54.390689 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.390659 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:05:54.409025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.408996 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:05:54.410422 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.410396 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:54.421507 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.421473 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" event={"ID":"cd39addc-aef3-41d3-aae4-6358ffbdf53e","Type":"ContainerStarted","Data":"4530bfe5562f98ab30ee7a33dfef936b647d877c98f64832a0b845e0e8b8a007"} Oct 30 17:05:54.422171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.422152 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5pch\" (UniqueName: \"kubernetes.io/projected/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-kube-api-access-c5pch\") pod \"maas-api-revoke-sub-690e7e86-krkbx\" (UID: \"f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx" Oct 30 17:05:54.422341 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.422321 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-690e7e86-krkbx\" (UID: \"f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx" Oct 30 17:05:54.422533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.422514 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-690e7e86-krkbx\" (UID: \"f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx" Oct 30 17:05:54.484791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.475626 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:05:54.484791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.475717 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:05:54.523985 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.523917 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-690e7e86-krkbx\" (UID: \"f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx" Oct 30 17:05:54.524292 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.524113 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5pch\" (UniqueName: \"kubernetes.io/projected/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-kube-api-access-c5pch\") pod \"maas-api-revoke-sub-690e7e86-krkbx\" (UID: \"f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx" Oct 30 17:05:54.524292 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.524224 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-690e7e86-krkbx\" (UID: \"f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx" Oct 30 17:05:54.524948 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.524924 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-690e7e86-krkbx\" (UID: \"f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx" Oct 30 17:05:54.526841 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.526820 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-690e7e86-krkbx\" (UID: \"f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx" Oct 30 17:05:54.567803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.561084 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5pch\" (UniqueName: \"kubernetes.io/projected/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-kube-api-access-c5pch\") pod \"maas-api-revoke-sub-690e7e86-krkbx\" (UID: \"f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx" Oct 30 17:05:54.699041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:54.698962 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx" Oct 30 17:05:55.032516 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:05:55.032487 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2759d9a_b6e1_4b4f_8caa_50e4ae8864fb.slice/crio-40fc4f48c6b77e9ae45e384a75b183d48615971defaf14b2e4537937d45c1771 WatchSource:0}: Error finding container 40fc4f48c6b77e9ae45e384a75b183d48615971defaf14b2e4537937d45c1771: Status 404 returned error can't find the container with id 40fc4f48c6b77e9ae45e384a75b183d48615971defaf14b2e4537937d45c1771 Oct 30 17:05:55.032855 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:55.032831 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx"] Oct 30 17:05:55.096756 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:55.096725 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:05:55.096964 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:55.096941 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Oct 30 17:05:55.097076 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:55.097008 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" podUID="cd39addc-aef3-41d3-aae4-6358ffbdf53e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Oct 30 17:05:55.419321 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:55.419282 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx" event={"ID":"f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb","Type":"ContainerStarted","Data":"40fc4f48c6b77e9ae45e384a75b183d48615971defaf14b2e4537937d45c1771"} Oct 30 17:05:56.129790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:56.128927 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Oct 30 17:05:56.129790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:56.129001 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" podUID="cd39addc-aef3-41d3-aae4-6358ffbdf53e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Oct 30 17:05:56.136564 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:56.136534 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:56.428872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:56.428745 6202 generic.go:356] "Generic (PLEG): container finished" podID="f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb" containerID="9fec06199b5da0002d0378c6365cd71905be72ab04e801ffb7395f4702059893" exitCode=0 Oct 30 17:05:56.428872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:56.428832 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx" event={"ID":"f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb","Type":"ContainerDied","Data":"9fec06199b5da0002d0378c6365cd71905be72ab04e801ffb7395f4702059893"} Oct 30 17:05:57.096851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:57.096815 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Oct 30 17:05:57.097083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:57.096876 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" podUID="cd39addc-aef3-41d3-aae4-6358ffbdf53e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Oct 30 17:05:57.690481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:57.690448 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx" Oct 30 17:05:57.714946 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:57.714921 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-maas-api-xks-ca\") pod \"f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb\" (UID: \"f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb\") " Oct 30 17:05:57.715090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:57.714980 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-maas-api-openshift-ca\") pod \"f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb\" (UID: \"f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb\") " Oct 30 17:05:57.715090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:57.715065 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-kube-api-access-c5pch\" (UniqueName: \"kubernetes.io/projected/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-kube-api-access-c5pch\") pod \"f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb\" (UID: \"f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb\") " Oct 30 17:05:57.715320 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:57.715296 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-maas-api-openshift-ca" pod "f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb" (UID: "f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:57.715587 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:57.715519 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:05:57.717166 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:57.717140 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-maas-api-xks-ca" pod "f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb" (UID: "f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:05:57.717284 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:57.717252 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-kube-api-access-c5pch" pod "f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb" (UID: "f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb"). InnerVolumeSpecName "kube-api-access-c5pch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:57.817547 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:57.817508 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5pch\" (UniqueName: \"kubernetes.io/projected/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-kube-api-access-c5pch\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:05:57.817547 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:57.817549 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:05:58.007455 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:58.006568 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:58.125830 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:58.117133 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Oct 30 17:05:58.125830 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:58.117211 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" podUID="cd39addc-aef3-41d3-aae4-6358ffbdf53e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Oct 30 17:05:58.476751 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:58.476714 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx" Oct 30 17:05:58.476963 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:58.476760 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx" event={"ID":"f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb","Type":"ContainerDied","Data":"40fc4f48c6b77e9ae45e384a75b183d48615971defaf14b2e4537937d45c1771"} Oct 30 17:05:58.476963 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:58.476851 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40fc4f48c6b77e9ae45e384a75b183d48615971defaf14b2e4537937d45c1771" Oct 30 17:05:59.096957 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:59.096914 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body= Oct 30 17:05:59.097403 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:59.096972 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" podUID="cd39addc-aef3-41d3-aae4-6358ffbdf53e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" Oct 30 17:05:59.660296 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:59.660254 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98"] Oct 30 17:05:59.778257 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:59.778199 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:59.910864 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:05:59.899330 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:00.023156 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:00.023123 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:00.191377 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:00.191285 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:06:00.192162 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:00.192145 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:06:00.192931 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:00.192877 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:06:01.482583 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:01.482543 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:03.169287 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:03.169246 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:04.661031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.660990 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98" podUID="3f7598e1-39ec-436a-a92e-371400e6580f" containerName="payload-pre-processing" containerID="cri-o://90fac49b2674ef30c862b79bc3b4f1e6cd7cae0fa75eb63ac3a3640111f9cf70" gracePeriod=25 Oct 30 17:06:04.797600 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.797559 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m"] Oct 30 17:06:04.798461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.798435 6202 cpu_manager.go:414] "containerMap: removing container" podUID="f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb" containerName="revoke-keys" Oct 30 17:06:04.798461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.798456 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb" containerName="revoke-keys" Oct 30 17:06:04.798649 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.798620 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb" containerName="revoke-keys" Oct 30 17:06:04.801336 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.801314 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m" Oct 30 17:06:04.817191 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.817160 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:06:04.827713 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.827634 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m"] Oct 30 17:06:04.829241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.829219 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:06:04.829506 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.829486 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:06:04.836636 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.836614 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:06:04.840893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.840808 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp2jf\" (UniqueName: \"kubernetes.io/projected/3c003e7f-5973-4479-88e5-cacb12f92fba-kube-api-access-tp2jf\") pod \"maas-api-revoke-sub-6f6b00e4-6jz7m\" (UID: \"3c003e7f-5973-4479-88e5-cacb12f92fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m" Oct 30 17:06:04.841191 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.841165 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c003e7f-5973-4479-88e5-cacb12f92fba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f6b00e4-6jz7m\" (UID: \"3c003e7f-5973-4479-88e5-cacb12f92fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m" Oct 30 17:06:04.841472 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.841443 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c003e7f-5973-4479-88e5-cacb12f92fba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f6b00e4-6jz7m\" (UID: \"3c003e7f-5973-4479-88e5-cacb12f92fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m" Oct 30 17:06:04.943072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.942972 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp2jf\" (UniqueName: \"kubernetes.io/projected/3c003e7f-5973-4479-88e5-cacb12f92fba-kube-api-access-tp2jf\") pod \"maas-api-revoke-sub-6f6b00e4-6jz7m\" (UID: \"3c003e7f-5973-4479-88e5-cacb12f92fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m" Oct 30 17:06:04.943072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.943064 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c003e7f-5973-4479-88e5-cacb12f92fba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f6b00e4-6jz7m\" (UID: \"3c003e7f-5973-4479-88e5-cacb12f92fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m" Oct 30 17:06:04.943314 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.943176 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c003e7f-5973-4479-88e5-cacb12f92fba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f6b00e4-6jz7m\" (UID: \"3c003e7f-5973-4479-88e5-cacb12f92fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m" Oct 30 17:06:04.944016 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.943945 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c003e7f-5973-4479-88e5-cacb12f92fba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f6b00e4-6jz7m\" (UID: \"3c003e7f-5973-4479-88e5-cacb12f92fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m" Oct 30 17:06:04.945972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.945941 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c003e7f-5973-4479-88e5-cacb12f92fba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f6b00e4-6jz7m\" (UID: \"3c003e7f-5973-4479-88e5-cacb12f92fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m" Oct 30 17:06:04.965950 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.965922 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp2jf\" (UniqueName: \"kubernetes.io/projected/3c003e7f-5973-4479-88e5-cacb12f92fba-kube-api-access-tp2jf\") pod \"maas-api-revoke-sub-6f6b00e4-6jz7m\" (UID: \"3c003e7f-5973-4479-88e5-cacb12f92fba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m" Oct 30 17:06:04.970577 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:04.970548 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:05.003350 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.003323 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98" Oct 30 17:06:05.044972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.044595 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7598e1-39ec-436a-a92e-371400e6580f-kube-api-access-bs9p7\" (UniqueName: \"kubernetes.io/projected/3f7598e1-39ec-436a-a92e-371400e6580f-kube-api-access-bs9p7\") pod \"3f7598e1-39ec-436a-a92e-371400e6580f\" (UID: \"3f7598e1-39ec-436a-a92e-371400e6580f\") " Oct 30 17:06:05.044972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.044825 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7598e1-39ec-436a-a92e-371400e6580f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7598e1-39ec-436a-a92e-371400e6580f-plugins-config-volume\") pod \"3f7598e1-39ec-436a-a92e-371400e6580f\" (UID: \"3f7598e1-39ec-436a-a92e-371400e6580f\") " Oct 30 17:06:05.049663 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.049620 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7598e1-39ec-436a-a92e-371400e6580f-plugins-config-volume" pod "3f7598e1-39ec-436a-a92e-371400e6580f" (UID: "3f7598e1-39ec-436a-a92e-371400e6580f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:05.069418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.069352 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7598e1-39ec-436a-a92e-371400e6580f-kube-api-access-bs9p7" pod "3f7598e1-39ec-436a-a92e-371400e6580f" (UID: "3f7598e1-39ec-436a-a92e-371400e6580f"). InnerVolumeSpecName "kube-api-access-bs9p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:05.116440 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.116405 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m" Oct 30 17:06:05.149120 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.147493 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs9p7\" (UniqueName: \"kubernetes.io/projected/3f7598e1-39ec-436a-a92e-371400e6580f-kube-api-access-bs9p7\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:06:05.149120 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.147528 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7598e1-39ec-436a-a92e-371400e6580f-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:06:05.433373 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:06:05.433334 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c003e7f_5973_4479_88e5_cacb12f92fba.slice/crio-56cc4c3814a3ea3df20b65cfcd400e48866115f497f39dc94dcc672215918e9b WatchSource:0}: Error finding container 56cc4c3814a3ea3df20b65cfcd400e48866115f497f39dc94dcc672215918e9b: Status 404 returned error can't find the container with id 56cc4c3814a3ea3df20b65cfcd400e48866115f497f39dc94dcc672215918e9b Oct 30 17:06:05.434104 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.434080 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m"] Oct 30 17:06:05.589474 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.589435 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m" event={"ID":"3c003e7f-5973-4479-88e5-cacb12f92fba","Type":"ContainerStarted","Data":"56cc4c3814a3ea3df20b65cfcd400e48866115f497f39dc94dcc672215918e9b"} Oct 30 17:06:05.591028 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.590996 6202 generic.go:356] "Generic (PLEG): container finished" podID="3f7598e1-39ec-436a-a92e-371400e6580f" containerID="90fac49b2674ef30c862b79bc3b4f1e6cd7cae0fa75eb63ac3a3640111f9cf70" exitCode=0 Oct 30 17:06:05.591156 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.591051 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98" Oct 30 17:06:05.591156 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.591082 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98" event={"ID":"3f7598e1-39ec-436a-a92e-371400e6580f","Type":"ContainerDied","Data":"90fac49b2674ef30c862b79bc3b4f1e6cd7cae0fa75eb63ac3a3640111f9cf70"} Oct 30 17:06:05.591156 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.591136 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98" event={"ID":"3f7598e1-39ec-436a-a92e-371400e6580f","Type":"ContainerDied","Data":"89f24c2d463bcb1870cbd93a642d71a3cf28a311840c552d0ec50992dcbc0cc2"} Oct 30 17:06:05.591277 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.591160 6202 scope.go:122] "RemoveContainer" containerID="90fac49b2674ef30c862b79bc3b4f1e6cd7cae0fa75eb63ac3a3640111f9cf70" Oct 30 17:06:05.603838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.603814 6202 scope.go:122] "RemoveContainer" containerID="90fac49b2674ef30c862b79bc3b4f1e6cd7cae0fa75eb63ac3a3640111f9cf70" Oct 30 17:06:05.604209 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:06:05.604178 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90fac49b2674ef30c862b79bc3b4f1e6cd7cae0fa75eb63ac3a3640111f9cf70\": container with ID starting with 90fac49b2674ef30c862b79bc3b4f1e6cd7cae0fa75eb63ac3a3640111f9cf70 not found: ID does not exist" containerID="90fac49b2674ef30c862b79bc3b4f1e6cd7cae0fa75eb63ac3a3640111f9cf70" Oct 30 17:06:05.604285 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.604222 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90fac49b2674ef30c862b79bc3b4f1e6cd7cae0fa75eb63ac3a3640111f9cf70"} err="failed to get container status \"90fac49b2674ef30c862b79bc3b4f1e6cd7cae0fa75eb63ac3a3640111f9cf70\": rpc error: code = NotFound desc = could not find container \"90fac49b2674ef30c862b79bc3b4f1e6cd7cae0fa75eb63ac3a3640111f9cf70\": container with ID starting with 90fac49b2674ef30c862b79bc3b4f1e6cd7cae0fa75eb63ac3a3640111f9cf70 not found: ID does not exist" Oct 30 17:06:05.646890 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.646859 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98"] Oct 30 17:06:05.657986 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:05.657949 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-7c87c98bf9-lsz98"] Oct 30 17:06:06.414539 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:06.414504 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7598e1-39ec-436a-a92e-371400e6580f" path="/var/lib/kubelet/pods/3f7598e1-39ec-436a-a92e-371400e6580f/volumes" Oct 30 17:06:06.583622 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:06.583586 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:06.600622 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:06.600586 6202 generic.go:356] "Generic (PLEG): container finished" podID="3c003e7f-5973-4479-88e5-cacb12f92fba" containerID="40413c7ad35a7b14921d00b8634c1e61d3959f83160e47863d357e08e7ef7d2c" exitCode=0 Oct 30 17:06:06.600795 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:06.600678 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m" event={"ID":"3c003e7f-5973-4479-88e5-cacb12f92fba","Type":"ContainerDied","Data":"40413c7ad35a7b14921d00b8634c1e61d3959f83160e47863d357e08e7ef7d2c"} Oct 30 17:06:07.796270 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:07.796245 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m" Oct 30 17:06:07.889345 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:07.889249 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c003e7f-5973-4479-88e5-cacb12f92fba-kube-api-access-tp2jf\" (UniqueName: \"kubernetes.io/projected/3c003e7f-5973-4479-88e5-cacb12f92fba-kube-api-access-tp2jf\") pod \"3c003e7f-5973-4479-88e5-cacb12f92fba\" (UID: \"3c003e7f-5973-4479-88e5-cacb12f92fba\") " Oct 30 17:06:07.889523 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:07.889431 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c003e7f-5973-4479-88e5-cacb12f92fba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c003e7f-5973-4479-88e5-cacb12f92fba-maas-api-openshift-ca\") pod \"3c003e7f-5973-4479-88e5-cacb12f92fba\" (UID: \"3c003e7f-5973-4479-88e5-cacb12f92fba\") " Oct 30 17:06:07.889523 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:07.889480 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c003e7f-5973-4479-88e5-cacb12f92fba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c003e7f-5973-4479-88e5-cacb12f92fba-maas-api-xks-ca\") pod \"3c003e7f-5973-4479-88e5-cacb12f92fba\" (UID: \"3c003e7f-5973-4479-88e5-cacb12f92fba\") " Oct 30 17:06:07.889838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:07.889808 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c003e7f-5973-4479-88e5-cacb12f92fba-maas-api-openshift-ca" pod "3c003e7f-5973-4479-88e5-cacb12f92fba" (UID: "3c003e7f-5973-4479-88e5-cacb12f92fba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:07.890187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:07.890167 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c003e7f-5973-4479-88e5-cacb12f92fba-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:06:07.891731 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:07.891703 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c003e7f-5973-4479-88e5-cacb12f92fba-kube-api-access-tp2jf" pod "3c003e7f-5973-4479-88e5-cacb12f92fba" (UID: "3c003e7f-5973-4479-88e5-cacb12f92fba"). InnerVolumeSpecName "kube-api-access-tp2jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:07.893889 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:07.893865 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c003e7f-5973-4479-88e5-cacb12f92fba-maas-api-xks-ca" pod "3c003e7f-5973-4479-88e5-cacb12f92fba" (UID: "3c003e7f-5973-4479-88e5-cacb12f92fba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:07.991505 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:07.991466 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp2jf\" (UniqueName: \"kubernetes.io/projected/3c003e7f-5973-4479-88e5-cacb12f92fba-kube-api-access-tp2jf\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:06:07.991505 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:07.991508 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c003e7f-5973-4479-88e5-cacb12f92fba-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:06:08.210713 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:08.210622 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:08.620781 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:08.620728 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m" event={"ID":"3c003e7f-5973-4479-88e5-cacb12f92fba","Type":"ContainerDied","Data":"56cc4c3814a3ea3df20b65cfcd400e48866115f497f39dc94dcc672215918e9b"} Oct 30 17:06:08.620781 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:08.620784 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m" Oct 30 17:06:08.621000 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:08.620804 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56cc4c3814a3ea3df20b65cfcd400e48866115f497f39dc94dcc672215918e9b" Oct 30 17:06:10.053959 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:10.053925 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:11.821078 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:11.821041 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:13.043459 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:13.043426 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:13.668233 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:13.668121 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:13.799323 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:13.799288 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:14.072881 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:06:14.072842 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:06:15.554098 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:15.554058 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:17.238610 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:17.238555 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:18.872733 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:18.872689 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:20.633121 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:20.633086 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:22.270673 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:22.270637 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:23.870405 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:23.870363 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:24.115216 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:24.115173 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:24.225931 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:24.225845 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:25.575678 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:25.575641 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:27.370313 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:27.370272 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:27.918179 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:27.918137 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:28.053526 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:28.053490 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:29.799283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:29.799159 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:29.972798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:29.959044 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:30.083893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.083848 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:30.198687 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.198632 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:30.268502 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.268467 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv"] Oct 30 17:06:30.269309 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.269289 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3f7598e1-39ec-436a-a92e-371400e6580f" containerName="payload-pre-processing" Oct 30 17:06:30.269371 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.269312 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7598e1-39ec-436a-a92e-371400e6580f" containerName="payload-pre-processing" Oct 30 17:06:30.269371 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.269360 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3c003e7f-5973-4479-88e5-cacb12f92fba" containerName="revoke-keys" Oct 30 17:06:30.269371 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.269365 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c003e7f-5973-4479-88e5-cacb12f92fba" containerName="revoke-keys" Oct 30 17:06:30.269513 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.269502 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c003e7f-5973-4479-88e5-cacb12f92fba" containerName="revoke-keys" Oct 30 17:06:30.269549 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.269519 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7598e1-39ec-436a-a92e-371400e6580f" containerName="payload-pre-processing" Oct 30 17:06:30.274137 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.274110 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv" Oct 30 17:06:30.360105 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.359993 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-330282\"" Oct 30 17:06:30.360514 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.360481 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-330282-dockercfg-nkcpr\"" Oct 30 17:06:30.364170 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.363154 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5150b64f-5c2e-46bc-9604-8edd2fb63956-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv\" (UID: \"5150b64f-5c2e-46bc-9604-8edd2fb63956\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv" Oct 30 17:06:30.364639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.364593 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fsqs\" (UniqueName: \"kubernetes.io/projected/5150b64f-5c2e-46bc-9604-8edd2fb63956-kube-api-access-4fsqs\") pod \"payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv\" (UID: \"5150b64f-5c2e-46bc-9604-8edd2fb63956\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv" Oct 30 17:06:30.373847 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.373816 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv"] Oct 30 17:06:30.498372 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.488393 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fsqs\" (UniqueName: \"kubernetes.io/projected/5150b64f-5c2e-46bc-9604-8edd2fb63956-kube-api-access-4fsqs\") pod \"payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv\" (UID: \"5150b64f-5c2e-46bc-9604-8edd2fb63956\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv" Oct 30 17:06:30.498372 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.491724 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5150b64f-5c2e-46bc-9604-8edd2fb63956-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv\" (UID: \"5150b64f-5c2e-46bc-9604-8edd2fb63956\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv" Oct 30 17:06:30.498372 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.490318 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5150b64f-5c2e-46bc-9604-8edd2fb63956-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv\" (UID: \"5150b64f-5c2e-46bc-9604-8edd2fb63956\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv" Oct 30 17:06:30.522390 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.515347 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fsqs\" (UniqueName: \"kubernetes.io/projected/5150b64f-5c2e-46bc-9604-8edd2fb63956-kube-api-access-4fsqs\") pod \"payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv\" (UID: \"5150b64f-5c2e-46bc-9604-8edd2fb63956\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv" Oct 30 17:06:30.767848 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:30.763686 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv" Oct 30 17:06:31.090911 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:31.089343 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv"] Oct 30 17:06:31.094990 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:06:31.094956 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5150b64f_5c2e_46bc_9604_8edd2fb63956.slice/crio-32a1f4e94f576e793c4d7ca983c9e8cc06d2279181ce6645262bfd989c65495b WatchSource:0}: Error finding container 32a1f4e94f576e793c4d7ca983c9e8cc06d2279181ce6645262bfd989c65495b: Status 404 returned error can't find the container with id 32a1f4e94f576e793c4d7ca983c9e8cc06d2279181ce6645262bfd989c65495b Oct 30 17:06:31.891931 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:31.891891 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv" event={"ID":"5150b64f-5c2e-46bc-9604-8edd2fb63956","Type":"ContainerStarted","Data":"b3230ffdd4985ac1988236a106e88864e37f661a3079ccba81f2162853007897"} Oct 30 17:06:31.892189 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:31.891961 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv" event={"ID":"5150b64f-5c2e-46bc-9604-8edd2fb63956","Type":"ContainerStarted","Data":"32a1f4e94f576e793c4d7ca983c9e8cc06d2279181ce6645262bfd989c65495b"} Oct 30 17:06:31.892189 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:31.892085 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv" Oct 30 17:06:31.936302 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:31.936267 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:31.941846 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:31.941755 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv" podStartSLOduration=1.941743156 podStartE2EDuration="1.941743156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:31.933175965 +0000 UTC m=+2705.643630708" watchObservedRunningTime="2026-10-30 17:06:31.941743156 +0000 UTC m=+2705.652197950" Oct 30 17:06:33.245752 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:33.245718 6202 scope.go:122] "RemoveContainer" containerID="f8745303692e66388c7ad2329f4eafbf0b57926a713dfcee8622693e29d22d3d" Oct 30 17:06:33.259841 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:33.259819 6202 scope.go:122] "RemoveContainer" containerID="bd56178b1d2c7a15be791712901fc0c208cad25642774662b7554149fb4c99cc" Oct 30 17:06:33.270986 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:33.270958 6202 scope.go:122] "RemoveContainer" containerID="c6c85e7b4f2fe1aa5ebbb97c071987c8f1fcf96f18bb6d92a1d3b223c64268b1" Oct 30 17:06:33.282005 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:33.281982 6202 scope.go:122] "RemoveContainer" containerID="35ee7d286ededb0d47a37b8e4da5d5e4cf82abaa882e7435dfd1fe44b39a1094" Oct 30 17:06:33.292452 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:33.292428 6202 scope.go:122] "RemoveContainer" containerID="61a93ba69069153b22cf72ccc2f01d01190adff8458ad3bad88a9b3ee91a6957" Oct 30 17:06:33.303715 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:33.303692 6202 scope.go:122] "RemoveContainer" containerID="76e7b53a4efcc895111b78e525eba994abce07b29a907b0d634ba17c7cc4e62a" Oct 30 17:06:33.314085 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:33.314050 6202 scope.go:122] "RemoveContainer" containerID="b60c30e7c6cb74346696e303b842e03ea4a81a28cd3e46a68fc8ea411dfd9455" Oct 30 17:06:33.697017 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:33.696986 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:35.350575 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:35.350539 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:37.038375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:37.038336 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:38.377828 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:38.377789 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:38.577050 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:38.571577 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:39.791472 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:39.791434 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:41.493076 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:41.493041 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:41.608258 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:41.608227 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:41.809427 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:41.809318 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:42.030202 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:42.030164 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:42.906857 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:42.906825 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv" Oct 30 17:06:43.192214 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:43.192102 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:44.076595 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:06:44.076557 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:06:44.264312 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:44.264268 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:44.988298 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:44.988265 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:46.601247 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:46.601208 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:48.295829 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:48.295758 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:49.032184 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:49.032144 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm"] Oct 30 17:06:49.046416 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:49.046378 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c02b63-xf7nm"] Oct 30 17:06:50.098096 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:50.098044 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:50.412317 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:50.412237 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a1bf2ff-180e-4b58-bd3e-078ae6e8201b" path="/var/lib/kubelet/pods/4a1bf2ff-180e-4b58-bd3e-078ae6e8201b/volumes" Oct 30 17:06:51.756404 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:51.756363 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:53.393041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:53.393006 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:55.184359 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:55.184322 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:56.881519 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:56.881459 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:56.991522 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:56.991486 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:57.426930 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:57.426879 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:57.772016 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:57.771919 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:57.935967 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:57.935928 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n"] Oct 30 17:06:57.940434 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:57.940407 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:57.969597 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:57.969561 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n"] Oct 30 17:06:58.028406 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.028331 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-830ba9-openshift-default-dockercfg-jp6bn\"" Oct 30 17:06:58.090732 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.090698 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-envoy\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.090938 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.090802 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5m8q\" (UniqueName: \"kubernetes.io/projected/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-kube-api-access-z5m8q\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.090938 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.090892 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istiod-ca-cert\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.091024 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.090938 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-workload-certs\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.091024 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.091016 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-podinfo\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.091172 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.091153 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-credential-socket\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.091305 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.091289 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-token\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.091368 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.091316 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-workload-socket\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.091368 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.091335 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-data\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.192466 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.192414 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-credential-socket\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.192686 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.192542 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-token\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.192686 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.192565 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-workload-socket\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.192686 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.192587 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-data\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.192951 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.192685 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-envoy\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.192951 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.192750 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5m8q\" (UniqueName: \"kubernetes.io/projected/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-kube-api-access-z5m8q\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.192951 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.192803 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istiod-ca-cert\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.192951 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.192842 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-workload-certs\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.192951 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.192911 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-podinfo\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.193245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.192980 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-credential-socket\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.193245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.193033 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-workload-socket\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.193395 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.193346 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-workload-certs\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.193465 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.193422 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-data\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.193691 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.193673 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istiod-ca-cert\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.195593 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.195556 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-podinfo\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.195697 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.195667 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-envoy\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.218697 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.218668 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-token\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.219027 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.219005 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5m8q\" (UniqueName: \"kubernetes.io/projected/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-kube-api-access-z5m8q\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.256733 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.256703 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:58.754279 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.754230 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n"] Oct 30 17:06:58.762062 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:06:58.762029 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b56de5c_cde8_4aeb_91dd_c4f1bb2d2d84.slice/crio-80de409fd00e85e7c16d92a8b3bcbbb46536154bf66ec4df706caf1765364088 WatchSource:0}: Error finding container 80de409fd00e85e7c16d92a8b3bcbbb46536154bf66ec4df706caf1765364088: Status 404 returned error can't find the container with id 80de409fd00e85e7c16d92a8b3bcbbb46536154bf66ec4df706caf1765364088 Oct 30 17:06:58.765083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.765045 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:06:58.765188 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.765123 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:06:58.765188 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.765150 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:06:58.892108 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:58.892075 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:59.200579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.200544 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" event={"ID":"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84","Type":"ContainerStarted","Data":"30af0129587010554123735f084032f73ae8a5333f133ee7db273937ebbe9651"} Oct 30 17:06:59.201001 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.200612 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" event={"ID":"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84","Type":"ContainerStarted","Data":"80de409fd00e85e7c16d92a8b3bcbbb46536154bf66ec4df706caf1765364088"} Oct 30 17:06:59.245325 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.245270 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" podStartSLOduration=2.245256515 podStartE2EDuration="2.245256515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:59.24431429 +0000 UTC m=+2732.954769035" watchObservedRunningTime="2026-10-30 17:06:59.245256515 +0000 UTC m=+2732.955711309" Oct 30 17:06:59.257660 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.257623 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:06:59.259346 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.259313 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Oct 30 17:06:59.259509 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.259380 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" podUID="9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Oct 30 17:06:59.572281 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.572244 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl"] Oct 30 17:06:59.578140 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.578118 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.607044 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.607016 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl"] Oct 30 17:06:59.611534 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.611471 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-bfecdd-openshift-default-dockercfg-2cb9m\"" Oct 30 17:06:59.678822 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.677356 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwk7t\" (UniqueName: \"kubernetes.io/projected/e2315e06-69db-44cc-ad5f-f875d7758205-kube-api-access-pwk7t\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.678822 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.677445 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2315e06-69db-44cc-ad5f-f875d7758205-istio-token\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.678822 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.677493 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-workload-certs\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.678822 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.677528 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-workload-socket\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.678822 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.677562 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-istio-data\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.678822 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.677640 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-credential-socket\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.678822 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.677667 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2315e06-69db-44cc-ad5f-f875d7758205-istio-podinfo\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.678822 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.677885 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2315e06-69db-44cc-ad5f-f875d7758205-istiod-ca-cert\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.678822 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.677932 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-istio-envoy\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.800038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.793828 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2315e06-69db-44cc-ad5f-f875d7758205-istiod-ca-cert\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.800038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.793896 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-istio-envoy\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.800038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.793987 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwk7t\" (UniqueName: \"kubernetes.io/projected/e2315e06-69db-44cc-ad5f-f875d7758205-kube-api-access-pwk7t\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.800038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.794060 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2315e06-69db-44cc-ad5f-f875d7758205-istio-token\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.800038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.794105 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-workload-certs\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.800038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.794137 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-workload-socket\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.800038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.794173 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-istio-data\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.800038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.794250 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-credential-socket\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.800038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.794275 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2315e06-69db-44cc-ad5f-f875d7758205-istio-podinfo\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.800038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.795376 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-workload-socket\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.800038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.795652 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-workload-certs\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.800038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.796001 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-istio-data\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.800038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.796250 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-credential-socket\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.812806 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.811513 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2315e06-69db-44cc-ad5f-f875d7758205-istiod-ca-cert\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.812806 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.812664 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2315e06-69db-44cc-ad5f-f875d7758205-istio-podinfo\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.824631 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.822843 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-istio-envoy\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.854804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.848511 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwk7t\" (UniqueName: \"kubernetes.io/projected/e2315e06-69db-44cc-ad5f-f875d7758205-kube-api-access-pwk7t\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.854804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.849009 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2315e06-69db-44cc-ad5f-f875d7758205-istio-token\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:06:59.918357 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:06:59.917021 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:07:00.316790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:00.314003 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Oct 30 17:07:00.316790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:00.314068 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" podUID="9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Oct 30 17:07:00.792589 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:00.792543 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw"] Oct 30 17:07:00.793482 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:00.793456 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77e226e9-pq9rw"] Oct 30 17:07:01.127172 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:01.127135 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl"] Oct 30 17:07:01.138106 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:01.138057 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:07:01.161389 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:01.158954 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:07:01.161389 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:01.159035 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:07:01.282794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:01.267360 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Oct 30 17:07:01.282794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:01.267424 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" podUID="9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Oct 30 17:07:01.282794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:01.268420 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:01.797930 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:01.797737 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" event={"ID":"e2315e06-69db-44cc-ad5f-f875d7758205","Type":"ContainerStarted","Data":"f901609bbc9425608a465bf3601cff6bbb1347324a8d93ffd4f87befbd6de8d4"} Oct 30 17:07:01.797930 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:01.797819 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" event={"ID":"e2315e06-69db-44cc-ad5f-f875d7758205","Type":"ContainerStarted","Data":"c2f6283d375cdf1bec5a0e350cd6fb3e2db2bd3e4b17a316ac853cef68e1622e"} Oct 30 17:07:01.906897 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:01.906849 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:01.910796 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:01.909310 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" podStartSLOduration=2.909293643 podStartE2EDuration="2.909293643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:01.872707242 +0000 UTC m=+2735.583161987" watchObservedRunningTime="2026-10-30 17:07:01.909293643 +0000 UTC m=+2735.619748443" Oct 30 17:07:01.918742 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:01.918707 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:07:02.129785 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.126576 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:02.263807 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.262293 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Oct 30 17:07:02.263807 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.262364 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" podUID="9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Oct 30 17:07:02.263807 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.263300 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:02.435096 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.435020 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5caf3f97-2914-46bf-a72c-ca024714d5d4" path="/var/lib/kubelet/pods/5caf3f97-2914-46bf-a72c-ca024714d5d4/volumes" Oct 30 17:07:02.479796 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.475187 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd"] Oct 30 17:07:02.524155 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.524119 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd" Oct 30 17:07:02.549793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.538025 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd"] Oct 30 17:07:02.554493 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.554462 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:07:02.554883 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.554862 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:07:02.555090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.555077 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:07:02.555265 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.555252 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:07:02.752599 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.752492 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0c0624f-t7sqd\" (UID: \"a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd" Oct 30 17:07:02.752811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.752663 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8jlz\" (UniqueName: \"kubernetes.io/projected/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-kube-api-access-q8jlz\") pod \"maas-api-revoke-sub-b0c0624f-t7sqd\" (UID: \"a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd" Oct 30 17:07:02.752811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.752738 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0c0624f-t7sqd\" (UID: \"a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd" Oct 30 17:07:02.863797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.859959 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0c0624f-t7sqd\" (UID: \"a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd" Oct 30 17:07:02.863797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.860173 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8jlz\" (UniqueName: \"kubernetes.io/projected/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-kube-api-access-q8jlz\") pod \"maas-api-revoke-sub-b0c0624f-t7sqd\" (UID: \"a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd" Oct 30 17:07:02.863797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.860265 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0c0624f-t7sqd\" (UID: \"a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd" Oct 30 17:07:02.885691 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.885645 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0c0624f-t7sqd\" (UID: \"a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd" Oct 30 17:07:02.926924 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.926888 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 17:07:02.927456 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.927411 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" podUID="e2315e06-69db-44cc-ad5f-f875d7758205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 17:07:02.928407 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.928349 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8jlz\" (UniqueName: \"kubernetes.io/projected/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-kube-api-access-q8jlz\") pod \"maas-api-revoke-sub-b0c0624f-t7sqd\" (UID: \"a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd" Oct 30 17:07:02.975825 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:02.969328 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0c0624f-t7sqd\" (UID: \"a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd" Oct 30 17:07:03.138806 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:03.137227 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd" Oct 30 17:07:03.338504 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:03.338448 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Oct 30 17:07:03.450199 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:03.450100 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" podUID="9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Oct 30 17:07:03.450969 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:03.365558 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:03.575959 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:03.575898 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 17:07:03.710855 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:03.708894 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:03.931019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:03.930974 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 17:07:03.931600 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:03.931567 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" podUID="e2315e06-69db-44cc-ad5f-f875d7758205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 17:07:04.018861 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:04.015125 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f"] Oct 30 17:07:04.123245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:04.123200 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:04.124153 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:04.124118 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:04.164793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:04.163911 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" podUID="e660a917-1db1-4c82-b938-92f1ba909939" containerName="main" probeResult="failure" output="Get \"https://10.133.1.20:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:04.275850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:04.274897 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Oct 30 17:07:04.275850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:04.274965 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" podUID="9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Oct 30 17:07:04.342793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:04.336523 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:04.426892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:04.426829 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f" Oct 30 17:07:04.475991 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:04.475894 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:04.547872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:04.535980 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0efcbc66-pzn2f\" (UID: \"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f" Oct 30 17:07:04.547872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:04.536325 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnxjw\" (UniqueName: \"kubernetes.io/projected/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-kube-api-access-nnxjw\") pod \"maas-api-revoke-sub-0efcbc66-pzn2f\" (UID: \"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f" Oct 30 17:07:04.547872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:04.536378 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0efcbc66-pzn2f\" (UID: \"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f" Oct 30 17:07:04.742002 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:04.741964 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0efcbc66-pzn2f\" (UID: \"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f" Oct 30 17:07:04.742642 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:04.742615 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnxjw\" (UniqueName: \"kubernetes.io/projected/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-kube-api-access-nnxjw\") pod \"maas-api-revoke-sub-0efcbc66-pzn2f\" (UID: \"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f" Oct 30 17:07:04.742898 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:04.742864 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0efcbc66-pzn2f\" (UID: \"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f" Oct 30 17:07:04.744143 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:04.744121 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0efcbc66-pzn2f\" (UID: \"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f" Oct 30 17:07:04.913511 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:04.913478 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnxjw\" (UniqueName: \"kubernetes.io/projected/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-kube-api-access-nnxjw\") pod \"maas-api-revoke-sub-0efcbc66-pzn2f\" (UID: \"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f" Oct 30 17:07:05.146545 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:04.996592 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0efcbc66-pzn2f\" (UID: \"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f" Oct 30 17:07:05.307811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.160835 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:05.307811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.168560 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 17:07:05.307811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.168613 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" podUID="e2315e06-69db-44cc-ad5f-f875d7758205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 17:07:05.307811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.173044 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" podUID="e660a917-1db1-4c82-b938-92f1ba909939" containerName="main" probeResult="failure" output="Get \"https://10.133.1.20:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:05.307811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.234858 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" podUID="9702f62c-b2ea-47b1-9e67-d50be44e166e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:05.307811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.256222 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 17:07:05.307811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.256271 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f"] Oct 30 17:07:05.307811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.256383 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd"] Oct 30 17:07:05.307811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.256435 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:07:05.307811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.256471 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 17:07:05.316837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.316122 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Oct 30 17:07:05.316837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.316176 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" podUID="9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Oct 30 17:07:05.320228 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.320201 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f" Oct 30 17:07:05.322481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.322456 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 17:07:05.385515 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.385482 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:05.617240 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.608930 6202 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:05.637334 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.637270 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:05.694794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.682143 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:05.747787 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.745938 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd" event={"ID":"a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43","Type":"ContainerStarted","Data":"6b571fd6e7249348d1083eb1db809b25cc5756cd88ad0fb35ca9a6e455e20c0b"} Oct 30 17:07:05.997965 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:05.997931 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 17:07:06.030866 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:06.029881 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" podUID="e2315e06-69db-44cc-ad5f-f875d7758205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 17:07:06.033147 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:06.032194 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:07:06.074016 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:06.073962 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:06.093827 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:06.078610 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8" Oct 30 17:07:06.317850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:06.311296 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Oct 30 17:07:06.317850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:06.311358 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" podUID="9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Oct 30 17:07:06.940858 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:06.935180 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 17:07:06.940858 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:06.935241 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" podUID="e2315e06-69db-44cc-ad5f-f875d7758205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 17:07:07.136245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:07.136187 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f"] Oct 30 17:07:07.199531 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:07.199492 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:07.468841 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:07.466249 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:07:07.468841 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:07.467953 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:07:07.479112 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:07.479035 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:07:07.919454 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:07.919423 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 17:07:07.919607 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:07.919480 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" podUID="e2315e06-69db-44cc-ad5f-f875d7758205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 17:07:08.092033 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:08.091998 6202 generic.go:356] "Generic (PLEG): container finished" podID="a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43" containerID="f86aca6678efaeb8feb4bb0123c468d0260294319d089a528cafb3c3aba72f70" exitCode=0 Oct 30 17:07:08.092275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:08.092259 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd" event={"ID":"a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43","Type":"ContainerDied","Data":"f86aca6678efaeb8feb4bb0123c468d0260294319d089a528cafb3c3aba72f70"} Oct 30 17:07:08.094889 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:08.094862 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f" event={"ID":"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8","Type":"ContainerStarted","Data":"2788e43573bdca91bb67fa73866e64780158ff7960de77aada5bc321cc0db56a"} Oct 30 17:07:08.095035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:08.094899 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f" event={"ID":"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8","Type":"ContainerStarted","Data":"f07872d11a1f63f747e245e9adf4b8ba960c8cd52f7e23885deb84d558633133"} Oct 30 17:07:08.186395 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:08.186205 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f" podStartSLOduration=5.186186955 podStartE2EDuration="5.186186955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:08.184340774 +0000 UTC m=+2741.894795519" watchObservedRunningTime="2026-10-30 17:07:08.186186955 +0000 UTC m=+2741.896641699" Oct 30 17:07:08.837374 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:08.837337 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:09.130579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.130482 6202 generic.go:356] "Generic (PLEG): container finished" podID="5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8" containerID="2788e43573bdca91bb67fa73866e64780158ff7960de77aada5bc321cc0db56a" exitCode=0 Oct 30 17:07:09.132279 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.132239 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f" event={"ID":"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8","Type":"ContainerDied","Data":"2788e43573bdca91bb67fa73866e64780158ff7960de77aada5bc321cc0db56a"} Oct 30 17:07:09.194569 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.194529 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:07:09.195147 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.195130 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:07:09.196012 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.195989 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:07:09.382174 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.382096 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd" Oct 30 17:07:09.414824 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.414788 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-kube-api-access-q8jlz\" (UniqueName: \"kubernetes.io/projected/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-kube-api-access-q8jlz\") pod \"a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43\" (UID: \"a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43\") " Oct 30 17:07:09.415001 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.414894 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-maas-api-openshift-ca\") pod \"a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43\" (UID: \"a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43\") " Oct 30 17:07:09.415165 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.415147 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-maas-api-xks-ca\") pod \"a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43\" (UID: \"a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43\") " Oct 30 17:07:09.416207 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.416172 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-maas-api-openshift-ca" pod "a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43" (UID: "a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:09.419229 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.419196 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-maas-api-xks-ca" pod "a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43" (UID: "a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:07:09.419350 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.419324 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-kube-api-access-q8jlz" pod "a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43" (UID: "a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43"). InnerVolumeSpecName "kube-api-access-q8jlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:09.485847 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.481756 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:09.518241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.518196 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:09.518241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.518242 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8jlz\" (UniqueName: \"kubernetes.io/projected/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-kube-api-access-q8jlz\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:09.518241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.518258 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:09.950380 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.950340 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm"] Oct 30 17:07:09.951448 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.951421 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43" containerName="revoke-keys" Oct 30 17:07:09.951559 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.951449 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43" containerName="revoke-keys" Oct 30 17:07:09.951633 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.951621 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43" containerName="revoke-keys" Oct 30 17:07:09.955807 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.955759 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm" Oct 30 17:07:09.977449 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:09.977416 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm"] Oct 30 17:07:10.029372 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.029318 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm\" (UID: \"0ba53a4d-10d1-44e6-8d75-c0f77e50e462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm" Oct 30 17:07:10.029559 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.029534 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7jm4\" (UniqueName: \"kubernetes.io/projected/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-kube-api-access-l7jm4\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm\" (UID: \"0ba53a4d-10d1-44e6-8d75-c0f77e50e462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm" Oct 30 17:07:10.029673 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.029645 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm\" (UID: \"0ba53a4d-10d1-44e6-8d75-c0f77e50e462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm" Oct 30 17:07:10.136792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.132025 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm\" (UID: \"0ba53a4d-10d1-44e6-8d75-c0f77e50e462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm" Oct 30 17:07:10.136792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.132163 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7jm4\" (UniqueName: \"kubernetes.io/projected/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-kube-api-access-l7jm4\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm\" (UID: \"0ba53a4d-10d1-44e6-8d75-c0f77e50e462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm" Oct 30 17:07:10.136792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.132208 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm\" (UID: \"0ba53a4d-10d1-44e6-8d75-c0f77e50e462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm" Oct 30 17:07:10.136792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.133147 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm\" (UID: \"0ba53a4d-10d1-44e6-8d75-c0f77e50e462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm" Oct 30 17:07:10.136792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.136232 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm\" (UID: \"0ba53a4d-10d1-44e6-8d75-c0f77e50e462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm" Oct 30 17:07:10.146545 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.146508 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd" Oct 30 17:07:10.147872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.147838 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd" event={"ID":"a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43","Type":"ContainerDied","Data":"6b571fd6e7249348d1083eb1db809b25cc5756cd88ad0fb35ca9a6e455e20c0b"} Oct 30 17:07:10.148035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.147910 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b571fd6e7249348d1083eb1db809b25cc5756cd88ad0fb35ca9a6e455e20c0b" Oct 30 17:07:10.163504 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.162811 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7jm4\" (UniqueName: \"kubernetes.io/projected/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-kube-api-access-l7jm4\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm\" (UID: \"0ba53a4d-10d1-44e6-8d75-c0f77e50e462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm" Oct 30 17:07:10.279343 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.279067 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm" Oct 30 17:07:10.393858 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.393598 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f" Oct 30 17:07:10.456904 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.439808 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-maas-api-xks-ca\") pod \"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8\" (UID: \"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8\") " Oct 30 17:07:10.456904 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.439891 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-kube-api-access-nnxjw\" (UniqueName: \"kubernetes.io/projected/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-kube-api-access-nnxjw\") pod \"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8\" (UID: \"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8\") " Oct 30 17:07:10.456904 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.440099 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-maas-api-openshift-ca\") pod \"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8\" (UID: \"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8\") " Oct 30 17:07:10.456904 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.441154 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-maas-api-openshift-ca" pod "5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8" (UID: "5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:10.456904 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.445514 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-maas-api-xks-ca" pod "5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8" (UID: "5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:07:10.456904 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.451951 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-kube-api-access-nnxjw" pod "5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8" (UID: "5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8"). InnerVolumeSpecName "kube-api-access-nnxjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:10.542485 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.542392 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.542485 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.542427 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.542485 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.542440 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnxjw\" (UniqueName: \"kubernetes.io/projected/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8-kube-api-access-nnxjw\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.604158 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.604115 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm"] Oct 30 17:07:10.604348 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:07:10.604121 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba53a4d_10d1_44e6_8d75_c0f77e50e462.slice/crio-0f855a1665041579c5b17a8fce86d60201c61bf4172c9fe1d178b4d2c5e33540 WatchSource:0}: Error finding container 0f855a1665041579c5b17a8fce86d60201c61bf4172c9fe1d178b4d2c5e33540: Status 404 returned error can't find the container with id 0f855a1665041579c5b17a8fce86d60201c61bf4172c9fe1d178b4d2c5e33540 Oct 30 17:07:10.657800 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:10.657372 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:11.163948 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:11.163920 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f" Oct 30 17:07:11.164416 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:11.163943 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f" event={"ID":"5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8","Type":"ContainerDied","Data":"f07872d11a1f63f747e245e9adf4b8ba960c8cd52f7e23885deb84d558633133"} Oct 30 17:07:11.164416 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:11.164005 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f07872d11a1f63f747e245e9adf4b8ba960c8cd52f7e23885deb84d558633133" Oct 30 17:07:11.166481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:11.166450 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm" event={"ID":"0ba53a4d-10d1-44e6-8d75-c0f77e50e462","Type":"ContainerStarted","Data":"0f855a1665041579c5b17a8fce86d60201c61bf4172c9fe1d178b4d2c5e33540"} Oct 30 17:07:11.207159 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:11.207084 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm" podStartSLOduration=2.207065734 podStartE2EDuration="2.207065734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:11.205341169 +0000 UTC m=+2744.915795912" watchObservedRunningTime="2026-10-30 17:07:11.207065734 +0000 UTC m=+2744.917520478" Oct 30 17:07:12.180236 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.180192 6202 generic.go:356] "Generic (PLEG): container finished" podID="0ba53a4d-10d1-44e6-8d75-c0f77e50e462" containerID="b6e844c4fdae53503460b0bbd6b81baa9caee85b8d6dee1d355b75b50f26e4a0" exitCode=0 Oct 30 17:07:12.180656 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.180289 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm" event={"ID":"0ba53a4d-10d1-44e6-8d75-c0f77e50e462","Type":"ContainerDied","Data":"b6e844c4fdae53503460b0bbd6b81baa9caee85b8d6dee1d355b75b50f26e4a0"} Oct 30 17:07:12.459332 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.459249 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:12.575171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.575128 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:12.688797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.687359 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:12.834977 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.834941 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2"] Oct 30 17:07:12.835819 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.835800 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8" containerName="revoke-keys" Oct 30 17:07:12.835892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.835821 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8" containerName="revoke-keys" Oct 30 17:07:12.835954 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.835944 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8" containerName="revoke-keys" Oct 30 17:07:12.838988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.838967 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2" Oct 30 17:07:12.867869 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.867834 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2"] Oct 30 17:07:12.891997 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.891962 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eaa54db-afe1-4a37-ba9f-1e0745826d93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a7c2cee-xdds2\" (UID: \"3eaa54db-afe1-4a37-ba9f-1e0745826d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2" Oct 30 17:07:12.892249 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.892225 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nnpp\" (UniqueName: \"kubernetes.io/projected/3eaa54db-afe1-4a37-ba9f-1e0745826d93-kube-api-access-9nnpp\") pod \"maas-api-revoke-sub-9a7c2cee-xdds2\" (UID: \"3eaa54db-afe1-4a37-ba9f-1e0745826d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2" Oct 30 17:07:12.892320 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.892270 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eaa54db-afe1-4a37-ba9f-1e0745826d93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a7c2cee-xdds2\" (UID: \"3eaa54db-afe1-4a37-ba9f-1e0745826d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2" Oct 30 17:07:12.994378 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.994344 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eaa54db-afe1-4a37-ba9f-1e0745826d93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a7c2cee-xdds2\" (UID: \"3eaa54db-afe1-4a37-ba9f-1e0745826d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2" Oct 30 17:07:12.994601 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.994504 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nnpp\" (UniqueName: \"kubernetes.io/projected/3eaa54db-afe1-4a37-ba9f-1e0745826d93-kube-api-access-9nnpp\") pod \"maas-api-revoke-sub-9a7c2cee-xdds2\" (UID: \"3eaa54db-afe1-4a37-ba9f-1e0745826d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2" Oct 30 17:07:12.994601 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.994526 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eaa54db-afe1-4a37-ba9f-1e0745826d93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a7c2cee-xdds2\" (UID: \"3eaa54db-afe1-4a37-ba9f-1e0745826d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2" Oct 30 17:07:12.995281 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.995263 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eaa54db-afe1-4a37-ba9f-1e0745826d93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a7c2cee-xdds2\" (UID: \"3eaa54db-afe1-4a37-ba9f-1e0745826d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2" Oct 30 17:07:12.997020 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:12.996997 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eaa54db-afe1-4a37-ba9f-1e0745826d93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a7c2cee-xdds2\" (UID: \"3eaa54db-afe1-4a37-ba9f-1e0745826d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2" Oct 30 17:07:13.017653 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.017622 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nnpp\" (UniqueName: \"kubernetes.io/projected/3eaa54db-afe1-4a37-ba9f-1e0745826d93-kube-api-access-9nnpp\") pod \"maas-api-revoke-sub-9a7c2cee-xdds2\" (UID: \"3eaa54db-afe1-4a37-ba9f-1e0745826d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2" Oct 30 17:07:13.153627 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.153534 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2" Oct 30 17:07:13.202869 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.202837 6202 generic.go:356] "Generic (PLEG): container finished" podID="6d43f09c-faf8-49d7-8c86-0cfd033cff9d" containerID="5e54e4c068085e8c968eacc72829d9c0169a8d4335b73138062df9664f8148ae" exitCode=0 Oct 30 17:07:13.203260 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.203180 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" event={"ID":"6d43f09c-faf8-49d7-8c86-0cfd033cff9d","Type":"ContainerDied","Data":"5e54e4c068085e8c968eacc72829d9c0169a8d4335b73138062df9664f8148ae"} Oct 30 17:07:13.203805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.203563 6202 scope.go:122] "RemoveContainer" containerID="5e54e4c068085e8c968eacc72829d9c0169a8d4335b73138062df9664f8148ae" Oct 30 17:07:13.206235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.206038 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:07:13.206235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.206142 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:07:13.206235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.206196 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:07:13.490552 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.490408 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2"] Oct 30 17:07:13.505257 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:07:13.505207 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eaa54db_afe1_4a37_ba9f_1e0745826d93.slice/crio-1e16c4456e329526158d61b76e7b949b2a2438f2654f74fbd3b08e4459b78ee4 WatchSource:0}: Error finding container 1e16c4456e329526158d61b76e7b949b2a2438f2654f74fbd3b08e4459b78ee4: Status 404 returned error can't find the container with id 1e16c4456e329526158d61b76e7b949b2a2438f2654f74fbd3b08e4459b78ee4 Oct 30 17:07:13.570901 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.570852 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm" Oct 30 17:07:13.657790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.657020 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-maas-api-xks-ca\") pod \"0ba53a4d-10d1-44e6-8d75-c0f77e50e462\" (UID: \"0ba53a4d-10d1-44e6-8d75-c0f77e50e462\") " Oct 30 17:07:13.657790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.657139 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-kube-api-access-l7jm4\" (UniqueName: \"kubernetes.io/projected/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-kube-api-access-l7jm4\") pod \"0ba53a4d-10d1-44e6-8d75-c0f77e50e462\" (UID: \"0ba53a4d-10d1-44e6-8d75-c0f77e50e462\") " Oct 30 17:07:13.657790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.657258 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-maas-api-openshift-ca\") pod \"0ba53a4d-10d1-44e6-8d75-c0f77e50e462\" (UID: \"0ba53a4d-10d1-44e6-8d75-c0f77e50e462\") " Oct 30 17:07:13.659813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.658599 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-maas-api-openshift-ca" pod "0ba53a4d-10d1-44e6-8d75-c0f77e50e462" (UID: "0ba53a4d-10d1-44e6-8d75-c0f77e50e462"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:13.659813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.658961 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:13.664743 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.664709 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-maas-api-xks-ca" pod "0ba53a4d-10d1-44e6-8d75-c0f77e50e462" (UID: "0ba53a4d-10d1-44e6-8d75-c0f77e50e462"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:07:13.665010 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.664982 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-kube-api-access-l7jm4" pod "0ba53a4d-10d1-44e6-8d75-c0f77e50e462" (UID: "0ba53a4d-10d1-44e6-8d75-c0f77e50e462"). InnerVolumeSpecName "kube-api-access-l7jm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:13.782851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.764285 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:13.782851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:13.764329 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7jm4\" (UniqueName: \"kubernetes.io/projected/0ba53a4d-10d1-44e6-8d75-c0f77e50e462-kube-api-access-l7jm4\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:14.084365 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:07:14.083667 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:07:14.215184 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.215155 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm" Oct 30 17:07:14.215691 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.215190 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm" event={"ID":"0ba53a4d-10d1-44e6-8d75-c0f77e50e462","Type":"ContainerDied","Data":"0f855a1665041579c5b17a8fce86d60201c61bf4172c9fe1d178b4d2c5e33540"} Oct 30 17:07:14.215691 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.215237 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f855a1665041579c5b17a8fce86d60201c61bf4172c9fe1d178b4d2c5e33540" Oct 30 17:07:14.217721 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.217694 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" event={"ID":"6d43f09c-faf8-49d7-8c86-0cfd033cff9d","Type":"ContainerStarted","Data":"ace4c0667da616ff9ae2e597cad10f3c3bc37c6f062ebb8796fc8b930bff50db"} Oct 30 17:07:14.221354 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.221330 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2" event={"ID":"3eaa54db-afe1-4a37-ba9f-1e0745826d93","Type":"ContainerStarted","Data":"74fba84a0f56ddab2a8b49b23bdbe8379b4b37f0ca59517373a23c21d1a795f9"} Oct 30 17:07:14.221489 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.221371 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2" event={"ID":"3eaa54db-afe1-4a37-ba9f-1e0745826d93","Type":"ContainerStarted","Data":"1e16c4456e329526158d61b76e7b949b2a2438f2654f74fbd3b08e4459b78ee4"} Oct 30 17:07:14.298546 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.298430 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:14.323572 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.323516 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2" podStartSLOduration=2.323499143 podStartE2EDuration="2.323499143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:14.319518035 +0000 UTC m=+2748.029972781" watchObservedRunningTime="2026-10-30 17:07:14.323499143 +0000 UTC m=+2748.033953895" Oct 30 17:07:14.503494 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.501870 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k"] Oct 30 17:07:14.503709 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.503682 6202 cpu_manager.go:414] "containerMap: removing container" podUID="0ba53a4d-10d1-44e6-8d75-c0f77e50e462" containerName="revoke-keys" Oct 30 17:07:14.503709 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.503702 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ba53a4d-10d1-44e6-8d75-c0f77e50e462" containerName="revoke-keys" Oct 30 17:07:14.505032 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.504979 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ba53a4d-10d1-44e6-8d75-c0f77e50e462" containerName="revoke-keys" Oct 30 17:07:14.508386 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.508364 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k"] Oct 30 17:07:14.508515 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.508462 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k" Oct 30 17:07:14.614152 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.614114 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:07:14.615867 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.615838 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Oct 30 17:07:14.616009 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.615895 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" podUID="6d43f09c-faf8-49d7-8c86-0cfd033cff9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Oct 30 17:07:14.616063 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.616021 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4a7f8be-zw49k\" (UID: \"721fc1d7-6ad4-4f60-ae9f-6503de4d0331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k" Oct 30 17:07:14.616114 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.616059 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdmgb\" (UniqueName: \"kubernetes.io/projected/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-kube-api-access-tdmgb\") pod \"maas-api-revoke-sub-d4a7f8be-zw49k\" (UID: \"721fc1d7-6ad4-4f60-ae9f-6503de4d0331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k" Oct 30 17:07:14.616212 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.616196 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4a7f8be-zw49k\" (UID: \"721fc1d7-6ad4-4f60-ae9f-6503de4d0331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k" Oct 30 17:07:14.717894 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.717857 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4a7f8be-zw49k\" (UID: \"721fc1d7-6ad4-4f60-ae9f-6503de4d0331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k" Oct 30 17:07:14.717894 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.717901 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdmgb\" (UniqueName: \"kubernetes.io/projected/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-kube-api-access-tdmgb\") pod \"maas-api-revoke-sub-d4a7f8be-zw49k\" (UID: \"721fc1d7-6ad4-4f60-ae9f-6503de4d0331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k" Oct 30 17:07:14.718189 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.717963 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4a7f8be-zw49k\" (UID: \"721fc1d7-6ad4-4f60-ae9f-6503de4d0331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k" Oct 30 17:07:14.718597 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.718569 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4a7f8be-zw49k\" (UID: \"721fc1d7-6ad4-4f60-ae9f-6503de4d0331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k" Oct 30 17:07:14.720584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.720562 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4a7f8be-zw49k\" (UID: \"721fc1d7-6ad4-4f60-ae9f-6503de4d0331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k" Oct 30 17:07:14.777790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.777019 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdmgb\" (UniqueName: \"kubernetes.io/projected/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-kube-api-access-tdmgb\") pod \"maas-api-revoke-sub-d4a7f8be-zw49k\" (UID: \"721fc1d7-6ad4-4f60-ae9f-6503de4d0331\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k" Oct 30 17:07:14.824849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:14.824809 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k" Oct 30 17:07:15.177994 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:15.177957 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k"] Oct 30 17:07:15.179121 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:07:15.179094 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721fc1d7_6ad4_4f60_ae9f_6503de4d0331.slice/crio-f581068773dc353a0a2b5f0103c78609b708c03ad264cf71380cf4cd36938064 WatchSource:0}: Error finding container f581068773dc353a0a2b5f0103c78609b708c03ad264cf71380cf4cd36938064: Status 404 returned error can't find the container with id f581068773dc353a0a2b5f0103c78609b708c03ad264cf71380cf4cd36938064 Oct 30 17:07:15.230952 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:15.230911 6202 generic.go:356] "Generic (PLEG): container finished" podID="3eaa54db-afe1-4a37-ba9f-1e0745826d93" containerID="74fba84a0f56ddab2a8b49b23bdbe8379b4b37f0ca59517373a23c21d1a795f9" exitCode=0 Oct 30 17:07:15.231383 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:15.230994 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2" event={"ID":"3eaa54db-afe1-4a37-ba9f-1e0745826d93","Type":"ContainerDied","Data":"74fba84a0f56ddab2a8b49b23bdbe8379b4b37f0ca59517373a23c21d1a795f9"} Oct 30 17:07:15.232544 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:15.232521 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k" event={"ID":"721fc1d7-6ad4-4f60-ae9f-6503de4d0331","Type":"ContainerStarted","Data":"f581068773dc353a0a2b5f0103c78609b708c03ad264cf71380cf4cd36938064"} Oct 30 17:07:15.614317 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:15.614282 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Oct 30 17:07:15.614499 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:15.614348 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" podUID="6d43f09c-faf8-49d7-8c86-0cfd033cff9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Oct 30 17:07:16.082110 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:16.082071 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:16.279096 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:16.279053 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k" event={"ID":"721fc1d7-6ad4-4f60-ae9f-6503de4d0331","Type":"ContainerStarted","Data":"7737e3c9db9e6c4136fe4fa3ac279eb48ccc30e09896e2862f7d1d9d5496bd6f"} Oct 30 17:07:16.321081 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:16.321028 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k" podStartSLOduration=2.321010563 podStartE2EDuration="2.321010563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:16.319194398 +0000 UTC m=+2750.029649141" watchObservedRunningTime="2026-10-30 17:07:16.321010563 +0000 UTC m=+2750.031465303" Oct 30 17:07:16.529956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:16.529930 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2" Oct 30 17:07:16.614489 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:16.614447 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Oct 30 17:07:16.614651 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:16.614519 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" podUID="6d43f09c-faf8-49d7-8c86-0cfd033cff9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Oct 30 17:07:16.683788 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:16.683679 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eaa54db-afe1-4a37-ba9f-1e0745826d93-kube-api-access-9nnpp\" (UniqueName: \"kubernetes.io/projected/3eaa54db-afe1-4a37-ba9f-1e0745826d93-kube-api-access-9nnpp\") pod \"3eaa54db-afe1-4a37-ba9f-1e0745826d93\" (UID: \"3eaa54db-afe1-4a37-ba9f-1e0745826d93\") " Oct 30 17:07:16.683944 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:16.683868 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eaa54db-afe1-4a37-ba9f-1e0745826d93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eaa54db-afe1-4a37-ba9f-1e0745826d93-maas-api-openshift-ca\") pod \"3eaa54db-afe1-4a37-ba9f-1e0745826d93\" (UID: \"3eaa54db-afe1-4a37-ba9f-1e0745826d93\") " Oct 30 17:07:16.684002 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:16.683965 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eaa54db-afe1-4a37-ba9f-1e0745826d93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eaa54db-afe1-4a37-ba9f-1e0745826d93-maas-api-xks-ca\") pod \"3eaa54db-afe1-4a37-ba9f-1e0745826d93\" (UID: \"3eaa54db-afe1-4a37-ba9f-1e0745826d93\") " Oct 30 17:07:16.685583 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:16.685551 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eaa54db-afe1-4a37-ba9f-1e0745826d93-maas-api-openshift-ca" pod "3eaa54db-afe1-4a37-ba9f-1e0745826d93" (UID: "3eaa54db-afe1-4a37-ba9f-1e0745826d93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:16.687900 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:16.687861 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eaa54db-afe1-4a37-ba9f-1e0745826d93-maas-api-xks-ca" pod "3eaa54db-afe1-4a37-ba9f-1e0745826d93" (UID: "3eaa54db-afe1-4a37-ba9f-1e0745826d93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:07:16.688432 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:16.688395 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eaa54db-afe1-4a37-ba9f-1e0745826d93-kube-api-access-9nnpp" pod "3eaa54db-afe1-4a37-ba9f-1e0745826d93" (UID: "3eaa54db-afe1-4a37-ba9f-1e0745826d93"). InnerVolumeSpecName "kube-api-access-9nnpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:16.786479 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:16.786437 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eaa54db-afe1-4a37-ba9f-1e0745826d93-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:16.786479 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:16.786473 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eaa54db-afe1-4a37-ba9f-1e0745826d93-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:16.786479 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:16.786485 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nnpp\" (UniqueName: \"kubernetes.io/projected/3eaa54db-afe1-4a37-ba9f-1e0745826d93-kube-api-access-9nnpp\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:17.302290 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:17.302251 6202 generic.go:356] "Generic (PLEG): container finished" podID="721fc1d7-6ad4-4f60-ae9f-6503de4d0331" containerID="7737e3c9db9e6c4136fe4fa3ac279eb48ccc30e09896e2862f7d1d9d5496bd6f" exitCode=0 Oct 30 17:07:17.302804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:17.302336 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k" event={"ID":"721fc1d7-6ad4-4f60-ae9f-6503de4d0331","Type":"ContainerDied","Data":"7737e3c9db9e6c4136fe4fa3ac279eb48ccc30e09896e2862f7d1d9d5496bd6f"} Oct 30 17:07:17.304017 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:17.303993 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2" event={"ID":"3eaa54db-afe1-4a37-ba9f-1e0745826d93","Type":"ContainerDied","Data":"1e16c4456e329526158d61b76e7b949b2a2438f2654f74fbd3b08e4459b78ee4"} Oct 30 17:07:17.304131 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:17.304023 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2" Oct 30 17:07:17.304131 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:17.304035 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e16c4456e329526158d61b76e7b949b2a2438f2654f74fbd3b08e4459b78ee4" Oct 30 17:07:17.630793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:17.630208 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Oct 30 17:07:17.630793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:17.630283 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" podUID="6d43f09c-faf8-49d7-8c86-0cfd033cff9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Oct 30 17:07:17.786791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:17.786590 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:18.582934 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:18.582912 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k" Oct 30 17:07:18.614584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:18.614553 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Oct 30 17:07:18.614893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:18.614867 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" podUID="6d43f09c-faf8-49d7-8c86-0cfd033cff9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Oct 30 17:07:18.781322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:18.781285 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-kube-api-access-tdmgb\" (UniqueName: \"kubernetes.io/projected/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-kube-api-access-tdmgb\") pod \"721fc1d7-6ad4-4f60-ae9f-6503de4d0331\" (UID: \"721fc1d7-6ad4-4f60-ae9f-6503de4d0331\") " Oct 30 17:07:18.781535 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:18.781490 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-maas-api-openshift-ca\") pod \"721fc1d7-6ad4-4f60-ae9f-6503de4d0331\" (UID: \"721fc1d7-6ad4-4f60-ae9f-6503de4d0331\") " Oct 30 17:07:18.781629 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:18.781609 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-maas-api-xks-ca\") pod \"721fc1d7-6ad4-4f60-ae9f-6503de4d0331\" (UID: \"721fc1d7-6ad4-4f60-ae9f-6503de4d0331\") " Oct 30 17:07:18.781936 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:18.781908 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-maas-api-openshift-ca" pod "721fc1d7-6ad4-4f60-ae9f-6503de4d0331" (UID: "721fc1d7-6ad4-4f60-ae9f-6503de4d0331"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:18.782320 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:18.782289 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:18.783578 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:18.783551 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-kube-api-access-tdmgb" pod "721fc1d7-6ad4-4f60-ae9f-6503de4d0331" (UID: "721fc1d7-6ad4-4f60-ae9f-6503de4d0331"). InnerVolumeSpecName "kube-api-access-tdmgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:18.784559 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:18.784521 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-maas-api-xks-ca" pod "721fc1d7-6ad4-4f60-ae9f-6503de4d0331" (UID: "721fc1d7-6ad4-4f60-ae9f-6503de4d0331"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:07:18.884097 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:18.883994 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:18.884097 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:18.884047 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdmgb\" (UniqueName: \"kubernetes.io/projected/721fc1d7-6ad4-4f60-ae9f-6503de4d0331-kube-api-access-tdmgb\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:19.323611 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:19.323580 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k" Oct 30 17:07:19.323825 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:19.323617 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k" event={"ID":"721fc1d7-6ad4-4f60-ae9f-6503de4d0331","Type":"ContainerDied","Data":"f581068773dc353a0a2b5f0103c78609b708c03ad264cf71380cf4cd36938064"} Oct 30 17:07:19.323825 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:19.323677 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f581068773dc353a0a2b5f0103c78609b708c03ad264cf71380cf4cd36938064" Oct 30 17:07:19.336352 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:19.336318 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:07:19.579048 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:19.577402 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:19.618399 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:19.615468 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Oct 30 17:07:19.618399 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:19.615551 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" podUID="6d43f09c-faf8-49d7-8c86-0cfd033cff9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Oct 30 17:07:20.343649 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:20.343618 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t"] Oct 30 17:07:20.480323 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:20.478859 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:20.595166 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:20.592030 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:20.654442 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:20.648398 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:07:20.654442 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:20.651375 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:07:21.643185 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:21.643149 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:21.754175 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:21.754132 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:21.910279 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:21.910199 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:23.409613 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:23.409573 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:25.343956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:25.343914 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" podUID="734b4608-cecf-48ad-961e-31b7a2a22871" containerName="payload-processing" containerID="cri-o://de2550337d35f9b2a850dcc2e3815c2d3cb0604e22bed49e2e07b0a5339b460d" gracePeriod=25 Oct 30 17:07:25.653258 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:25.653169 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:27.259183 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:27.259137 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:27.499387 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:27.499346 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:29.259512 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:29.259472 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:30.858524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:30.858488 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:30.972319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:30.972270 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:31.857665 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:31.857622 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" podUID="734b4608-cecf-48ad-961e-31b7a2a22871" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:07:32.794883 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:32.794842 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:33.388832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:33.388800 6202 scope.go:122] "RemoveContainer" containerID="dd6f18579b0f3ef11dd17eb784c5d22967b10fe2748712728b2e3ee341c14901" Oct 30 17:07:33.399984 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:33.399957 6202 scope.go:122] "RemoveContainer" containerID="6248c45be4c073d36828a7d3be3db72d0d33d04c33daa762a1bccd59694ac3aa" Oct 30 17:07:33.410174 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:33.410154 6202 scope.go:122] "RemoveContainer" containerID="9fae3a4175cb1aef23e97825b8fee2350ecad498c198c5848215a26094794671" Oct 30 17:07:34.475298 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:34.475259 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:36.165471 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:36.165429 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:37.979054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:37.979016 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:38.418392 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:38.418359 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9"] Oct 30 17:07:38.454706 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:38.454670 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:39.064168 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:39.064117 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:39.226272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:39.226239 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:39.343675 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:39.343629 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:39.628485 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:39.628374 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:40.363911 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:40.363855 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:41.686502 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:41.686467 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:41.796078 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:41.796042 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:41.858620 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:41.858581 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" podUID="734b4608-cecf-48ad-961e-31b7a2a22871" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:07:43.081227 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:43.081191 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:43.597444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:43.597404 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:43.720077 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:43.720035 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:43.828837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:43.828800 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:44.075620 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:07:44.075588 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:07:45.282689 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:45.282647 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:45.403000 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:45.402969 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:45.522078 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:45.522038 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:45.921867 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:45.921827 6202 generic.go:356] "Generic (PLEG): container finished" podID="734b4608-cecf-48ad-961e-31b7a2a22871" containerID="de2550337d35f9b2a850dcc2e3815c2d3cb0604e22bed49e2e07b0a5339b460d" exitCode=0 Oct 30 17:07:45.922076 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:45.922047 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" event={"ID":"734b4608-cecf-48ad-961e-31b7a2a22871","Type":"ContainerDied","Data":"de2550337d35f9b2a850dcc2e3815c2d3cb0604e22bed49e2e07b0a5339b460d"} Oct 30 17:07:45.957575 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:45.957540 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" Oct 30 17:07:46.011463 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:46.011430 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734b4608-cecf-48ad-961e-31b7a2a22871-kube-api-access-7bklg\" (UniqueName: \"kubernetes.io/projected/734b4608-cecf-48ad-961e-31b7a2a22871-kube-api-access-7bklg\") pod \"734b4608-cecf-48ad-961e-31b7a2a22871\" (UID: \"734b4608-cecf-48ad-961e-31b7a2a22871\") " Oct 30 17:07:46.011717 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:46.011699 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/734b4608-cecf-48ad-961e-31b7a2a22871-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734b4608-cecf-48ad-961e-31b7a2a22871-plugins-config-volume\") pod \"734b4608-cecf-48ad-961e-31b7a2a22871\" (UID: \"734b4608-cecf-48ad-961e-31b7a2a22871\") " Oct 30 17:07:46.012291 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:46.012261 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/734b4608-cecf-48ad-961e-31b7a2a22871-plugins-config-volume" pod "734b4608-cecf-48ad-961e-31b7a2a22871" (UID: "734b4608-cecf-48ad-961e-31b7a2a22871"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:46.014955 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:46.014927 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734b4608-cecf-48ad-961e-31b7a2a22871-kube-api-access-7bklg" pod "734b4608-cecf-48ad-961e-31b7a2a22871" (UID: "734b4608-cecf-48ad-961e-31b7a2a22871"). InnerVolumeSpecName "kube-api-access-7bklg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:46.113715 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:46.113681 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734b4608-cecf-48ad-961e-31b7a2a22871-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:46.113715 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:46.113713 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bklg\" (UniqueName: \"kubernetes.io/projected/734b4608-cecf-48ad-961e-31b7a2a22871-kube-api-access-7bklg\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:46.932492 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:46.932458 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" Oct 30 17:07:46.932492 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:46.932481 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t" event={"ID":"734b4608-cecf-48ad-961e-31b7a2a22871","Type":"ContainerDied","Data":"4fc2a5f685c057cdecf19367b32afd6faa20d817fc471555ad140b7ec2d478e0"} Oct 30 17:07:46.933149 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:46.932539 6202 scope.go:122] "RemoveContainer" containerID="de2550337d35f9b2a850dcc2e3815c2d3cb0604e22bed49e2e07b0a5339b460d" Oct 30 17:07:46.987060 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:46.987027 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t"] Oct 30 17:07:46.998882 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:46.998843 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-830ba9-8548b78694-dj29t"] Oct 30 17:07:47.361117 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:47.361076 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:47.477330 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:47.477288 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:48.412175 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:48.412140 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="734b4608-cecf-48ad-961e-31b7a2a22871" path="/var/lib/kubelet/pods/734b4608-cecf-48ad-961e-31b7a2a22871/volumes" Oct 30 17:07:49.317562 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:49.317507 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:50.829126 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:50.829089 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:51.465822 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:51.447742 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:51.667536 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:51.667486 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:51.690729 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:51.690665 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm"] Oct 30 17:07:51.701308 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:51.701269 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-830ba9-h9rzm"] Oct 30 17:07:51.946001 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:51.942403 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:52.169960 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:52.168831 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:52.491864 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:52.491756 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ba53a4d-10d1-44e6-8d75-c0f77e50e462" path="/var/lib/kubelet/pods/0ba53a4d-10d1-44e6-8d75-c0f77e50e462/volumes" Oct 30 17:07:53.441839 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.441799 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" podUID="e660a917-1db1-4c82-b938-92f1ba909939" containerName="main" containerID="cri-o://f5a405694d525cc93c58e26ce6639a50e8d777ebc71e04b7108256bacd01f99e" gracePeriod=45 Oct 30 17:07:53.739011 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.738981 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:07:53.824570 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.824524 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-home\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-home\") pod \"e660a917-1db1-4c82-b938-92f1ba909939\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " Oct 30 17:07:53.824802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.824677 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-dshm\") pod \"e660a917-1db1-4c82-b938-92f1ba909939\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " Oct 30 17:07:53.824908 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.824817 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-model-cache\") pod \"e660a917-1db1-4c82-b938-92f1ba909939\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " Oct 30 17:07:53.824908 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.824900 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e660a917-1db1-4c82-b938-92f1ba909939-kube-api-access-rh7j5\" (UniqueName: \"kubernetes.io/projected/e660a917-1db1-4c82-b938-92f1ba909939-kube-api-access-rh7j5\") pod \"e660a917-1db1-4c82-b938-92f1ba909939\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " Oct 30 17:07:53.825037 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.824906 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-home" pod "e660a917-1db1-4c82-b938-92f1ba909939" (UID: "e660a917-1db1-4c82-b938-92f1ba909939"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:53.825037 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.824978 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e660a917-1db1-4c82-b938-92f1ba909939-tls-certs\" (UniqueName: \"kubernetes.io/secret/e660a917-1db1-4c82-b938-92f1ba909939-tls-certs\") pod \"e660a917-1db1-4c82-b938-92f1ba909939\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " Oct 30 17:07:53.825163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.825063 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-tmp-dir\") pod \"e660a917-1db1-4c82-b938-92f1ba909939\" (UID: \"e660a917-1db1-4c82-b938-92f1ba909939\") " Oct 30 17:07:53.825163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.825113 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-model-cache" pod "e660a917-1db1-4c82-b938-92f1ba909939" (UID: "e660a917-1db1-4c82-b938-92f1ba909939"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:53.826091 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.826039 6202 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-model-cache\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:53.826091 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.826066 6202 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-home\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:53.826391 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.826352 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-tmp-dir" pod "e660a917-1db1-4c82-b938-92f1ba909939" (UID: "e660a917-1db1-4c82-b938-92f1ba909939"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:53.829204 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.829172 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-dshm" pod "e660a917-1db1-4c82-b938-92f1ba909939" (UID: "e660a917-1db1-4c82-b938-92f1ba909939"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:53.829991 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.829891 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e660a917-1db1-4c82-b938-92f1ba909939-tls-certs" pod "e660a917-1db1-4c82-b938-92f1ba909939" (UID: "e660a917-1db1-4c82-b938-92f1ba909939"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:07:53.829991 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.829954 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e660a917-1db1-4c82-b938-92f1ba909939-kube-api-access-rh7j5" pod "e660a917-1db1-4c82-b938-92f1ba909939" (UID: "e660a917-1db1-4c82-b938-92f1ba909939"). InnerVolumeSpecName "kube-api-access-rh7j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:53.927286 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.927248 6202 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-tmp-dir\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:53.927286 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.927280 6202 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e660a917-1db1-4c82-b938-92f1ba909939-dshm\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:53.927286 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.927290 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh7j5\" (UniqueName: \"kubernetes.io/projected/e660a917-1db1-4c82-b938-92f1ba909939-kube-api-access-rh7j5\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:53.927552 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:53.927299 6202 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e660a917-1db1-4c82-b938-92f1ba909939-tls-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:07:54.156977 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:54.156938 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:54.182296 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:54.182260 6202 generic.go:356] "Generic (PLEG): container finished" podID="e660a917-1db1-4c82-b938-92f1ba909939" containerID="f5a405694d525cc93c58e26ce6639a50e8d777ebc71e04b7108256bacd01f99e" exitCode=0 Oct 30 17:07:54.182513 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:54.182468 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" Oct 30 17:07:54.182602 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:54.182557 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" event={"ID":"e660a917-1db1-4c82-b938-92f1ba909939","Type":"ContainerDied","Data":"f5a405694d525cc93c58e26ce6639a50e8d777ebc71e04b7108256bacd01f99e"} Oct 30 17:07:54.182602 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:54.182591 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9" event={"ID":"e660a917-1db1-4c82-b938-92f1ba909939","Type":"ContainerDied","Data":"9fd5030ba2d40e972cd8229aaa010fb353c9ec7e688d189dc396ac7c8825f35e"} Oct 30 17:07:54.182727 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:54.182617 6202 scope.go:122] "RemoveContainer" containerID="f5a405694d525cc93c58e26ce6639a50e8d777ebc71e04b7108256bacd01f99e" Oct 30 17:07:54.199815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:54.199785 6202 scope.go:122] "RemoveContainer" containerID="f5a405694d525cc93c58e26ce6639a50e8d777ebc71e04b7108256bacd01f99e" Oct 30 17:07:54.200544 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:07:54.200493 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a405694d525cc93c58e26ce6639a50e8d777ebc71e04b7108256bacd01f99e\": container with ID starting with f5a405694d525cc93c58e26ce6639a50e8d777ebc71e04b7108256bacd01f99e not found: ID does not exist" containerID="f5a405694d525cc93c58e26ce6639a50e8d777ebc71e04b7108256bacd01f99e" Oct 30 17:07:54.200644 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:54.200543 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a405694d525cc93c58e26ce6639a50e8d777ebc71e04b7108256bacd01f99e"} err="failed to get container status \"f5a405694d525cc93c58e26ce6639a50e8d777ebc71e04b7108256bacd01f99e\": rpc error: code = NotFound desc = could not find container \"f5a405694d525cc93c58e26ce6639a50e8d777ebc71e04b7108256bacd01f99e\": container with ID starting with f5a405694d525cc93c58e26ce6639a50e8d777ebc71e04b7108256bacd01f99e not found: ID does not exist" Oct 30 17:07:54.244186 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:54.244055 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9"] Oct 30 17:07:54.251319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:54.249110 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-661509/rate-test-model-661509-kserve-56db4b6ccc-b22x9"] Oct 30 17:07:54.281136 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:54.281100 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:54.430460 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:54.430367 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e660a917-1db1-4c82-b938-92f1ba909939" path="/var/lib/kubelet/pods/e660a917-1db1-4c82-b938-92f1ba909939/volumes" Oct 30 17:07:54.919056 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:54.919023 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:55.671704 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:55.671667 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:55.986706 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:55.986603 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:56.278851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:56.278301 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:58.191444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:58.191404 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:58.492608 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:58.492527 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:58.605437 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:58.605385 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:58.804204 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:58.804108 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n"] Oct 30 17:07:58.804459 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:58.804418 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" podUID="9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" containerName="istio-proxy" containerID="cri-o://30af0129587010554123735f084032f73ae8a5333f133ee7db273937ebbe9651" gracePeriod=30 Oct 30 17:07:58.827401 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:58.827372 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:59.373174 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.373134 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs"] Oct 30 17:07:59.374328 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.374305 6202 cpu_manager.go:414] "containerMap: removing container" podUID="e660a917-1db1-4c82-b938-92f1ba909939" containerName="main" Oct 30 17:07:59.374328 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.374327 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e660a917-1db1-4c82-b938-92f1ba909939" containerName="main" Oct 30 17:07:59.374524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.374349 6202 cpu_manager.go:414] "containerMap: removing container" podUID="734b4608-cecf-48ad-961e-31b7a2a22871" containerName="payload-processing" Oct 30 17:07:59.374524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.374355 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734b4608-cecf-48ad-961e-31b7a2a22871" containerName="payload-processing" Oct 30 17:07:59.374524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.374377 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3eaa54db-afe1-4a37-ba9f-1e0745826d93" containerName="revoke-keys" Oct 30 17:07:59.374524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.374382 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eaa54db-afe1-4a37-ba9f-1e0745826d93" containerName="revoke-keys" Oct 30 17:07:59.374524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.374395 6202 cpu_manager.go:414] "containerMap: removing container" podUID="721fc1d7-6ad4-4f60-ae9f-6503de4d0331" containerName="revoke-keys" Oct 30 17:07:59.374524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.374400 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721fc1d7-6ad4-4f60-ae9f-6503de4d0331" containerName="revoke-keys" Oct 30 17:07:59.374851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.374534 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="734b4608-cecf-48ad-961e-31b7a2a22871" containerName="payload-processing" Oct 30 17:07:59.374851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.374553 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eaa54db-afe1-4a37-ba9f-1e0745826d93" containerName="revoke-keys" Oct 30 17:07:59.374851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.374564 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="721fc1d7-6ad4-4f60-ae9f-6503de4d0331" containerName="revoke-keys" Oct 30 17:07:59.374851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.374579 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="e660a917-1db1-4c82-b938-92f1ba909939" containerName="main" Oct 30 17:07:59.380809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.380712 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs" Oct 30 17:07:59.396264 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.396235 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs"] Oct 30 17:07:59.396928 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.396910 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:07:59.397022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.396976 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:07:59.480990 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.480958 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:07:59.481169 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.481020 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:07:59.492978 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.492954 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1487c570-b515-4f4d-acbc-a8114e215dd6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs\" (UID: \"1487c570-b515-4f4d-acbc-a8114e215dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs" Oct 30 17:07:59.493119 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.492989 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1487c570-b515-4f4d-acbc-a8114e215dd6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs\" (UID: \"1487c570-b515-4f4d-acbc-a8114e215dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs" Oct 30 17:07:59.493177 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.493159 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvcs5\" (UniqueName: \"kubernetes.io/projected/1487c570-b515-4f4d-acbc-a8114e215dd6-kube-api-access-jvcs5\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs\" (UID: \"1487c570-b515-4f4d-acbc-a8114e215dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs" Oct 30 17:07:59.594384 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.594351 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1487c570-b515-4f4d-acbc-a8114e215dd6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs\" (UID: \"1487c570-b515-4f4d-acbc-a8114e215dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs" Oct 30 17:07:59.594597 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.594393 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1487c570-b515-4f4d-acbc-a8114e215dd6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs\" (UID: \"1487c570-b515-4f4d-acbc-a8114e215dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs" Oct 30 17:07:59.594597 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.594465 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvcs5\" (UniqueName: \"kubernetes.io/projected/1487c570-b515-4f4d-acbc-a8114e215dd6-kube-api-access-jvcs5\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs\" (UID: \"1487c570-b515-4f4d-acbc-a8114e215dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs" Oct 30 17:07:59.595197 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.595166 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1487c570-b515-4f4d-acbc-a8114e215dd6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs\" (UID: \"1487c570-b515-4f4d-acbc-a8114e215dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs" Oct 30 17:07:59.596922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.596900 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1487c570-b515-4f4d-acbc-a8114e215dd6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs\" (UID: \"1487c570-b515-4f4d-acbc-a8114e215dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs" Oct 30 17:07:59.617929 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.617901 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvcs5\" (UniqueName: \"kubernetes.io/projected/1487c570-b515-4f4d-acbc-a8114e215dd6-kube-api-access-jvcs5\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs\" (UID: \"1487c570-b515-4f4d-acbc-a8114e215dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs" Oct 30 17:07:59.701957 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:07:59.701878 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs" Oct 30 17:08:00.004610 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:08:00.004575 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1487c570_b515_4f4d_acbc_a8114e215dd6.slice/crio-7acb612ab03fa0008253cb3536b4752d179bbd2c3b74b7788ed97f0c4afa5cbb WatchSource:0}: Error finding container 7acb612ab03fa0008253cb3536b4752d179bbd2c3b74b7788ed97f0c4afa5cbb: Status 404 returned error can't find the container with id 7acb612ab03fa0008253cb3536b4752d179bbd2c3b74b7788ed97f0c4afa5cbb Oct 30 17:08:00.004850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:00.004827 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs"] Oct 30 17:08:00.237739 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:00.237701 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs" event={"ID":"1487c570-b515-4f4d-acbc-a8114e215dd6","Type":"ContainerStarted","Data":"7acb612ab03fa0008253cb3536b4752d179bbd2c3b74b7788ed97f0c4afa5cbb"} Oct 30 17:08:00.242578 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:00.242540 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:00.402757 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:00.402703 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:00.516452 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:00.516418 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:00.639758 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:00.639665 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:01.250007 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:01.249973 6202 generic.go:356] "Generic (PLEG): container finished" podID="1487c570-b515-4f4d-acbc-a8114e215dd6" containerID="dba2e9da5c056d578af128e805dc441dbc1c37ace9f46aa9a713b93d465e3396" exitCode=0 Oct 30 17:08:01.250199 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:01.250055 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs" event={"ID":"1487c570-b515-4f4d-acbc-a8114e215dd6","Type":"ContainerDied","Data":"dba2e9da5c056d578af128e805dc441dbc1c37ace9f46aa9a713b93d465e3396"} Oct 30 17:08:02.123231 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:02.123194 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:02.241661 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:02.241618 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:02.467962 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:02.446669 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:02.711096 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:02.711068 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs" Oct 30 17:08:02.862953 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:02.862916 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1487c570-b515-4f4d-acbc-a8114e215dd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1487c570-b515-4f4d-acbc-a8114e215dd6-maas-api-xks-ca\") pod \"1487c570-b515-4f4d-acbc-a8114e215dd6\" (UID: \"1487c570-b515-4f4d-acbc-a8114e215dd6\") " Oct 30 17:08:02.862953 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:02.862958 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1487c570-b515-4f4d-acbc-a8114e215dd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1487c570-b515-4f4d-acbc-a8114e215dd6-maas-api-openshift-ca\") pod \"1487c570-b515-4f4d-acbc-a8114e215dd6\" (UID: \"1487c570-b515-4f4d-acbc-a8114e215dd6\") " Oct 30 17:08:02.863257 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:02.863125 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1487c570-b515-4f4d-acbc-a8114e215dd6-kube-api-access-jvcs5\" (UniqueName: \"kubernetes.io/projected/1487c570-b515-4f4d-acbc-a8114e215dd6-kube-api-access-jvcs5\") pod \"1487c570-b515-4f4d-acbc-a8114e215dd6\" (UID: \"1487c570-b515-4f4d-acbc-a8114e215dd6\") " Oct 30 17:08:02.863396 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:02.863372 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1487c570-b515-4f4d-acbc-a8114e215dd6-maas-api-openshift-ca" pod "1487c570-b515-4f4d-acbc-a8114e215dd6" (UID: "1487c570-b515-4f4d-acbc-a8114e215dd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:02.863945 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:02.863919 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1487c570-b515-4f4d-acbc-a8114e215dd6-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:02.865477 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:02.865449 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1487c570-b515-4f4d-acbc-a8114e215dd6-maas-api-xks-ca" pod "1487c570-b515-4f4d-acbc-a8114e215dd6" (UID: "1487c570-b515-4f4d-acbc-a8114e215dd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:08:02.865794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:02.865753 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1487c570-b515-4f4d-acbc-a8114e215dd6-kube-api-access-jvcs5" pod "1487c570-b515-4f4d-acbc-a8114e215dd6" (UID: "1487c570-b515-4f4d-acbc-a8114e215dd6"). InnerVolumeSpecName "kube-api-access-jvcs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:02.966303 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:02.966209 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvcs5\" (UniqueName: \"kubernetes.io/projected/1487c570-b515-4f4d-acbc-a8114e215dd6-kube-api-access-jvcs5\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:02.966303 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:02.966249 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1487c570-b515-4f4d-acbc-a8114e215dd6-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:03.274832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:03.274712 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs" event={"ID":"1487c570-b515-4f4d-acbc-a8114e215dd6","Type":"ContainerDied","Data":"7acb612ab03fa0008253cb3536b4752d179bbd2c3b74b7788ed97f0c4afa5cbb"} Oct 30 17:08:03.274832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:03.274797 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7acb612ab03fa0008253cb3536b4752d179bbd2c3b74b7788ed97f0c4afa5cbb" Oct 30 17:08:03.275262 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:03.274866 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs" Oct 30 17:08:04.114913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.114889 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:08:04.295198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.290462 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-token\" (UniqueName: \"kubernetes.io/projected/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-token\") pod \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " Oct 30 17:08:04.295198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.290561 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-kube-api-access-z5m8q\" (UniqueName: \"kubernetes.io/projected/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-kube-api-access-z5m8q\") pod \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " Oct 30 17:08:04.295198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.290632 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-envoy\") pod \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " Oct 30 17:08:04.295198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.290659 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-podinfo\") pod \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " Oct 30 17:08:04.295198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.290718 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-workload-certs\") pod \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " Oct 30 17:08:04.295198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.290802 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istiod-ca-cert\") pod \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " Oct 30 17:08:04.295198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.290846 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-workload-socket\") pod \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " Oct 30 17:08:04.295198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.291013 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-credential-socket\") pod \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " Oct 30 17:08:04.295198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.291195 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-data\") pod \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\" (UID: \"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84\") " Oct 30 17:08:04.295198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.294480 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-workload-socket" pod "9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" (UID: "9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:04.295198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.294816 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istiod-ca-cert" pod "9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" (UID: "9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:04.295198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.294841 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-data" pod "9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" (UID: "9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:04.295198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.295074 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-credential-socket" pod "9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" (UID: "9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:04.327337 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.296624 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-workload-certs" pod "9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" (UID: "9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:04.338858 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.321390 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-kube-api-access-z5m8q" pod "9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" (UID: "9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84"). InnerVolumeSpecName "kube-api-access-z5m8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:04.344725 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.344685 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:04.352735 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.347141 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-podinfo" pod "9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" (UID: "9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:08:04.352735 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.347944 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-token" pod "9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" (UID: "9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:04.352735 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.352163 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-envoy" pod "9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" (UID: "9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:04.358556 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.354639 6202 generic.go:356] "Generic (PLEG): container finished" podID="9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" containerID="30af0129587010554123735f084032f73ae8a5333f133ee7db273937ebbe9651" exitCode=0 Oct 30 17:08:04.358556 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.354847 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" event={"ID":"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84","Type":"ContainerDied","Data":"30af0129587010554123735f084032f73ae8a5333f133ee7db273937ebbe9651"} Oct 30 17:08:04.358556 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.354925 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" event={"ID":"9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84","Type":"ContainerDied","Data":"80de409fd00e85e7c16d92a8b3bcbbb46536154bf66ec4df706caf1765364088"} Oct 30 17:08:04.358556 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.354949 6202 scope.go:122] "RemoveContainer" containerID="30af0129587010554123735f084032f73ae8a5333f133ee7db273937ebbe9651" Oct 30 17:08:04.358556 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.355086 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n" Oct 30 17:08:04.399791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.397806 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:04.399791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.397848 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:04.399791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.397861 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:04.399791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.397874 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:04.399791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.397889 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:04.399791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.397903 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:04.399791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.397916 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5m8q\" (UniqueName: \"kubernetes.io/projected/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-kube-api-access-z5m8q\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:04.419868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.403957 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:04.419868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.403992 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:04.427357 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.426883 6202 scope.go:122] "RemoveContainer" containerID="30af0129587010554123735f084032f73ae8a5333f133ee7db273937ebbe9651" Oct 30 17:08:04.427468 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:08:04.427354 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30af0129587010554123735f084032f73ae8a5333f133ee7db273937ebbe9651\": container with ID starting with 30af0129587010554123735f084032f73ae8a5333f133ee7db273937ebbe9651 not found: ID does not exist" containerID="30af0129587010554123735f084032f73ae8a5333f133ee7db273937ebbe9651" Oct 30 17:08:04.427468 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.427391 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30af0129587010554123735f084032f73ae8a5333f133ee7db273937ebbe9651"} err="failed to get container status \"30af0129587010554123735f084032f73ae8a5333f133ee7db273937ebbe9651\": rpc error: code = NotFound desc = could not find container \"30af0129587010554123735f084032f73ae8a5333f133ee7db273937ebbe9651\": container with ID starting with 30af0129587010554123735f084032f73ae8a5333f133ee7db273937ebbe9651 not found: ID does not exist" Oct 30 17:08:04.443779 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.443729 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n"] Oct 30 17:08:04.450533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.450504 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-m9n5n"] Oct 30 17:08:04.885562 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:04.885530 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv"] Oct 30 17:08:05.051075 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:05.051028 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:05.166192 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:05.166085 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:05.281484 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:05.281451 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:05.628792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:05.628238 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:05.814709 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:05.814586 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:06.124794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.123819 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg"] Oct 30 17:08:06.125012 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.124940 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" containerName="istio-proxy" Oct 30 17:08:06.125012 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.124959 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" containerName="istio-proxy" Oct 30 17:08:06.125012 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.124980 6202 cpu_manager.go:414] "containerMap: removing container" podUID="1487c570-b515-4f4d-acbc-a8114e215dd6" containerName="revoke-keys" Oct 30 17:08:06.125012 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.124987 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1487c570-b515-4f4d-acbc-a8114e215dd6" containerName="revoke-keys" Oct 30 17:08:06.125227 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.125191 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" containerName="istio-proxy" Oct 30 17:08:06.125227 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.125206 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="1487c570-b515-4f4d-acbc-a8114e215dd6" containerName="revoke-keys" Oct 30 17:08:06.156795 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.151807 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg" Oct 30 17:08:06.168792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.157080 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg"] Oct 30 17:08:06.194328 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.192166 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-55e83e\"" Oct 30 17:08:06.243789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.242973 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zpbb\" (UniqueName: \"kubernetes.io/projected/2dad0e9c-8af9-45d7-b040-f144b1aac81d-kube-api-access-4zpbb\") pod \"payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg\" (UID: \"2dad0e9c-8af9-45d7-b040-f144b1aac81d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg" Oct 30 17:08:06.245820 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.244911 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dad0e9c-8af9-45d7-b040-f144b1aac81d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg\" (UID: \"2dad0e9c-8af9-45d7-b040-f144b1aac81d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg" Oct 30 17:08:06.347095 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.347046 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zpbb\" (UniqueName: \"kubernetes.io/projected/2dad0e9c-8af9-45d7-b040-f144b1aac81d-kube-api-access-4zpbb\") pod \"payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg\" (UID: \"2dad0e9c-8af9-45d7-b040-f144b1aac81d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg" Oct 30 17:08:06.347402 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.347384 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dad0e9c-8af9-45d7-b040-f144b1aac81d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg\" (UID: \"2dad0e9c-8af9-45d7-b040-f144b1aac81d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg" Oct 30 17:08:06.348160 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.348138 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dad0e9c-8af9-45d7-b040-f144b1aac81d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg\" (UID: \"2dad0e9c-8af9-45d7-b040-f144b1aac81d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg" Oct 30 17:08:06.370568 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.370533 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zpbb\" (UniqueName: \"kubernetes.io/projected/2dad0e9c-8af9-45d7-b040-f144b1aac81d-kube-api-access-4zpbb\") pod \"payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg\" (UID: \"2dad0e9c-8af9-45d7-b040-f144b1aac81d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg" Oct 30 17:08:06.412620 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.412546 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84" path="/var/lib/kubelet/pods/9b56de5c-cde8-4aeb-91dd-c4f1bb2d2d84/volumes" Oct 30 17:08:06.540657 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.540617 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg" Oct 30 17:08:06.844188 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:08:06.844156 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dad0e9c_8af9_45d7_b040_f144b1aac81d.slice/crio-71744aab5a7ceeb926b7171f380d5153284d41cf886974d417ffc7ffab08ae9b WatchSource:0}: Error finding container 71744aab5a7ceeb926b7171f380d5153284d41cf886974d417ffc7ffab08ae9b: Status 404 returned error can't find the container with id 71744aab5a7ceeb926b7171f380d5153284d41cf886974d417ffc7ffab08ae9b Oct 30 17:08:06.845392 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.845361 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg"] Oct 30 17:08:06.845982 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.845956 6202 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:08:06.986350 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:06.986311 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:07.390830 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:07.390794 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg" event={"ID":"2dad0e9c-8af9-45d7-b040-f144b1aac81d","Type":"ContainerStarted","Data":"71744aab5a7ceeb926b7171f380d5153284d41cf886974d417ffc7ffab08ae9b"} Oct 30 17:08:07.393636 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:07.393611 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:08.057818 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:08.056038 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:08.732105 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:08.732056 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:09.412347 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:09.412315 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg" event={"ID":"2dad0e9c-8af9-45d7-b040-f144b1aac81d","Type":"ContainerStarted","Data":"c9b202937c906a5408346b9ac913cb5f3cee367036ed4b01f624565fcb3c349b"} Oct 30 17:08:09.412845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:09.412404 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg" Oct 30 17:08:09.455280 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:09.455227 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg" podStartSLOduration=1.636237666 podStartE2EDuration="3.455212837s" totalImagesPullingTime="1.818975171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:09.451880008 +0000 UTC m=+2803.162334754" watchObservedRunningTime="2026-10-30 17:08:09.455212837 +0000 UTC m=+2803.165667581" Oct 30 17:08:09.892037 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:09.891987 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv" podUID="5150b64f-5c2e-46bc-9604-8edd2fb63956" containerName="payload-processing" containerID="cri-o://b3230ffdd4985ac1988236a106e88864e37f661a3079ccba81f2162853007897" gracePeriod=25 Oct 30 17:08:09.943918 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:09.943850 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:10.065736 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:10.065660 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:10.269385 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:10.269354 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv" Oct 30 17:08:10.359427 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:10.359389 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5150b64f-5c2e-46bc-9604-8edd2fb63956-kube-api-access-4fsqs\" (UniqueName: \"kubernetes.io/projected/5150b64f-5c2e-46bc-9604-8edd2fb63956-kube-api-access-4fsqs\") pod \"5150b64f-5c2e-46bc-9604-8edd2fb63956\" (UID: \"5150b64f-5c2e-46bc-9604-8edd2fb63956\") " Oct 30 17:08:10.360130 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:10.360099 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5150b64f-5c2e-46bc-9604-8edd2fb63956-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5150b64f-5c2e-46bc-9604-8edd2fb63956-plugins-config-volume\") pod \"5150b64f-5c2e-46bc-9604-8edd2fb63956\" (UID: \"5150b64f-5c2e-46bc-9604-8edd2fb63956\") " Oct 30 17:08:10.438610 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:10.420220 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5150b64f-5c2e-46bc-9604-8edd2fb63956-plugins-config-volume" pod "5150b64f-5c2e-46bc-9604-8edd2fb63956" (UID: "5150b64f-5c2e-46bc-9604-8edd2fb63956"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:10.469285 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:10.454090 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5150b64f-5c2e-46bc-9604-8edd2fb63956-kube-api-access-4fsqs" pod "5150b64f-5c2e-46bc-9604-8edd2fb63956" (UID: "5150b64f-5c2e-46bc-9604-8edd2fb63956"). InnerVolumeSpecName "kube-api-access-4fsqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:10.505649 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:10.505552 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5150b64f-5c2e-46bc-9604-8edd2fb63956-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:10.505909 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:10.505888 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fsqs\" (UniqueName: \"kubernetes.io/projected/5150b64f-5c2e-46bc-9604-8edd2fb63956-kube-api-access-4fsqs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:10.562895 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:10.562856 6202 generic.go:356] "Generic (PLEG): container finished" podID="5150b64f-5c2e-46bc-9604-8edd2fb63956" containerID="b3230ffdd4985ac1988236a106e88864e37f661a3079ccba81f2162853007897" exitCode=0 Oct 30 17:08:10.563574 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:10.563548 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv" Oct 30 17:08:10.565755 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:10.565045 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv" event={"ID":"5150b64f-5c2e-46bc-9604-8edd2fb63956","Type":"ContainerDied","Data":"b3230ffdd4985ac1988236a106e88864e37f661a3079ccba81f2162853007897"} Oct 30 17:08:10.566181 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:10.566159 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv" event={"ID":"5150b64f-5c2e-46bc-9604-8edd2fb63956","Type":"ContainerDied","Data":"32a1f4e94f576e793c4d7ca983c9e8cc06d2279181ce6645262bfd989c65495b"} Oct 30 17:08:10.566362 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:10.566242 6202 scope.go:122] "RemoveContainer" containerID="b3230ffdd4985ac1988236a106e88864e37f661a3079ccba81f2162853007897" Oct 30 17:08:10.594000 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:10.593976 6202 scope.go:122] "RemoveContainer" containerID="b3230ffdd4985ac1988236a106e88864e37f661a3079ccba81f2162853007897" Oct 30 17:08:10.594360 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:08:10.594337 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3230ffdd4985ac1988236a106e88864e37f661a3079ccba81f2162853007897\": container with ID starting with b3230ffdd4985ac1988236a106e88864e37f661a3079ccba81f2162853007897 not found: ID does not exist" containerID="b3230ffdd4985ac1988236a106e88864e37f661a3079ccba81f2162853007897" Oct 30 17:08:10.594414 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:10.594369 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3230ffdd4985ac1988236a106e88864e37f661a3079ccba81f2162853007897"} err="failed to get container status \"b3230ffdd4985ac1988236a106e88864e37f661a3079ccba81f2162853007897\": rpc error: code = NotFound desc = could not find container \"b3230ffdd4985ac1988236a106e88864e37f661a3079ccba81f2162853007897\": container with ID starting with b3230ffdd4985ac1988236a106e88864e37f661a3079ccba81f2162853007897 not found: ID does not exist" Oct 30 17:08:10.626539 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:10.626500 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv"] Oct 30 17:08:10.630630 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:10.630598 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-330282-849cb94dd-prxnv"] Oct 30 17:08:10.872201 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:10.872128 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:12.354865 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.354826 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5"] Oct 30 17:08:12.355676 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.355660 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5150b64f-5c2e-46bc-9604-8edd2fb63956" containerName="payload-processing" Oct 30 17:08:12.355723 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.355678 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5150b64f-5c2e-46bc-9604-8edd2fb63956" containerName="payload-processing" Oct 30 17:08:12.355827 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.355816 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5150b64f-5c2e-46bc-9604-8edd2fb63956" containerName="payload-processing" Oct 30 17:08:12.361304 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.361282 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.393301 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.393267 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5"] Oct 30 17:08:12.417159 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.417124 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5150b64f-5c2e-46bc-9604-8edd2fb63956" path="/var/lib/kubelet/pods/5150b64f-5c2e-46bc-9604-8edd2fb63956/volumes" Oct 30 17:08:12.435199 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.435166 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-credential-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.435377 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.435210 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-workload-certs\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.435377 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.435352 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-envoy\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.435501 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.435383 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istiod-ca-cert\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.435558 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.435501 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7pr6\" (UniqueName: \"kubernetes.io/projected/84ccd5fe-cc47-482f-b2cb-e315080fae2a-kube-api-access-k7pr6\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.435633 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.435598 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-token\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.435837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.435822 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-data\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.435909 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.435877 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-podinfo\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.436008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.435989 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-workload-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.537585 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.537547 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-podinfo\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.537790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.537612 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-workload-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.537790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.537699 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-credential-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.537790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.537728 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-workload-certs\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.537966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.537823 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-envoy\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.537966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.537860 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istiod-ca-cert\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.537966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.537911 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7pr6\" (UniqueName: \"kubernetes.io/projected/84ccd5fe-cc47-482f-b2cb-e315080fae2a-kube-api-access-k7pr6\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.537966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.537962 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-token\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.538197 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.538080 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-workload-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.538197 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.538084 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-data\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.538197 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.538143 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-credential-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.538493 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.538464 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-workload-certs\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.538493 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.538485 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-data\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.539443 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.539420 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istiod-ca-cert\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.540956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.540928 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-envoy\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.541097 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.541079 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-podinfo\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.561671 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.561644 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-token\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.561671 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.561657 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7pr6\" (UniqueName: \"kubernetes.io/projected/84ccd5fe-cc47-482f-b2cb-e315080fae2a-kube-api-access-k7pr6\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.676255 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.676164 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:12.843320 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:12.843278 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:13.006073 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:13.006043 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5"] Oct 30 17:08:13.006517 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:08:13.006489 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ccd5fe_cc47_482f_b2cb_e315080fae2a.slice/crio-3cf847654dcdb6998e22326dacd51f158a2e396d3c7e7362f053aae61dd0497b WatchSource:0}: Error finding container 3cf847654dcdb6998e22326dacd51f158a2e396d3c7e7362f053aae61dd0497b: Status 404 returned error can't find the container with id 3cf847654dcdb6998e22326dacd51f158a2e396d3c7e7362f053aae61dd0497b Oct 30 17:08:13.009668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:13.009639 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:08:13.009748 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:13.009704 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:08:13.009748 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:13.009732 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:08:13.594348 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:13.594304 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" event={"ID":"84ccd5fe-cc47-482f-b2cb-e315080fae2a","Type":"ContainerStarted","Data":"c587cbb17d17c9cc9d611a6300d579abe2a9b4994f187f7d0c92f9620b070a0d"} Oct 30 17:08:13.594812 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:13.594377 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" event={"ID":"84ccd5fe-cc47-482f-b2cb-e315080fae2a","Type":"ContainerStarted","Data":"3cf847654dcdb6998e22326dacd51f158a2e396d3c7e7362f053aae61dd0497b"} Oct 30 17:08:13.678137 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:13.678098 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:13.679300 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:13.679248 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" podStartSLOduration=1.679228847 podStartE2EDuration="1.679228847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:13.657232001 +0000 UTC m=+2807.367686745" watchObservedRunningTime="2026-10-30 17:08:13.679228847 +0000 UTC m=+2807.389683593" Oct 30 17:08:14.100439 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:08:14.100392 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:08:14.679116 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:14.679086 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Oct 30 17:08:14.679507 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:14.679151 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" podUID="84ccd5fe-cc47-482f-b2cb-e315080fae2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Oct 30 17:08:15.172835 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:15.172757 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:15.677096 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:15.677054 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Oct 30 17:08:15.677302 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:15.677134 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" podUID="84ccd5fe-cc47-482f-b2cb-e315080fae2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Oct 30 17:08:16.676793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:16.676492 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Oct 30 17:08:16.676793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:16.676550 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" podUID="84ccd5fe-cc47-482f-b2cb-e315080fae2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Oct 30 17:08:16.833683 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:16.833647 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:17.677964 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:17.677926 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Oct 30 17:08:17.678493 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:17.678031 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" podUID="84ccd5fe-cc47-482f-b2cb-e315080fae2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Oct 30 17:08:18.589912 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:18.589877 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:18.686981 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:18.686947 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Oct 30 17:08:18.687701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:18.687574 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" podUID="84ccd5fe-cc47-482f-b2cb-e315080fae2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Oct 30 17:08:19.681864 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:19.681829 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:19.682331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:19.682301 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:19.683034 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:19.683011 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:08:20.278282 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:20.278250 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:20.568333 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:20.568236 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg" Oct 30 17:08:21.928603 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:21.928559 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:23.394264 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:23.394176 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:23.506125 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:23.506083 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:24.001261 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:24.001223 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:24.151094 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:24.151061 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:24.355081 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:24.355047 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:24.389008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:24.388975 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs"] Oct 30 17:08:24.398585 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:24.398555 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e5bfb0-29chs"] Oct 30 17:08:24.412141 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:24.412115 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1487c570-b515-4f4d-acbc-a8114e215dd6" path="/var/lib/kubelet/pods/1487c570-b515-4f4d-acbc-a8114e215dd6/volumes" Oct 30 17:08:24.490813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:24.490564 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:24.719750 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:24.719659 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:24.897711 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:24.897641 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:25.239930 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:25.239828 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:25.439806 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:25.432344 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:26.493033 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:26.492994 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:28.325040 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:28.325001 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:28.989300 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:28.989265 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:29.121091 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:29.121051 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:30.327126 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:30.327087 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:30.666412 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:30.666331 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:30.757444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:30.757409 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj"] Oct 30 17:08:30.762521 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:30.762494 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj" Oct 30 17:08:30.782678 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:30.782649 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:30.789536 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:30.789499 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-97a47d\"" Oct 30 17:08:30.795365 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:30.795341 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj"] Oct 30 17:08:30.835118 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:30.835079 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft"] Oct 30 17:08:30.853403 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:30.853369 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft" Oct 30 17:08:30.854478 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:30.854445 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q4kj\" (UniqueName: \"kubernetes.io/projected/c7b73230-da08-40d4-8ecb-93cf4c5a6b6f-kube-api-access-6q4kj\") pod \"payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj\" (UID: \"c7b73230-da08-40d4-8ecb-93cf4c5a6b6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj" Oct 30 17:08:30.854781 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:30.854734 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b73230-da08-40d4-8ecb-93cf4c5a6b6f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj\" (UID: \"c7b73230-da08-40d4-8ecb-93cf4c5a6b6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj" Oct 30 17:08:30.869189 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:30.865710 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft"] Oct 30 17:08:30.869189 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:30.868565 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-97a47d-dockercfg-nf2bh\"" Oct 30 17:08:30.960041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:30.958037 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csvsh\" (UniqueName: \"kubernetes.io/projected/9bba5eae-bd9d-4134-8945-15ec55fe4391-kube-api-access-csvsh\") pod \"payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft\" (UID: \"9bba5eae-bd9d-4134-8945-15ec55fe4391\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft" Oct 30 17:08:30.960041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:30.958092 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bba5eae-bd9d-4134-8945-15ec55fe4391-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft\" (UID: \"9bba5eae-bd9d-4134-8945-15ec55fe4391\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft" Oct 30 17:08:30.960041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:30.958237 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q4kj\" (UniqueName: \"kubernetes.io/projected/c7b73230-da08-40d4-8ecb-93cf4c5a6b6f-kube-api-access-6q4kj\") pod \"payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj\" (UID: \"c7b73230-da08-40d4-8ecb-93cf4c5a6b6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj" Oct 30 17:08:30.960041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:30.958486 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b73230-da08-40d4-8ecb-93cf4c5a6b6f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj\" (UID: \"c7b73230-da08-40d4-8ecb-93cf4c5a6b6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj" Oct 30 17:08:30.960041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:30.959597 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b73230-da08-40d4-8ecb-93cf4c5a6b6f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj\" (UID: \"c7b73230-da08-40d4-8ecb-93cf4c5a6b6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj" Oct 30 17:08:31.064122 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:31.064079 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csvsh\" (UniqueName: \"kubernetes.io/projected/9bba5eae-bd9d-4134-8945-15ec55fe4391-kube-api-access-csvsh\") pod \"payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft\" (UID: \"9bba5eae-bd9d-4134-8945-15ec55fe4391\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft" Oct 30 17:08:31.064398 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:31.064376 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bba5eae-bd9d-4134-8945-15ec55fe4391-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft\" (UID: \"9bba5eae-bd9d-4134-8945-15ec55fe4391\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft" Oct 30 17:08:31.065746 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:31.065658 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bba5eae-bd9d-4134-8945-15ec55fe4391-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft\" (UID: \"9bba5eae-bd9d-4134-8945-15ec55fe4391\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft" Oct 30 17:08:31.098895 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:31.073508 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q4kj\" (UniqueName: \"kubernetes.io/projected/c7b73230-da08-40d4-8ecb-93cf4c5a6b6f-kube-api-access-6q4kj\") pod \"payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj\" (UID: \"c7b73230-da08-40d4-8ecb-93cf4c5a6b6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj" Oct 30 17:08:31.101933 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:31.101654 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj" Oct 30 17:08:31.170931 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:31.170897 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csvsh\" (UniqueName: \"kubernetes.io/projected/9bba5eae-bd9d-4134-8945-15ec55fe4391-kube-api-access-csvsh\") pod \"payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft\" (UID: \"9bba5eae-bd9d-4134-8945-15ec55fe4391\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft" Oct 30 17:08:31.475193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:31.474729 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft" Oct 30 17:08:31.480978 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:31.480946 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj"] Oct 30 17:08:31.696112 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:31.696067 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:31.808810 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:31.808705 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:31.817378 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:08:31.817345 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bba5eae_bd9d_4134_8945_15ec55fe4391.slice/crio-9485e64d74f8756e712b60abd1626aebe1235e8c3b3056877d664e5e08973e1e WatchSource:0}: Error finding container 9485e64d74f8756e712b60abd1626aebe1235e8c3b3056877d664e5e08973e1e: Status 404 returned error can't find the container with id 9485e64d74f8756e712b60abd1626aebe1235e8c3b3056877d664e5e08973e1e Oct 30 17:08:31.817970 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:31.817902 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft"] Oct 30 17:08:32.051216 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:32.051162 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:32.226849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:32.226805 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft" event={"ID":"9bba5eae-bd9d-4134-8945-15ec55fe4391","Type":"ContainerStarted","Data":"4a56376d3426b735dc78b68e4974ebdd8c73d9d78c314a66201f38f21c7cb917"} Oct 30 17:08:32.227205 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:32.227179 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft" event={"ID":"9bba5eae-bd9d-4134-8945-15ec55fe4391","Type":"ContainerStarted","Data":"9485e64d74f8756e712b60abd1626aebe1235e8c3b3056877d664e5e08973e1e"} Oct 30 17:08:32.236791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:32.228578 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft" Oct 30 17:08:32.261646 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:32.261607 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj" event={"ID":"c7b73230-da08-40d4-8ecb-93cf4c5a6b6f","Type":"ContainerStarted","Data":"770e6a9a06b1032b42342d6c86512d92858b696ccff7ab3793f0404453d1870b"} Oct 30 17:08:32.261960 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:32.261937 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj" event={"ID":"c7b73230-da08-40d4-8ecb-93cf4c5a6b6f","Type":"ContainerStarted","Data":"4bafa4b0e31c670e2c5eba57d83bc13886992dc684ad96c9eadb5623f3155e3c"} Oct 30 17:08:32.275790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:32.262960 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj" Oct 30 17:08:32.321313 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:32.321242 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft" podStartSLOduration=2.321206838 podStartE2EDuration="2.321206838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:32.287990316 +0000 UTC m=+2825.998445061" watchObservedRunningTime="2026-10-30 17:08:32.321206838 +0000 UTC m=+2826.031661583" Oct 30 17:08:32.545416 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:32.545308 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:32.677671 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:32.677631 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:32.787337 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:32.787298 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:33.039579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.039516 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj" podStartSLOduration=3.039496774 podStartE2EDuration="3.039496774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:32.336372106 +0000 UTC m=+2826.046826854" watchObservedRunningTime="2026-10-30 17:08:33.039496774 +0000 UTC m=+2826.749951518" Oct 30 17:08:33.039810 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.039705 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q"] Oct 30 17:08:33.045789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.045745 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q" Oct 30 17:08:33.061164 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.060533 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:08:33.064168 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.064142 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q"] Oct 30 17:08:33.070226 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.070200 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:08:33.113526 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.113465 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:08:33.114204 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.113652 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:08:33.122063 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.122036 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c7da8ff-a3e0-42e2-a369-6c6351686979-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q\" (UID: \"8c7da8ff-a3e0-42e2-a369-6c6351686979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q" Oct 30 17:08:33.122223 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.122087 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvjld\" (UniqueName: \"kubernetes.io/projected/8c7da8ff-a3e0-42e2-a369-6c6351686979-kube-api-access-pvjld\") pod \"maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q\" (UID: \"8c7da8ff-a3e0-42e2-a369-6c6351686979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q" Oct 30 17:08:33.122337 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.122317 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c7da8ff-a3e0-42e2-a369-6c6351686979-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q\" (UID: \"8c7da8ff-a3e0-42e2-a369-6c6351686979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q" Oct 30 17:08:33.224305 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.224267 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c7da8ff-a3e0-42e2-a369-6c6351686979-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q\" (UID: \"8c7da8ff-a3e0-42e2-a369-6c6351686979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q" Oct 30 17:08:33.224484 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.224330 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjld\" (UniqueName: \"kubernetes.io/projected/8c7da8ff-a3e0-42e2-a369-6c6351686979-kube-api-access-pvjld\") pod \"maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q\" (UID: \"8c7da8ff-a3e0-42e2-a369-6c6351686979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q" Oct 30 17:08:33.224563 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.224545 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c7da8ff-a3e0-42e2-a369-6c6351686979-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q\" (UID: \"8c7da8ff-a3e0-42e2-a369-6c6351686979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q" Oct 30 17:08:33.225173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.225153 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c7da8ff-a3e0-42e2-a369-6c6351686979-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q\" (UID: \"8c7da8ff-a3e0-42e2-a369-6c6351686979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q" Oct 30 17:08:33.226916 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.226885 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c7da8ff-a3e0-42e2-a369-6c6351686979-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q\" (UID: \"8c7da8ff-a3e0-42e2-a369-6c6351686979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q" Oct 30 17:08:33.247414 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.247387 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvjld\" (UniqueName: \"kubernetes.io/projected/8c7da8ff-a3e0-42e2-a369-6c6351686979-kube-api-access-pvjld\") pod \"maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q\" (UID: \"8c7da8ff-a3e0-42e2-a369-6c6351686979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q" Oct 30 17:08:33.367033 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.366995 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q" Oct 30 17:08:33.459269 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.459119 6202 scope.go:122] "RemoveContainer" containerID="072c6cb23b019931b588a13b20b07e6791ad6d820c78efec4de9889cd9e8757c" Oct 30 17:08:33.484074 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.484042 6202 scope.go:122] "RemoveContainer" containerID="d2e5c9e92f7b6f9ab44f0dd3111f09e1bea40207022203fad3c0e398a890e0a4" Oct 30 17:08:33.499047 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.499016 6202 scope.go:122] "RemoveContainer" containerID="91299af34799ddd367f45b7112f35af8d88d4d77d930232978752b5993968812" Oct 30 17:08:33.676867 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:08:33.676741 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7da8ff_a3e0_42e2_a369_6c6351686979.slice/crio-11dd8e5ce55a2801b863f05de2c7ac7c5630368423a5992d40aff02cda3d8db2 WatchSource:0}: Error finding container 11dd8e5ce55a2801b863f05de2c7ac7c5630368423a5992d40aff02cda3d8db2: Status 404 returned error can't find the container with id 11dd8e5ce55a2801b863f05de2c7ac7c5630368423a5992d40aff02cda3d8db2 Oct 30 17:08:33.677418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:33.677392 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q"] Oct 30 17:08:34.284383 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:34.284346 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q" event={"ID":"8c7da8ff-a3e0-42e2-a369-6c6351686979","Type":"ContainerStarted","Data":"811a9c382f91a857c074ecbba4c74f404110c48fb509a9bd0c69398932926663"} Oct 30 17:08:34.284567 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:34.284405 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q" event={"ID":"8c7da8ff-a3e0-42e2-a369-6c6351686979","Type":"ContainerStarted","Data":"11dd8e5ce55a2801b863f05de2c7ac7c5630368423a5992d40aff02cda3d8db2"} Oct 30 17:08:34.329797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:34.329713 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q" podStartSLOduration=1.329695136 podStartE2EDuration="1.329695136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:34.325541371 +0000 UTC m=+2828.035996117" watchObservedRunningTime="2026-10-30 17:08:34.329695136 +0000 UTC m=+2828.040149883" Oct 30 17:08:35.139368 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:35.139282 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:35.431193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:35.430558 6202 generic.go:356] "Generic (PLEG): container finished" podID="8c7da8ff-a3e0-42e2-a369-6c6351686979" containerID="811a9c382f91a857c074ecbba4c74f404110c48fb509a9bd0c69398932926663" exitCode=0 Oct 30 17:08:35.431193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:35.430724 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q" event={"ID":"8c7da8ff-a3e0-42e2-a369-6c6351686979","Type":"ContainerDied","Data":"811a9c382f91a857c074ecbba4c74f404110c48fb509a9bd0c69398932926663"} Oct 30 17:08:36.818803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:36.817617 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q" Oct 30 17:08:36.905815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:36.899002 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c7da8ff-a3e0-42e2-a369-6c6351686979-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c7da8ff-a3e0-42e2-a369-6c6351686979-maas-api-xks-ca\") pod \"8c7da8ff-a3e0-42e2-a369-6c6351686979\" (UID: \"8c7da8ff-a3e0-42e2-a369-6c6351686979\") " Oct 30 17:08:36.905815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:36.899679 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c7da8ff-a3e0-42e2-a369-6c6351686979-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c7da8ff-a3e0-42e2-a369-6c6351686979-maas-api-openshift-ca\") pod \"8c7da8ff-a3e0-42e2-a369-6c6351686979\" (UID: \"8c7da8ff-a3e0-42e2-a369-6c6351686979\") " Oct 30 17:08:36.905815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:36.900347 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c7da8ff-a3e0-42e2-a369-6c6351686979-maas-api-openshift-ca" pod "8c7da8ff-a3e0-42e2-a369-6c6351686979" (UID: "8c7da8ff-a3e0-42e2-a369-6c6351686979"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:36.905815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:36.901617 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c7da8ff-a3e0-42e2-a369-6c6351686979-kube-api-access-pvjld\" (UniqueName: \"kubernetes.io/projected/8c7da8ff-a3e0-42e2-a369-6c6351686979-kube-api-access-pvjld\") pod \"8c7da8ff-a3e0-42e2-a369-6c6351686979\" (UID: \"8c7da8ff-a3e0-42e2-a369-6c6351686979\") " Oct 30 17:08:36.905815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:36.902302 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c7da8ff-a3e0-42e2-a369-6c6351686979-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:36.922304 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:36.912631 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7da8ff-a3e0-42e2-a369-6c6351686979-maas-api-xks-ca" pod "8c7da8ff-a3e0-42e2-a369-6c6351686979" (UID: "8c7da8ff-a3e0-42e2-a369-6c6351686979"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:08:36.922304 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:36.914029 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7da8ff-a3e0-42e2-a369-6c6351686979-kube-api-access-pvjld" pod "8c7da8ff-a3e0-42e2-a369-6c6351686979" (UID: "8c7da8ff-a3e0-42e2-a369-6c6351686979"). InnerVolumeSpecName "kube-api-access-pvjld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:37.004804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:37.004722 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvjld\" (UniqueName: \"kubernetes.io/projected/8c7da8ff-a3e0-42e2-a369-6c6351686979-kube-api-access-pvjld\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:37.004804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:37.004791 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c7da8ff-a3e0-42e2-a369-6c6351686979-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:37.458814 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:37.458779 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q" Oct 30 17:08:37.459029 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:37.458823 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q" event={"ID":"8c7da8ff-a3e0-42e2-a369-6c6351686979","Type":"ContainerDied","Data":"11dd8e5ce55a2801b863f05de2c7ac7c5630368423a5992d40aff02cda3d8db2"} Oct 30 17:08:37.459029 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:37.458876 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11dd8e5ce55a2801b863f05de2c7ac7c5630368423a5992d40aff02cda3d8db2" Oct 30 17:08:37.510095 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:37.510059 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd"] Oct 30 17:08:37.520546 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:37.520503 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8"] Oct 30 17:08:37.623002 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:37.621468 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:37.808456 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:37.808338 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:39.744338 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:39.744075 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:41.545492 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.545460 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn"] Oct 30 17:08:41.546996 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.546969 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8c7da8ff-a3e0-42e2-a369-6c6351686979" containerName="revoke-keys" Oct 30 17:08:41.546996 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.546992 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c7da8ff-a3e0-42e2-a369-6c6351686979" containerName="revoke-keys" Oct 30 17:08:41.547205 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.547180 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c7da8ff-a3e0-42e2-a369-6c6351686979" containerName="revoke-keys" Oct 30 17:08:41.552700 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.552675 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn" Oct 30 17:08:41.572378 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.572344 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn"] Oct 30 17:08:41.577086 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.577047 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:08:41.577265 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.577248 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:08:41.577350 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.577336 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:08:41.577427 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.577415 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:08:41.687540 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.687508 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43be2c46-9b99-4174-a409-c67eb7563505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1601645-4qpgn\" (UID: \"43be2c46-9b99-4174-a409-c67eb7563505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn" Oct 30 17:08:41.687706 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.687691 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kvbr\" (UniqueName: \"kubernetes.io/projected/43be2c46-9b99-4174-a409-c67eb7563505-kube-api-access-8kvbr\") pod \"maas-api-revoke-sub-b1601645-4qpgn\" (UID: \"43be2c46-9b99-4174-a409-c67eb7563505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn" Oct 30 17:08:41.687945 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.687758 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43be2c46-9b99-4174-a409-c67eb7563505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1601645-4qpgn\" (UID: \"43be2c46-9b99-4174-a409-c67eb7563505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn" Oct 30 17:08:41.789544 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.789504 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kvbr\" (UniqueName: \"kubernetes.io/projected/43be2c46-9b99-4174-a409-c67eb7563505-kube-api-access-8kvbr\") pod \"maas-api-revoke-sub-b1601645-4qpgn\" (UID: \"43be2c46-9b99-4174-a409-c67eb7563505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn" Oct 30 17:08:41.789734 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.789615 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43be2c46-9b99-4174-a409-c67eb7563505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1601645-4qpgn\" (UID: \"43be2c46-9b99-4174-a409-c67eb7563505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn" Oct 30 17:08:41.789842 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.789749 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43be2c46-9b99-4174-a409-c67eb7563505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1601645-4qpgn\" (UID: \"43be2c46-9b99-4174-a409-c67eb7563505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn" Oct 30 17:08:41.790331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.790306 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43be2c46-9b99-4174-a409-c67eb7563505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1601645-4qpgn\" (UID: \"43be2c46-9b99-4174-a409-c67eb7563505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn" Oct 30 17:08:41.792340 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.792319 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43be2c46-9b99-4174-a409-c67eb7563505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1601645-4qpgn\" (UID: \"43be2c46-9b99-4174-a409-c67eb7563505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn" Oct 30 17:08:41.813576 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.813517 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kvbr\" (UniqueName: \"kubernetes.io/projected/43be2c46-9b99-4174-a409-c67eb7563505-kube-api-access-8kvbr\") pod \"maas-api-revoke-sub-b1601645-4qpgn\" (UID: \"43be2c46-9b99-4174-a409-c67eb7563505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn" Oct 30 17:08:41.839752 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.839687 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:41.874665 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.874638 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn" Oct 30 17:08:41.953492 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:41.953438 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:42.187091 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:42.187048 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:42.234177 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:08:42.234141 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43be2c46_9b99_4174_a409_c67eb7563505.slice/crio-d9d3f495590e9f752edd83e5920f8040478b1154b3106cc32e23d12daec0a3bd WatchSource:0}: Error finding container d9d3f495590e9f752edd83e5920f8040478b1154b3106cc32e23d12daec0a3bd: Status 404 returned error can't find the container with id d9d3f495590e9f752edd83e5920f8040478b1154b3106cc32e23d12daec0a3bd Oct 30 17:08:42.236102 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:42.236078 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn"] Oct 30 17:08:42.525874 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:42.525745 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn" event={"ID":"43be2c46-9b99-4174-a409-c67eb7563505","Type":"ContainerStarted","Data":"d9d3f495590e9f752edd83e5920f8040478b1154b3106cc32e23d12daec0a3bd"} Oct 30 17:08:43.428803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:43.428722 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn"] Oct 30 17:08:43.462018 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:43.461979 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j"] Oct 30 17:08:43.556920 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:43.556882 6202 generic.go:356] "Generic (PLEG): container finished" podID="43be2c46-9b99-4174-a409-c67eb7563505" containerID="0c526cb1090d03e6480ec3b5b9c5f45d21c2f395c5ca863da1b877c29e7063b7" exitCode=0 Oct 30 17:08:43.557212 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:43.557188 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn" event={"ID":"43be2c46-9b99-4174-a409-c67eb7563505","Type":"ContainerDied","Data":"0c526cb1090d03e6480ec3b5b9c5f45d21c2f395c5ca863da1b877c29e7063b7"} Oct 30 17:08:43.569492 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:43.569421 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:43.702868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:43.691904 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:43.902262 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:43.902215 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:44.053876 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:44.048307 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:44.302322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:44.302084 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft" Oct 30 17:08:44.305679 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:44.305599 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj" Oct 30 17:08:44.310539 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:08:44.310501 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:08:44.367260 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:44.367220 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:44.483108 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:44.483075 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:44.788699 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:44.788670 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn" Oct 30 17:08:44.848792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:44.845480 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43be2c46-9b99-4174-a409-c67eb7563505-kube-api-access-8kvbr\" (UniqueName: \"kubernetes.io/projected/43be2c46-9b99-4174-a409-c67eb7563505-kube-api-access-8kvbr\") pod \"43be2c46-9b99-4174-a409-c67eb7563505\" (UID: \"43be2c46-9b99-4174-a409-c67eb7563505\") " Oct 30 17:08:44.848792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:44.845532 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43be2c46-9b99-4174-a409-c67eb7563505-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43be2c46-9b99-4174-a409-c67eb7563505-maas-api-xks-ca\") pod \"43be2c46-9b99-4174-a409-c67eb7563505\" (UID: \"43be2c46-9b99-4174-a409-c67eb7563505\") " Oct 30 17:08:44.848792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:44.845867 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43be2c46-9b99-4174-a409-c67eb7563505-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43be2c46-9b99-4174-a409-c67eb7563505-maas-api-openshift-ca\") pod \"43be2c46-9b99-4174-a409-c67eb7563505\" (UID: \"43be2c46-9b99-4174-a409-c67eb7563505\") " Oct 30 17:08:44.848792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:44.847105 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43be2c46-9b99-4174-a409-c67eb7563505-maas-api-openshift-ca" pod "43be2c46-9b99-4174-a409-c67eb7563505" (UID: "43be2c46-9b99-4174-a409-c67eb7563505"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:44.878544 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:44.861320 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43be2c46-9b99-4174-a409-c67eb7563505-maas-api-xks-ca" pod "43be2c46-9b99-4174-a409-c67eb7563505" (UID: "43be2c46-9b99-4174-a409-c67eb7563505"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:08:44.901637 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:44.897432 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43be2c46-9b99-4174-a409-c67eb7563505-kube-api-access-8kvbr" pod "43be2c46-9b99-4174-a409-c67eb7563505" (UID: "43be2c46-9b99-4174-a409-c67eb7563505"). InnerVolumeSpecName "kube-api-access-8kvbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:44.960803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:44.958160 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43be2c46-9b99-4174-a409-c67eb7563505-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:44.960803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:44.958192 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kvbr\" (UniqueName: \"kubernetes.io/projected/43be2c46-9b99-4174-a409-c67eb7563505-kube-api-access-8kvbr\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:44.960803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:44.958208 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43be2c46-9b99-4174-a409-c67eb7563505-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:45.577736 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:45.577706 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn" Oct 30 17:08:45.578283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:45.577740 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn" event={"ID":"43be2c46-9b99-4174-a409-c67eb7563505","Type":"ContainerDied","Data":"d9d3f495590e9f752edd83e5920f8040478b1154b3106cc32e23d12daec0a3bd"} Oct 30 17:08:45.578283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:45.577839 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9d3f495590e9f752edd83e5920f8040478b1154b3106cc32e23d12daec0a3bd" Oct 30 17:08:46.191438 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:46.191394 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:48.092492 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:48.092456 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:48.429122 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:48.429018 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" podUID="64359b58-0df8-41d1-9461-c87f3f6e6fa6" containerName="payload-processing" containerID="cri-o://0cbd64b6238566314d62f047e7d9d3bdb37424e497a9f43c9c13f2ec9197a7ab" gracePeriod=25 Oct 30 17:08:48.463079 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:48.463039 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" podUID="040124e8-ee08-4658-8088-a7563fd57c1b" containerName="payload-pre-processing" containerID="cri-o://6049d01c230b5f2608e81db63a56dffe38bb0b26517faab3b19bdf32476325e0" gracePeriod=25 Oct 30 17:08:49.991232 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:49.991186 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:51.709567 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:51.709531 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:51.824289 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:51.824257 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:51.833164 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:51.833130 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" podUID="040124e8-ee08-4658-8088-a7563fd57c1b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:08:51.907985 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:51.905274 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k"] Oct 30 17:08:51.907985 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:51.905908 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" podUID="64359b58-0df8-41d1-9461-c87f3f6e6fa6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:08:51.907985 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:51.907530 6202 cpu_manager.go:414] "containerMap: removing container" podUID="43be2c46-9b99-4174-a409-c67eb7563505" containerName="revoke-keys" Oct 30 17:08:51.907985 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:51.907552 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43be2c46-9b99-4174-a409-c67eb7563505" containerName="revoke-keys" Oct 30 17:08:51.910018 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:51.909991 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="43be2c46-9b99-4174-a409-c67eb7563505" containerName="revoke-keys" Oct 30 17:08:51.924136 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:51.924094 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k" Oct 30 17:08:51.944257 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:51.942062 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:08:51.945317 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:51.945265 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:51.952066 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:51.952034 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k"] Oct 30 17:08:51.953567 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:51.953543 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:08:51.954966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:51.953862 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:08:51.955106 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:51.953931 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:08:52.105065 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:52.105031 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-725f3b09-qk25k\" (UID: \"c63c6cc3-5fb1-46d2-8418-fd5fb737e585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k" Oct 30 17:08:52.105244 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:52.105206 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5d6t\" (UniqueName: \"kubernetes.io/projected/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-kube-api-access-m5d6t\") pod \"maas-api-revoke-sub-725f3b09-qk25k\" (UID: \"c63c6cc3-5fb1-46d2-8418-fd5fb737e585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k" Oct 30 17:08:52.105333 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:52.105315 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-725f3b09-qk25k\" (UID: \"c63c6cc3-5fb1-46d2-8418-fd5fb737e585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k" Oct 30 17:08:52.206988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:52.206945 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-725f3b09-qk25k\" (UID: \"c63c6cc3-5fb1-46d2-8418-fd5fb737e585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k" Oct 30 17:08:52.207167 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:52.207136 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-725f3b09-qk25k\" (UID: \"c63c6cc3-5fb1-46d2-8418-fd5fb737e585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k" Oct 30 17:08:52.207288 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:52.207272 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5d6t\" (UniqueName: \"kubernetes.io/projected/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-kube-api-access-m5d6t\") pod \"maas-api-revoke-sub-725f3b09-qk25k\" (UID: \"c63c6cc3-5fb1-46d2-8418-fd5fb737e585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k" Oct 30 17:08:52.207713 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:52.207694 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-725f3b09-qk25k\" (UID: \"c63c6cc3-5fb1-46d2-8418-fd5fb737e585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k" Oct 30 17:08:52.209440 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:52.209416 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-725f3b09-qk25k\" (UID: \"c63c6cc3-5fb1-46d2-8418-fd5fb737e585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k" Oct 30 17:08:52.230753 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:52.230701 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5d6t\" (UniqueName: \"kubernetes.io/projected/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-kube-api-access-m5d6t\") pod \"maas-api-revoke-sub-725f3b09-qk25k\" (UID: \"c63c6cc3-5fb1-46d2-8418-fd5fb737e585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k" Oct 30 17:08:52.275017 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:52.274982 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k" Oct 30 17:08:52.533953 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:52.533851 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" containerID="cri-o://4b5cc4e94f89ebfcea44f54ca22fc470dc0b551645f369cb543c966e5e6abb70" gracePeriod=45 Oct 30 17:08:52.542731 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:52.542690 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" containerID="cri-o://6e44b6d8fb7bd1165a810c718a9e100b2abbc0ab36e99d9f7ddc2f0d91749ba0" gracePeriod=45 Oct 30 17:08:52.543020 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:52.542989 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": dial tcp 10.133.0.51:8000: connect: connection refused" Oct 30 17:08:52.578514 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:52.578486 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k"] Oct 30 17:08:52.586694 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:08:52.586663 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63c6cc3_5fb1_46d2_8418_fd5fb737e585.slice/crio-e6ea905a25faa2791794ca64c3932d47f4f9af44afeba30c29b0d0e2f9be6c2c WatchSource:0}: Error finding container e6ea905a25faa2791794ca64c3932d47f4f9af44afeba30c29b0d0e2f9be6c2c: Status 404 returned error can't find the container with id e6ea905a25faa2791794ca64c3932d47f4f9af44afeba30c29b0d0e2f9be6c2c Oct 30 17:08:52.833155 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:52.833129 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 17:08:53.021053 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.021011 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-tmp-dir\") pod \"f04b04a7-bcaf-43da-9379-eed11564c668\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " Oct 30 17:08:53.021255 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.021199 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f04b04a7-bcaf-43da-9379-eed11564c668-tls-certs\" (UniqueName: \"kubernetes.io/secret/f04b04a7-bcaf-43da-9379-eed11564c668-tls-certs\") pod \"f04b04a7-bcaf-43da-9379-eed11564c668\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " Oct 30 17:08:53.021329 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.021259 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-tmp-dir" pod "f04b04a7-bcaf-43da-9379-eed11564c668" (UID: "f04b04a7-bcaf-43da-9379-eed11564c668"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:53.021409 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.021267 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04b04a7-bcaf-43da-9379-eed11564c668-kube-api-access-dvmbc\" (UniqueName: \"kubernetes.io/projected/f04b04a7-bcaf-43da-9379-eed11564c668-kube-api-access-dvmbc\") pod \"f04b04a7-bcaf-43da-9379-eed11564c668\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " Oct 30 17:08:53.021511 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.021489 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-dshm\") pod \"f04b04a7-bcaf-43da-9379-eed11564c668\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " Oct 30 17:08:53.021689 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.021549 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-home\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-home\") pod \"f04b04a7-bcaf-43da-9379-eed11564c668\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " Oct 30 17:08:53.021689 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.021607 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-model-cache\") pod \"f04b04a7-bcaf-43da-9379-eed11564c668\" (UID: \"f04b04a7-bcaf-43da-9379-eed11564c668\") " Oct 30 17:08:53.022358 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.022328 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-model-cache" pod "f04b04a7-bcaf-43da-9379-eed11564c668" (UID: "f04b04a7-bcaf-43da-9379-eed11564c668"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:53.022534 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.022514 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-home" pod "f04b04a7-bcaf-43da-9379-eed11564c668" (UID: "f04b04a7-bcaf-43da-9379-eed11564c668"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:53.023528 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.023485 6202 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-home\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.023528 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.023517 6202 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-model-cache\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.023695 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.023532 6202 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-tmp-dir\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.038564 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.038527 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f04b04a7-bcaf-43da-9379-eed11564c668-tls-certs" pod "f04b04a7-bcaf-43da-9379-eed11564c668" (UID: "f04b04a7-bcaf-43da-9379-eed11564c668"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:08:53.038915 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.038887 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04b04a7-bcaf-43da-9379-eed11564c668-kube-api-access-dvmbc" pod "f04b04a7-bcaf-43da-9379-eed11564c668" (UID: "f04b04a7-bcaf-43da-9379-eed11564c668"). InnerVolumeSpecName "kube-api-access-dvmbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:53.039052 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.039028 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-dshm" pod "f04b04a7-bcaf-43da-9379-eed11564c668" (UID: "f04b04a7-bcaf-43da-9379-eed11564c668"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:53.051149 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.051122 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 17:08:53.140795 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.136155 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvmbc\" (UniqueName: \"kubernetes.io/projected/f04b04a7-bcaf-43da-9379-eed11564c668-kube-api-access-dvmbc\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.140795 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.136190 6202 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f04b04a7-bcaf-43da-9379-eed11564c668-dshm\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.140795 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.136207 6202 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f04b04a7-bcaf-43da-9379-eed11564c668-tls-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.275578 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.266481 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-kube-api-access-4zg24\" (UniqueName: \"kubernetes.io/projected/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-kube-api-access-4zg24\") pod \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " Oct 30 17:08:53.275578 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.266560 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-home\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-home\") pod \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " Oct 30 17:08:53.275578 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.266856 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-tmp-dir\") pod \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " Oct 30 17:08:53.275578 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.266879 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-home" pod "4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" (UID: "4a27d606-60b7-43e4-b8ec-3ca9bb95a51c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:53.275578 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.266909 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-tls-certs\") pod \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " Oct 30 17:08:53.275578 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.267070 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-dshm\") pod \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " Oct 30 17:08:53.275578 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.267139 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-tmp-dir" pod "4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" (UID: "4a27d606-60b7-43e4-b8ec-3ca9bb95a51c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:53.275578 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.267231 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-model-cache\") pod \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\" (UID: \"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c\") " Oct 30 17:08:53.275578 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.267928 6202 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-home\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.275578 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.267952 6202 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-tmp-dir\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.275578 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.268176 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-model-cache" pod "4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" (UID: "4a27d606-60b7-43e4-b8ec-3ca9bb95a51c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:53.332991 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.332952 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-kube-api-access-4zg24" pod "4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" (UID: "4a27d606-60b7-43e4-b8ec-3ca9bb95a51c"). InnerVolumeSpecName "kube-api-access-4zg24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:53.333204 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.333102 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-tls-certs" pod "4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" (UID: "4a27d606-60b7-43e4-b8ec-3ca9bb95a51c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:08:53.333544 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.333521 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-dshm" pod "4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" (UID: "4a27d606-60b7-43e4-b8ec-3ca9bb95a51c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:53.369619 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.369580 6202 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-dshm\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.369619 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.369614 6202 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-model-cache\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.369884 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.369625 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zg24\" (UniqueName: \"kubernetes.io/projected/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-kube-api-access-4zg24\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.369884 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.369638 6202 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c-tls-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.652812 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.652737 6202 generic.go:356] "Generic (PLEG): container finished" podID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerID="6e44b6d8fb7bd1165a810c718a9e100b2abbc0ab36e99d9f7ddc2f0d91749ba0" exitCode=0 Oct 30 17:08:53.652812 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.652805 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" Oct 30 17:08:53.653083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.652822 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" event={"ID":"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c","Type":"ContainerDied","Data":"6e44b6d8fb7bd1165a810c718a9e100b2abbc0ab36e99d9f7ddc2f0d91749ba0"} Oct 30 17:08:53.653083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.652878 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" event={"ID":"4a27d606-60b7-43e4-b8ec-3ca9bb95a51c","Type":"ContainerDied","Data":"1f7f5e3740dea46b86beb1afa71fc7b9b909137d9a8ed30ba6f28b7895f75eac"} Oct 30 17:08:53.653083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.652902 6202 scope.go:122] "RemoveContainer" containerID="6e44b6d8fb7bd1165a810c718a9e100b2abbc0ab36e99d9f7ddc2f0d91749ba0" Oct 30 17:08:53.656116 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.655373 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k" event={"ID":"c63c6cc3-5fb1-46d2-8418-fd5fb737e585","Type":"ContainerStarted","Data":"74e95b718948f2ba46c804eff67fffee6281c7540f591999e4fb27cb30ebde4b"} Oct 30 17:08:53.656116 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.655426 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k" event={"ID":"c63c6cc3-5fb1-46d2-8418-fd5fb737e585","Type":"ContainerStarted","Data":"e6ea905a25faa2791794ca64c3932d47f4f9af44afeba30c29b0d0e2f9be6c2c"} Oct 30 17:08:53.657374 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.657349 6202 generic.go:356] "Generic (PLEG): container finished" podID="f04b04a7-bcaf-43da-9379-eed11564c668" containerID="4b5cc4e94f89ebfcea44f54ca22fc470dc0b551645f369cb543c966e5e6abb70" exitCode=0 Oct 30 17:08:53.657374 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.657410 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" Oct 30 17:08:53.657374 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.657414 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" event={"ID":"f04b04a7-bcaf-43da-9379-eed11564c668","Type":"ContainerDied","Data":"4b5cc4e94f89ebfcea44f54ca22fc470dc0b551645f369cb543c966e5e6abb70"} Oct 30 17:08:53.657374 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.657456 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd" event={"ID":"f04b04a7-bcaf-43da-9379-eed11564c668","Type":"ContainerDied","Data":"f80d9f002a4eb65273285a20cd40bb514c0e117a8e2bd0d231b3bc164d4bebb1"} Oct 30 17:08:53.668750 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.668724 6202 scope.go:122] "RemoveContainer" containerID="6e44b6d8fb7bd1165a810c718a9e100b2abbc0ab36e99d9f7ddc2f0d91749ba0" Oct 30 17:08:53.669949 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:08:53.669918 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e44b6d8fb7bd1165a810c718a9e100b2abbc0ab36e99d9f7ddc2f0d91749ba0\": container with ID starting with 6e44b6d8fb7bd1165a810c718a9e100b2abbc0ab36e99d9f7ddc2f0d91749ba0 not found: ID does not exist" containerID="6e44b6d8fb7bd1165a810c718a9e100b2abbc0ab36e99d9f7ddc2f0d91749ba0" Oct 30 17:08:53.670072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.669960 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e44b6d8fb7bd1165a810c718a9e100b2abbc0ab36e99d9f7ddc2f0d91749ba0"} err="failed to get container status \"6e44b6d8fb7bd1165a810c718a9e100b2abbc0ab36e99d9f7ddc2f0d91749ba0\": rpc error: code = NotFound desc = could not find container \"6e44b6d8fb7bd1165a810c718a9e100b2abbc0ab36e99d9f7ddc2f0d91749ba0\": container with ID starting with 6e44b6d8fb7bd1165a810c718a9e100b2abbc0ab36e99d9f7ddc2f0d91749ba0 not found: ID does not exist" Oct 30 17:08:53.670072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.669986 6202 scope.go:122] "RemoveContainer" containerID="4b5cc4e94f89ebfcea44f54ca22fc470dc0b551645f369cb543c966e5e6abb70" Oct 30 17:08:53.711264 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.711211 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k" podStartSLOduration=2.711195375 podStartE2EDuration="2.711195375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:53.707998081 +0000 UTC m=+2847.418452825" watchObservedRunningTime="2026-10-30 17:08:53.711195375 +0000 UTC m=+2847.421650118" Oct 30 17:08:53.731820 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.731798 6202 scope.go:122] "RemoveContainer" containerID="4b5cc4e94f89ebfcea44f54ca22fc470dc0b551645f369cb543c966e5e6abb70" Oct 30 17:08:53.732145 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:08:53.732122 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b5cc4e94f89ebfcea44f54ca22fc470dc0b551645f369cb543c966e5e6abb70\": container with ID starting with 4b5cc4e94f89ebfcea44f54ca22fc470dc0b551645f369cb543c966e5e6abb70 not found: ID does not exist" containerID="4b5cc4e94f89ebfcea44f54ca22fc470dc0b551645f369cb543c966e5e6abb70" Oct 30 17:08:53.732240 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.732156 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5cc4e94f89ebfcea44f54ca22fc470dc0b551645f369cb543c966e5e6abb70"} err="failed to get container status \"4b5cc4e94f89ebfcea44f54ca22fc470dc0b551645f369cb543c966e5e6abb70\": rpc error: code = NotFound desc = could not find container \"4b5cc4e94f89ebfcea44f54ca22fc470dc0b551645f369cb543c966e5e6abb70\": container with ID starting with 4b5cc4e94f89ebfcea44f54ca22fc470dc0b551645f369cb543c966e5e6abb70 not found: ID does not exist" Oct 30 17:08:53.743669 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.743641 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:53.746501 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.746471 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8"] Oct 30 17:08:53.751789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.751748 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8"] Oct 30 17:08:53.817798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.812030 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd"] Oct 30 17:08:53.824342 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:53.824231 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/premium-w6-d693d0-kserve-787f95c794-xxrkd"] Oct 30 17:08:54.037149 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:54.035153 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/simulator-w6-d693d0-kserve-69d5ff7bbc-z44v8" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:08:54.412501 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:54.412458 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" path="/var/lib/kubelet/pods/4a27d606-60b7-43e4-b8ec-3ca9bb95a51c/volumes" Oct 30 17:08:54.413103 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:54.413078 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" path="/var/lib/kubelet/pods/f04b04a7-bcaf-43da-9379-eed11564c668/volumes" Oct 30 17:08:54.670970 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:54.670879 6202 generic.go:356] "Generic (PLEG): container finished" podID="c63c6cc3-5fb1-46d2-8418-fd5fb737e585" containerID="74e95b718948f2ba46c804eff67fffee6281c7540f591999e4fb27cb30ebde4b" exitCode=0 Oct 30 17:08:54.671168 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:54.670965 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k" event={"ID":"c63c6cc3-5fb1-46d2-8418-fd5fb737e585","Type":"ContainerDied","Data":"74e95b718948f2ba46c804eff67fffee6281c7540f591999e4fb27cb30ebde4b"} Oct 30 17:08:55.114290 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.114256 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:55.323798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.320811 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:55.738664 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.738635 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr"] Oct 30 17:08:55.740316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.740289 6202 cpu_manager.go:414] "containerMap: removing container" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" Oct 30 17:08:55.740479 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.740466 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" Oct 30 17:08:55.740632 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.740621 6202 cpu_manager.go:414] "containerMap: removing container" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" Oct 30 17:08:55.740706 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.740696 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" Oct 30 17:08:55.741055 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.741037 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a27d606-60b7-43e4-b8ec-3ca9bb95a51c" containerName="main" Oct 30 17:08:55.741178 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.741168 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04b04a7-bcaf-43da-9379-eed11564c668" containerName="main" Oct 30 17:08:55.747518 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.747493 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr" Oct 30 17:08:55.753521 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.753433 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr"] Oct 30 17:08:55.790792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.783639 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-330282-dockercfg-6dlgc\"" Oct 30 17:08:55.790792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.783779 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-330282\"" Oct 30 17:08:55.841894 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.838704 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h26vc\" (UniqueName: \"kubernetes.io/projected/a1bab548-0f80-4d1b-ba3a-d63c8fff6eee-kube-api-access-h26vc\") pod \"payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr\" (UID: \"a1bab548-0f80-4d1b-ba3a-d63c8fff6eee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr" Oct 30 17:08:55.841894 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.838952 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1bab548-0f80-4d1b-ba3a-d63c8fff6eee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr\" (UID: \"a1bab548-0f80-4d1b-ba3a-d63c8fff6eee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr" Oct 30 17:08:55.862362 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.862308 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv"] Oct 30 17:08:55.871932 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.871876 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv" Oct 30 17:08:55.889428 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.887188 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv"] Oct 30 17:08:55.940400 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.940356 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h26vc\" (UniqueName: \"kubernetes.io/projected/a1bab548-0f80-4d1b-ba3a-d63c8fff6eee-kube-api-access-h26vc\") pod \"payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr\" (UID: \"a1bab548-0f80-4d1b-ba3a-d63c8fff6eee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr" Oct 30 17:08:55.940586 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.940552 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1bab548-0f80-4d1b-ba3a-d63c8fff6eee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr\" (UID: \"a1bab548-0f80-4d1b-ba3a-d63c8fff6eee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr" Oct 30 17:08:55.940679 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.940608 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhxg5\" (UniqueName: \"kubernetes.io/projected/81111551-33d9-4996-bc84-d7b27e6ab40c-kube-api-access-dhxg5\") pod \"payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv\" (UID: \"81111551-33d9-4996-bc84-d7b27e6ab40c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv" Oct 30 17:08:55.940747 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.940703 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81111551-33d9-4996-bc84-d7b27e6ab40c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv\" (UID: \"81111551-33d9-4996-bc84-d7b27e6ab40c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv" Oct 30 17:08:55.941820 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.941789 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1bab548-0f80-4d1b-ba3a-d63c8fff6eee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr\" (UID: \"a1bab548-0f80-4d1b-ba3a-d63c8fff6eee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr" Oct 30 17:08:55.965321 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:55.965227 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h26vc\" (UniqueName: \"kubernetes.io/projected/a1bab548-0f80-4d1b-ba3a-d63c8fff6eee-kube-api-access-h26vc\") pod \"payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr\" (UID: \"a1bab548-0f80-4d1b-ba3a-d63c8fff6eee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr" Oct 30 17:08:56.012328 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.012301 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k" Oct 30 17:08:56.042513 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.042464 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-kube-api-access-m5d6t\" (UniqueName: \"kubernetes.io/projected/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-kube-api-access-m5d6t\") pod \"c63c6cc3-5fb1-46d2-8418-fd5fb737e585\" (UID: \"c63c6cc3-5fb1-46d2-8418-fd5fb737e585\") " Oct 30 17:08:56.042721 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.042566 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-maas-api-xks-ca\") pod \"c63c6cc3-5fb1-46d2-8418-fd5fb737e585\" (UID: \"c63c6cc3-5fb1-46d2-8418-fd5fb737e585\") " Oct 30 17:08:56.042721 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.042627 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-maas-api-openshift-ca\") pod \"c63c6cc3-5fb1-46d2-8418-fd5fb737e585\" (UID: \"c63c6cc3-5fb1-46d2-8418-fd5fb737e585\") " Oct 30 17:08:56.043093 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.043064 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-maas-api-openshift-ca" pod "c63c6cc3-5fb1-46d2-8418-fd5fb737e585" (UID: "c63c6cc3-5fb1-46d2-8418-fd5fb737e585"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:56.043332 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.043223 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhxg5\" (UniqueName: \"kubernetes.io/projected/81111551-33d9-4996-bc84-d7b27e6ab40c-kube-api-access-dhxg5\") pod \"payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv\" (UID: \"81111551-33d9-4996-bc84-d7b27e6ab40c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv" Oct 30 17:08:56.043519 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.043500 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81111551-33d9-4996-bc84-d7b27e6ab40c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv\" (UID: \"81111551-33d9-4996-bc84-d7b27e6ab40c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv" Oct 30 17:08:56.043669 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.043652 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:56.044432 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.044407 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81111551-33d9-4996-bc84-d7b27e6ab40c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv\" (UID: \"81111551-33d9-4996-bc84-d7b27e6ab40c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv" Oct 30 17:08:56.044966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.044941 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-maas-api-xks-ca" pod "c63c6cc3-5fb1-46d2-8418-fd5fb737e585" (UID: "c63c6cc3-5fb1-46d2-8418-fd5fb737e585"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:08:56.045141 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.045116 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-kube-api-access-m5d6t" pod "c63c6cc3-5fb1-46d2-8418-fd5fb737e585" (UID: "c63c6cc3-5fb1-46d2-8418-fd5fb737e585"). InnerVolumeSpecName "kube-api-access-m5d6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:56.070675 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.070189 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhxg5\" (UniqueName: \"kubernetes.io/projected/81111551-33d9-4996-bc84-d7b27e6ab40c-kube-api-access-dhxg5\") pod \"payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv\" (UID: \"81111551-33d9-4996-bc84-d7b27e6ab40c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv" Oct 30 17:08:56.145239 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.145200 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5d6t\" (UniqueName: \"kubernetes.io/projected/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-kube-api-access-m5d6t\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:56.145239 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.145240 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63c6cc3-5fb1-46d2-8418-fd5fb737e585-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:08:56.149990 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.149962 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr" Oct 30 17:08:56.199593 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.199266 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv" Oct 30 17:08:56.461023 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:08:56.460972 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bab548_0f80_4d1b_ba3a_d63c8fff6eee.slice/crio-7d8f1fb4aff7ac612fa4acfcfb17bec8e3608dc93e96f5c615760e6d71f0fedd WatchSource:0}: Error finding container 7d8f1fb4aff7ac612fa4acfcfb17bec8e3608dc93e96f5c615760e6d71f0fedd: Status 404 returned error can't find the container with id 7d8f1fb4aff7ac612fa4acfcfb17bec8e3608dc93e96f5c615760e6d71f0fedd Oct 30 17:08:56.461705 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.461680 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr"] Oct 30 17:08:56.558856 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:08:56.558820 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81111551_33d9_4996_bc84_d7b27e6ab40c.slice/crio-da7ded078c64b0e9c01e12258698eec8c7af4e6d3b52d98bb199e5a33beea45a WatchSource:0}: Error finding container da7ded078c64b0e9c01e12258698eec8c7af4e6d3b52d98bb199e5a33beea45a: Status 404 returned error can't find the container with id da7ded078c64b0e9c01e12258698eec8c7af4e6d3b52d98bb199e5a33beea45a Oct 30 17:08:56.559082 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.559063 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv"] Oct 30 17:08:56.582477 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.582384 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:56.697313 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.697282 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:56.717843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.717758 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k" event={"ID":"c63c6cc3-5fb1-46d2-8418-fd5fb737e585","Type":"ContainerDied","Data":"e6ea905a25faa2791794ca64c3932d47f4f9af44afeba30c29b0d0e2f9be6c2c"} Oct 30 17:08:56.718003 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.717868 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6ea905a25faa2791794ca64c3932d47f4f9af44afeba30c29b0d0e2f9be6c2c" Oct 30 17:08:56.718003 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.717935 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k" Oct 30 17:08:56.723915 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.723871 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr" event={"ID":"a1bab548-0f80-4d1b-ba3a-d63c8fff6eee","Type":"ContainerStarted","Data":"ff7e6986241e394280180c8179aff6285ca1247243f0362bf3b2908a13635b58"} Oct 30 17:08:56.724088 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.723929 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr" event={"ID":"a1bab548-0f80-4d1b-ba3a-d63c8fff6eee","Type":"ContainerStarted","Data":"7d8f1fb4aff7ac612fa4acfcfb17bec8e3608dc93e96f5c615760e6d71f0fedd"} Oct 30 17:08:56.725819 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.725793 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv" event={"ID":"81111551-33d9-4996-bc84-d7b27e6ab40c","Type":"ContainerStarted","Data":"32a0721c8542123e42a450bffc478c8bb61af4a1b680bb5d27efb642caae5302"} Oct 30 17:08:56.725932 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.725843 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv" event={"ID":"81111551-33d9-4996-bc84-d7b27e6ab40c","Type":"ContainerStarted","Data":"da7ded078c64b0e9c01e12258698eec8c7af4e6d3b52d98bb199e5a33beea45a"} Oct 30 17:08:56.726560 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.726542 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv" Oct 30 17:08:56.785015 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.784954 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr" podStartSLOduration=1.78493522 podStartE2EDuration="1.78493522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:56.780631086 +0000 UTC m=+2850.491085831" watchObservedRunningTime="2026-10-30 17:08:56.78493522 +0000 UTC m=+2850.495389966" Oct 30 17:08:56.822424 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:56.822365 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv" podStartSLOduration=1.8223395610000002 podStartE2EDuration="1.822339561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:56.819908159 +0000 UTC m=+2850.530362903" watchObservedRunningTime="2026-10-30 17:08:56.822339561 +0000 UTC m=+2850.532794305" Oct 30 17:08:57.738203 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:57.738169 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr" Oct 30 17:08:58.312414 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:08:58.312379 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:00.610453 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:00.610417 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:01.870790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:01.861864 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" podUID="040124e8-ee08-4658-8088-a7563fd57c1b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:09:01.880796 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:01.874909 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" podUID="64359b58-0df8-41d1-9461-c87f3f6e6fa6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:09:02.745825 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:02.745786 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:02.839409 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:02.839370 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q"] Oct 30 17:09:02.840675 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:02.840648 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c63c6cc3-5fb1-46d2-8418-fd5fb737e585" containerName="revoke-keys" Oct 30 17:09:02.840675 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:02.840674 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63c6cc3-5fb1-46d2-8418-fd5fb737e585" containerName="revoke-keys" Oct 30 17:09:02.840942 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:02.840927 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63c6cc3-5fb1-46d2-8418-fd5fb737e585" containerName="revoke-keys" Oct 30 17:09:02.846050 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:02.846029 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q" Oct 30 17:09:02.854516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:02.854488 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q"] Oct 30 17:09:02.861026 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:02.861004 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:09:02.878246 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:02.877331 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:09:02.878246 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:02.877430 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:09:02.878246 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:02.877503 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:09:02.999301 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:02.999206 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa9db899-8ed9-4944-8c2c-9ffea59595d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96da0afe-rgg4q\" (UID: \"fa9db899-8ed9-4944-8c2c-9ffea59595d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q" Oct 30 17:09:02.999495 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:02.999365 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh6vg\" (UniqueName: \"kubernetes.io/projected/fa9db899-8ed9-4944-8c2c-9ffea59595d1-kube-api-access-kh6vg\") pod \"maas-api-revoke-sub-96da0afe-rgg4q\" (UID: \"fa9db899-8ed9-4944-8c2c-9ffea59595d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q" Oct 30 17:09:02.999696 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:02.999674 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa9db899-8ed9-4944-8c2c-9ffea59595d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96da0afe-rgg4q\" (UID: \"fa9db899-8ed9-4944-8c2c-9ffea59595d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q" Oct 30 17:09:03.101647 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:03.101611 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa9db899-8ed9-4944-8c2c-9ffea59595d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96da0afe-rgg4q\" (UID: \"fa9db899-8ed9-4944-8c2c-9ffea59595d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q" Oct 30 17:09:03.101863 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:03.101670 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa9db899-8ed9-4944-8c2c-9ffea59595d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96da0afe-rgg4q\" (UID: \"fa9db899-8ed9-4944-8c2c-9ffea59595d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q" Oct 30 17:09:03.101863 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:03.101756 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh6vg\" (UniqueName: \"kubernetes.io/projected/fa9db899-8ed9-4944-8c2c-9ffea59595d1-kube-api-access-kh6vg\") pod \"maas-api-revoke-sub-96da0afe-rgg4q\" (UID: \"fa9db899-8ed9-4944-8c2c-9ffea59595d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q" Oct 30 17:09:03.102379 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:03.102355 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa9db899-8ed9-4944-8c2c-9ffea59595d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96da0afe-rgg4q\" (UID: \"fa9db899-8ed9-4944-8c2c-9ffea59595d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q" Oct 30 17:09:03.104838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:03.104808 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa9db899-8ed9-4944-8c2c-9ffea59595d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96da0afe-rgg4q\" (UID: \"fa9db899-8ed9-4944-8c2c-9ffea59595d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q" Oct 30 17:09:03.125635 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:03.125603 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh6vg\" (UniqueName: \"kubernetes.io/projected/fa9db899-8ed9-4944-8c2c-9ffea59595d1-kube-api-access-kh6vg\") pod \"maas-api-revoke-sub-96da0afe-rgg4q\" (UID: \"fa9db899-8ed9-4944-8c2c-9ffea59595d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q" Oct 30 17:09:03.166715 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:03.166686 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q" Oct 30 17:09:03.566308 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:03.560163 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q"] Oct 30 17:09:03.811100 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:03.811063 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q" event={"ID":"fa9db899-8ed9-4944-8c2c-9ffea59595d1","Type":"ContainerStarted","Data":"aba728430a2403686f36692507800ca242f667293c1f01f73117dd8c9897bd45"} Oct 30 17:09:04.682898 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:04.682820 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:04.822637 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:04.822600 6202 generic.go:356] "Generic (PLEG): container finished" podID="fa9db899-8ed9-4944-8c2c-9ffea59595d1" containerID="3d1a586e7c2ef804c9a6a10576aa413db9b40b9e2d6884182dbacb48fc66e236" exitCode=0 Oct 30 17:09:04.822842 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:04.822664 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q" event={"ID":"fa9db899-8ed9-4944-8c2c-9ffea59595d1","Type":"ContainerDied","Data":"3d1a586e7c2ef804c9a6a10576aa413db9b40b9e2d6884182dbacb48fc66e236"} Oct 30 17:09:06.024242 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:06.024213 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q" Oct 30 17:09:06.160496 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:06.160410 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa9db899-8ed9-4944-8c2c-9ffea59595d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa9db899-8ed9-4944-8c2c-9ffea59595d1-maas-api-xks-ca\") pod \"fa9db899-8ed9-4944-8c2c-9ffea59595d1\" (UID: \"fa9db899-8ed9-4944-8c2c-9ffea59595d1\") " Oct 30 17:09:06.160496 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:06.160474 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa9db899-8ed9-4944-8c2c-9ffea59595d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa9db899-8ed9-4944-8c2c-9ffea59595d1-maas-api-openshift-ca\") pod \"fa9db899-8ed9-4944-8c2c-9ffea59595d1\" (UID: \"fa9db899-8ed9-4944-8c2c-9ffea59595d1\") " Oct 30 17:09:06.160706 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:06.160583 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa9db899-8ed9-4944-8c2c-9ffea59595d1-kube-api-access-kh6vg\" (UniqueName: \"kubernetes.io/projected/fa9db899-8ed9-4944-8c2c-9ffea59595d1-kube-api-access-kh6vg\") pod \"fa9db899-8ed9-4944-8c2c-9ffea59595d1\" (UID: \"fa9db899-8ed9-4944-8c2c-9ffea59595d1\") " Oct 30 17:09:06.160922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:06.160888 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa9db899-8ed9-4944-8c2c-9ffea59595d1-maas-api-openshift-ca" pod "fa9db899-8ed9-4944-8c2c-9ffea59595d1" (UID: "fa9db899-8ed9-4944-8c2c-9ffea59595d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:06.161332 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:06.161311 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa9db899-8ed9-4944-8c2c-9ffea59595d1-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:06.162806 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:06.162753 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa9db899-8ed9-4944-8c2c-9ffea59595d1-kube-api-access-kh6vg" pod "fa9db899-8ed9-4944-8c2c-9ffea59595d1" (UID: "fa9db899-8ed9-4944-8c2c-9ffea59595d1"). InnerVolumeSpecName "kube-api-access-kh6vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:06.162918 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:06.162906 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa9db899-8ed9-4944-8c2c-9ffea59595d1-maas-api-xks-ca" pod "fa9db899-8ed9-4944-8c2c-9ffea59595d1" (UID: "fa9db899-8ed9-4944-8c2c-9ffea59595d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:09:06.263350 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:06.263310 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa9db899-8ed9-4944-8c2c-9ffea59595d1-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:06.263350 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:06.263340 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh6vg\" (UniqueName: \"kubernetes.io/projected/fa9db899-8ed9-4944-8c2c-9ffea59595d1-kube-api-access-kh6vg\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:06.415342 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:06.415246 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:06.848856 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:06.848825 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q" Oct 30 17:09:06.848856 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:06.848843 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q" event={"ID":"fa9db899-8ed9-4944-8c2c-9ffea59595d1","Type":"ContainerDied","Data":"aba728430a2403686f36692507800ca242f667293c1f01f73117dd8c9897bd45"} Oct 30 17:09:06.849117 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:06.848896 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aba728430a2403686f36692507800ca242f667293c1f01f73117dd8c9897bd45" Oct 30 17:09:08.631074 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.631041 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:08.753149 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.753117 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr" Oct 30 17:09:08.753727 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.753693 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv" Oct 30 17:09:08.869210 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.868601 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" Oct 30 17:09:08.873816 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.873759 6202 generic.go:356] "Generic (PLEG): container finished" podID="64359b58-0df8-41d1-9461-c87f3f6e6fa6" containerID="0cbd64b6238566314d62f047e7d9d3bdb37424e497a9f43c9c13f2ec9197a7ab" exitCode=0 Oct 30 17:09:08.873982 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.873919 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" event={"ID":"64359b58-0df8-41d1-9461-c87f3f6e6fa6","Type":"ContainerDied","Data":"0cbd64b6238566314d62f047e7d9d3bdb37424e497a9f43c9c13f2ec9197a7ab"} Oct 30 17:09:08.873982 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.873971 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" event={"ID":"64359b58-0df8-41d1-9461-c87f3f6e6fa6","Type":"ContainerDied","Data":"d651b8ee5cc90b651db6f027b66e824b4a91289f8515d7a618a293a6650be80b"} Oct 30 17:09:08.874104 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.874003 6202 scope.go:122] "RemoveContainer" containerID="0cbd64b6238566314d62f047e7d9d3bdb37424e497a9f43c9c13f2ec9197a7ab" Oct 30 17:09:08.874171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.874151 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn" Oct 30 17:09:08.878486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.878438 6202 generic.go:356] "Generic (PLEG): container finished" podID="040124e8-ee08-4658-8088-a7563fd57c1b" containerID="6049d01c230b5f2608e81db63a56dffe38bb0b26517faab3b19bdf32476325e0" exitCode=0 Oct 30 17:09:08.883192 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.878520 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" event={"ID":"040124e8-ee08-4658-8088-a7563fd57c1b","Type":"ContainerDied","Data":"6049d01c230b5f2608e81db63a56dffe38bb0b26517faab3b19bdf32476325e0"} Oct 30 17:09:08.883192 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.883049 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" event={"ID":"040124e8-ee08-4658-8088-a7563fd57c1b","Type":"ContainerDied","Data":"9c0d2dcc7cbd4ae050ce62ed76c0977a31f81c942b1907f65e0d19abf8b64917"} Oct 30 17:09:08.883192 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.883078 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c0d2dcc7cbd4ae050ce62ed76c0977a31f81c942b1907f65e0d19abf8b64917" Oct 30 17:09:08.883192 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.879660 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" Oct 30 17:09:08.890983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.890961 6202 scope.go:122] "RemoveContainer" containerID="0cbd64b6238566314d62f047e7d9d3bdb37424e497a9f43c9c13f2ec9197a7ab" Oct 30 17:09:08.891934 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:09:08.891902 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cbd64b6238566314d62f047e7d9d3bdb37424e497a9f43c9c13f2ec9197a7ab\": container with ID starting with 0cbd64b6238566314d62f047e7d9d3bdb37424e497a9f43c9c13f2ec9197a7ab not found: ID does not exist" containerID="0cbd64b6238566314d62f047e7d9d3bdb37424e497a9f43c9c13f2ec9197a7ab" Oct 30 17:09:08.892058 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.891940 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbd64b6238566314d62f047e7d9d3bdb37424e497a9f43c9c13f2ec9197a7ab"} err="failed to get container status \"0cbd64b6238566314d62f047e7d9d3bdb37424e497a9f43c9c13f2ec9197a7ab\": rpc error: code = NotFound desc = could not find container \"0cbd64b6238566314d62f047e7d9d3bdb37424e497a9f43c9c13f2ec9197a7ab\": container with ID starting with 0cbd64b6238566314d62f047e7d9d3bdb37424e497a9f43c9c13f2ec9197a7ab not found: ID does not exist" Oct 30 17:09:08.917814 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.917358 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64359b58-0df8-41d1-9461-c87f3f6e6fa6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64359b58-0df8-41d1-9461-c87f3f6e6fa6-plugins-config-volume\") pod \"64359b58-0df8-41d1-9461-c87f3f6e6fa6\" (UID: \"64359b58-0df8-41d1-9461-c87f3f6e6fa6\") " Oct 30 17:09:08.918001 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.917840 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64359b58-0df8-41d1-9461-c87f3f6e6fa6-plugins-config-volume" pod "64359b58-0df8-41d1-9461-c87f3f6e6fa6" (UID: "64359b58-0df8-41d1-9461-c87f3f6e6fa6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:08.918472 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:08.918441 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64359b58-0df8-41d1-9461-c87f3f6e6fa6-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:09.019289 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:09.019189 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/040124e8-ee08-4658-8088-a7563fd57c1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/040124e8-ee08-4658-8088-a7563fd57c1b-plugins-config-volume\") pod \"040124e8-ee08-4658-8088-a7563fd57c1b\" (UID: \"040124e8-ee08-4658-8088-a7563fd57c1b\") " Oct 30 17:09:09.019476 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:09.019311 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040124e8-ee08-4658-8088-a7563fd57c1b-kube-api-access-v4qpn\" (UniqueName: \"kubernetes.io/projected/040124e8-ee08-4658-8088-a7563fd57c1b-kube-api-access-v4qpn\") pod \"040124e8-ee08-4658-8088-a7563fd57c1b\" (UID: \"040124e8-ee08-4658-8088-a7563fd57c1b\") " Oct 30 17:09:09.019476 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:09.019445 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64359b58-0df8-41d1-9461-c87f3f6e6fa6-kube-api-access-z6zfn\" (UniqueName: \"kubernetes.io/projected/64359b58-0df8-41d1-9461-c87f3f6e6fa6-kube-api-access-z6zfn\") pod \"64359b58-0df8-41d1-9461-c87f3f6e6fa6\" (UID: \"64359b58-0df8-41d1-9461-c87f3f6e6fa6\") " Oct 30 17:09:09.019678 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:09.019644 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/040124e8-ee08-4658-8088-a7563fd57c1b-plugins-config-volume" pod "040124e8-ee08-4658-8088-a7563fd57c1b" (UID: "040124e8-ee08-4658-8088-a7563fd57c1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:09.020342 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:09.020251 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/040124e8-ee08-4658-8088-a7563fd57c1b-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:09.022176 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:09.022142 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64359b58-0df8-41d1-9461-c87f3f6e6fa6-kube-api-access-z6zfn" pod "64359b58-0df8-41d1-9461-c87f3f6e6fa6" (UID: "64359b58-0df8-41d1-9461-c87f3f6e6fa6"). InnerVolumeSpecName "kube-api-access-z6zfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:09.022176 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:09.022173 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040124e8-ee08-4658-8088-a7563fd57c1b-kube-api-access-v4qpn" pod "040124e8-ee08-4658-8088-a7563fd57c1b" (UID: "040124e8-ee08-4658-8088-a7563fd57c1b"). InnerVolumeSpecName "kube-api-access-v4qpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:09.121266 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:09.121235 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4qpn\" (UniqueName: \"kubernetes.io/projected/040124e8-ee08-4658-8088-a7563fd57c1b-kube-api-access-v4qpn\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:09.121266 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:09.121264 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6zfn\" (UniqueName: \"kubernetes.io/projected/64359b58-0df8-41d1-9461-c87f3f6e6fa6-kube-api-access-z6zfn\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:09.229509 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:09.229469 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn"] Oct 30 17:09:09.233824 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:09.233796 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-d693d0-6f45b86d8b-xjtmn"] Oct 30 17:09:09.893218 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:09.893190 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j" Oct 30 17:09:09.945881 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:09.945847 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j"] Oct 30 17:09:09.957423 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:09.957393 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-d693d0-64d59cd5bc-rlv9j"] Oct 30 17:09:10.086029 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.085991 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g"] Oct 30 17:09:10.087160 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.087141 6202 cpu_manager.go:414] "containerMap: removing container" podUID="64359b58-0df8-41d1-9461-c87f3f6e6fa6" containerName="payload-processing" Oct 30 17:09:10.087232 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.087162 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64359b58-0df8-41d1-9461-c87f3f6e6fa6" containerName="payload-processing" Oct 30 17:09:10.087232 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.087204 6202 cpu_manager.go:414] "containerMap: removing container" podUID="fa9db899-8ed9-4944-8c2c-9ffea59595d1" containerName="revoke-keys" Oct 30 17:09:10.087232 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.087213 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa9db899-8ed9-4944-8c2c-9ffea59595d1" containerName="revoke-keys" Oct 30 17:09:10.087349 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.087250 6202 cpu_manager.go:414] "containerMap: removing container" podUID="040124e8-ee08-4658-8088-a7563fd57c1b" containerName="payload-pre-processing" Oct 30 17:09:10.087349 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.087258 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040124e8-ee08-4658-8088-a7563fd57c1b" containerName="payload-pre-processing" Oct 30 17:09:10.087437 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.087425 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="040124e8-ee08-4658-8088-a7563fd57c1b" containerName="payload-pre-processing" Oct 30 17:09:10.087470 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.087448 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="64359b58-0df8-41d1-9461-c87f3f6e6fa6" containerName="payload-processing" Oct 30 17:09:10.087470 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.087462 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa9db899-8ed9-4944-8c2c-9ffea59595d1" containerName="revoke-keys" Oct 30 17:09:10.093597 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.093576 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g" Oct 30 17:09:10.106115 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.106091 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs"] Oct 30 17:09:10.111795 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.110651 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:09:10.111795 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.110924 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:09:10.111795 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.110985 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:09:10.111795 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.111009 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:09:10.111795 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.111027 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs" Oct 30 17:09:10.113756 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.112620 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g"] Oct 30 17:09:10.137122 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.137086 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94dpr\" (UniqueName: \"kubernetes.io/projected/a339742e-1b29-4044-9908-0304b50ba5e2-kube-api-access-94dpr\") pod \"maas-api-revoke-sub-cbed3744-d5g5g\" (UID: \"a339742e-1b29-4044-9908-0304b50ba5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g" Oct 30 17:09:10.137316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.137157 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a339742e-1b29-4044-9908-0304b50ba5e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbed3744-d5g5g\" (UID: \"a339742e-1b29-4044-9908-0304b50ba5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g" Oct 30 17:09:10.137316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.137194 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b6640d8-kntqs\" (UID: \"c2175025-c963-4a01-9c1e-d8fdff9cc0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs" Oct 30 17:09:10.137316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.137248 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qh79\" (UniqueName: \"kubernetes.io/projected/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-kube-api-access-4qh79\") pod \"maas-api-revoke-sub-3b6640d8-kntqs\" (UID: \"c2175025-c963-4a01-9c1e-d8fdff9cc0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs" Oct 30 17:09:10.137562 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.137380 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b6640d8-kntqs\" (UID: \"c2175025-c963-4a01-9c1e-d8fdff9cc0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs" Oct 30 17:09:10.137562 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.137524 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a339742e-1b29-4044-9908-0304b50ba5e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbed3744-d5g5g\" (UID: \"a339742e-1b29-4044-9908-0304b50ba5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g" Oct 30 17:09:10.168954 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.140931 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs"] Oct 30 17:09:10.238431 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.238389 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a339742e-1b29-4044-9908-0304b50ba5e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbed3744-d5g5g\" (UID: \"a339742e-1b29-4044-9908-0304b50ba5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g" Oct 30 17:09:10.238639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.238529 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94dpr\" (UniqueName: \"kubernetes.io/projected/a339742e-1b29-4044-9908-0304b50ba5e2-kube-api-access-94dpr\") pod \"maas-api-revoke-sub-cbed3744-d5g5g\" (UID: \"a339742e-1b29-4044-9908-0304b50ba5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g" Oct 30 17:09:10.238639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.238577 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a339742e-1b29-4044-9908-0304b50ba5e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbed3744-d5g5g\" (UID: \"a339742e-1b29-4044-9908-0304b50ba5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g" Oct 30 17:09:10.238902 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.238872 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b6640d8-kntqs\" (UID: \"c2175025-c963-4a01-9c1e-d8fdff9cc0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs" Oct 30 17:09:10.239047 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.238956 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qh79\" (UniqueName: \"kubernetes.io/projected/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-kube-api-access-4qh79\") pod \"maas-api-revoke-sub-3b6640d8-kntqs\" (UID: \"c2175025-c963-4a01-9c1e-d8fdff9cc0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs" Oct 30 17:09:10.239133 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.239113 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b6640d8-kntqs\" (UID: \"c2175025-c963-4a01-9c1e-d8fdff9cc0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs" Oct 30 17:09:10.239375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.239342 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a339742e-1b29-4044-9908-0304b50ba5e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbed3744-d5g5g\" (UID: \"a339742e-1b29-4044-9908-0304b50ba5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g" Oct 30 17:09:10.239958 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.239927 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b6640d8-kntqs\" (UID: \"c2175025-c963-4a01-9c1e-d8fdff9cc0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs" Oct 30 17:09:10.241438 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.241411 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a339742e-1b29-4044-9908-0304b50ba5e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbed3744-d5g5g\" (UID: \"a339742e-1b29-4044-9908-0304b50ba5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g" Oct 30 17:09:10.241610 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.241589 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b6640d8-kntqs\" (UID: \"c2175025-c963-4a01-9c1e-d8fdff9cc0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs" Oct 30 17:09:10.262022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.261992 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qh79\" (UniqueName: \"kubernetes.io/projected/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-kube-api-access-4qh79\") pod \"maas-api-revoke-sub-3b6640d8-kntqs\" (UID: \"c2175025-c963-4a01-9c1e-d8fdff9cc0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs" Oct 30 17:09:10.262180 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.262047 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94dpr\" (UniqueName: \"kubernetes.io/projected/a339742e-1b29-4044-9908-0304b50ba5e2-kube-api-access-94dpr\") pod \"maas-api-revoke-sub-cbed3744-d5g5g\" (UID: \"a339742e-1b29-4044-9908-0304b50ba5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g" Oct 30 17:09:10.409565 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.409532 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g" Oct 30 17:09:10.413245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.413216 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="040124e8-ee08-4658-8088-a7563fd57c1b" path="/var/lib/kubelet/pods/040124e8-ee08-4658-8088-a7563fd57c1b/volumes" Oct 30 17:09:10.414139 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.413669 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64359b58-0df8-41d1-9461-c87f3f6e6fa6" path="/var/lib/kubelet/pods/64359b58-0df8-41d1-9461-c87f3f6e6fa6/volumes" Oct 30 17:09:10.416292 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.416264 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:10.442238 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.441940 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs" Oct 30 17:09:10.796663 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:09:10.796633 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda339742e_1b29_4044_9908_0304b50ba5e2.slice/crio-cf87869dbf2b30807c396e6ac55743b1ae19ffe4b9a35e00b03d33e70bea6efe WatchSource:0}: Error finding container cf87869dbf2b30807c396e6ac55743b1ae19ffe4b9a35e00b03d33e70bea6efe: Status 404 returned error can't find the container with id cf87869dbf2b30807c396e6ac55743b1ae19ffe4b9a35e00b03d33e70bea6efe Oct 30 17:09:10.798308 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.798283 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g"] Oct 30 17:09:10.843484 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.843452 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs"] Oct 30 17:09:10.902837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.902794 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g" event={"ID":"a339742e-1b29-4044-9908-0304b50ba5e2","Type":"ContainerStarted","Data":"cf87869dbf2b30807c396e6ac55743b1ae19ffe4b9a35e00b03d33e70bea6efe"} Oct 30 17:09:10.904203 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:10.904178 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs" event={"ID":"c2175025-c963-4a01-9c1e-d8fdff9cc0ae","Type":"ContainerStarted","Data":"21e49ddf3afabf61ca001fcf4540764954750ec7bedcf1bb65db0b2fa3273ba6"} Oct 30 17:09:11.914937 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:11.914895 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs" event={"ID":"c2175025-c963-4a01-9c1e-d8fdff9cc0ae","Type":"ContainerStarted","Data":"0747f8c08fcdd36e17d4eff875c601746683d848cfe8189e0c23d0abac6920b5"} Oct 30 17:09:11.916699 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:11.916668 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g" event={"ID":"a339742e-1b29-4044-9908-0304b50ba5e2","Type":"ContainerStarted","Data":"e99400e25b36a5d64a0739a06dd1be37be205d7fb28dc16bf812859af8b46231"} Oct 30 17:09:11.956214 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:11.956158 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs" podStartSLOduration=1.956138446 podStartE2EDuration="1.956138446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:11.954758203 +0000 UTC m=+2865.665212971" watchObservedRunningTime="2026-10-30 17:09:11.956138446 +0000 UTC m=+2865.666593192" Oct 30 17:09:11.998030 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:11.997973 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g" podStartSLOduration=1.997960871 podStartE2EDuration="1.997960871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:11.99673068 +0000 UTC m=+2865.707185424" watchObservedRunningTime="2026-10-30 17:09:11.997960871 +0000 UTC m=+2865.708415657" Oct 30 17:09:12.299833 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:12.299720 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:12.929217 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:12.929185 6202 generic.go:356] "Generic (PLEG): container finished" podID="c2175025-c963-4a01-9c1e-d8fdff9cc0ae" containerID="0747f8c08fcdd36e17d4eff875c601746683d848cfe8189e0c23d0abac6920b5" exitCode=0 Oct 30 17:09:12.929668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:12.929288 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs" event={"ID":"c2175025-c963-4a01-9c1e-d8fdff9cc0ae","Type":"ContainerDied","Data":"0747f8c08fcdd36e17d4eff875c601746683d848cfe8189e0c23d0abac6920b5"} Oct 30 17:09:12.931406 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:12.931376 6202 generic.go:356] "Generic (PLEG): container finished" podID="a339742e-1b29-4044-9908-0304b50ba5e2" containerID="e99400e25b36a5d64a0739a06dd1be37be205d7fb28dc16bf812859af8b46231" exitCode=0 Oct 30 17:09:12.931539 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:12.931439 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g" event={"ID":"a339742e-1b29-4044-9908-0304b50ba5e2","Type":"ContainerDied","Data":"e99400e25b36a5d64a0739a06dd1be37be205d7fb28dc16bf812859af8b46231"} Oct 30 17:09:13.336599 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:13.336564 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp"] Oct 30 17:09:13.342288 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:13.342255 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp" Oct 30 17:09:13.363045 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:13.363012 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp"] Oct 30 17:09:13.410651 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:13.410614 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13c2d9bc-cpdjp\" (UID: \"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp" Oct 30 17:09:13.410861 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:13.410785 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13c2d9bc-cpdjp\" (UID: \"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp" Oct 30 17:09:13.410978 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:13.410951 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b7cz\" (UniqueName: \"kubernetes.io/projected/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-kube-api-access-6b7cz\") pod \"maas-api-revoke-sub-13c2d9bc-cpdjp\" (UID: \"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp" Oct 30 17:09:13.512274 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:13.512237 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b7cz\" (UniqueName: \"kubernetes.io/projected/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-kube-api-access-6b7cz\") pod \"maas-api-revoke-sub-13c2d9bc-cpdjp\" (UID: \"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp" Oct 30 17:09:13.512490 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:13.512395 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13c2d9bc-cpdjp\" (UID: \"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp" Oct 30 17:09:13.512490 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:13.512482 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13c2d9bc-cpdjp\" (UID: \"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp" Oct 30 17:09:13.513159 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:13.513118 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13c2d9bc-cpdjp\" (UID: \"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp" Oct 30 17:09:13.515034 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:13.515005 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13c2d9bc-cpdjp\" (UID: \"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp" Oct 30 17:09:13.535597 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:13.535574 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b7cz\" (UniqueName: \"kubernetes.io/projected/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-kube-api-access-6b7cz\") pod \"maas-api-revoke-sub-13c2d9bc-cpdjp\" (UID: \"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp" Oct 30 17:09:13.665627 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:13.665543 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp" Oct 30 17:09:13.888497 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:13.888466 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:13.982903 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:13.982760 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp"] Oct 30 17:09:14.007794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.007725 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:14.296785 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.290785 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:14.308324 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:09:14.308286 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:09:14.312272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.312242 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g" Oct 30 17:09:14.502682 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.502540 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:14.583369 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.579357 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a339742e-1b29-4044-9908-0304b50ba5e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a339742e-1b29-4044-9908-0304b50ba5e2-maas-api-openshift-ca\") pod \"a339742e-1b29-4044-9908-0304b50ba5e2\" (UID: \"a339742e-1b29-4044-9908-0304b50ba5e2\") " Oct 30 17:09:14.583369 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.579439 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a339742e-1b29-4044-9908-0304b50ba5e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a339742e-1b29-4044-9908-0304b50ba5e2-maas-api-xks-ca\") pod \"a339742e-1b29-4044-9908-0304b50ba5e2\" (UID: \"a339742e-1b29-4044-9908-0304b50ba5e2\") " Oct 30 17:09:14.583369 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.579620 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a339742e-1b29-4044-9908-0304b50ba5e2-kube-api-access-94dpr\" (UniqueName: \"kubernetes.io/projected/a339742e-1b29-4044-9908-0304b50ba5e2-kube-api-access-94dpr\") pod \"a339742e-1b29-4044-9908-0304b50ba5e2\" (UID: \"a339742e-1b29-4044-9908-0304b50ba5e2\") " Oct 30 17:09:14.593286 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.593135 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a339742e-1b29-4044-9908-0304b50ba5e2-maas-api-openshift-ca" pod "a339742e-1b29-4044-9908-0304b50ba5e2" (UID: "a339742e-1b29-4044-9908-0304b50ba5e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:14.623878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.623544 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a339742e-1b29-4044-9908-0304b50ba5e2-maas-api-xks-ca" pod "a339742e-1b29-4044-9908-0304b50ba5e2" (UID: "a339742e-1b29-4044-9908-0304b50ba5e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:09:14.637925 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.634943 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a339742e-1b29-4044-9908-0304b50ba5e2-kube-api-access-94dpr" pod "a339742e-1b29-4044-9908-0304b50ba5e2" (UID: "a339742e-1b29-4044-9908-0304b50ba5e2"). InnerVolumeSpecName "kube-api-access-94dpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:14.637925 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.636149 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs" Oct 30 17:09:14.647008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.646977 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:14.683187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.683151 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94dpr\" (UniqueName: \"kubernetes.io/projected/a339742e-1b29-4044-9908-0304b50ba5e2-kube-api-access-94dpr\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:14.683187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.683188 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a339742e-1b29-4044-9908-0304b50ba5e2-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:14.683448 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.683207 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a339742e-1b29-4044-9908-0304b50ba5e2-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:14.790849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.789314 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-maas-api-xks-ca\") pod \"c2175025-c963-4a01-9c1e-d8fdff9cc0ae\" (UID: \"c2175025-c963-4a01-9c1e-d8fdff9cc0ae\") " Oct 30 17:09:14.790849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.789394 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-kube-api-access-4qh79\" (UniqueName: \"kubernetes.io/projected/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-kube-api-access-4qh79\") pod \"c2175025-c963-4a01-9c1e-d8fdff9cc0ae\" (UID: \"c2175025-c963-4a01-9c1e-d8fdff9cc0ae\") " Oct 30 17:09:14.790849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.789472 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-maas-api-openshift-ca\") pod \"c2175025-c963-4a01-9c1e-d8fdff9cc0ae\" (UID: \"c2175025-c963-4a01-9c1e-d8fdff9cc0ae\") " Oct 30 17:09:14.802797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.802135 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-maas-api-openshift-ca" pod "c2175025-c963-4a01-9c1e-d8fdff9cc0ae" (UID: "c2175025-c963-4a01-9c1e-d8fdff9cc0ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:14.818793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.806881 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-maas-api-xks-ca" pod "c2175025-c963-4a01-9c1e-d8fdff9cc0ae" (UID: "c2175025-c963-4a01-9c1e-d8fdff9cc0ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:09:14.835031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.826508 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-kube-api-access-4qh79" pod "c2175025-c963-4a01-9c1e-d8fdff9cc0ae" (UID: "c2175025-c963-4a01-9c1e-d8fdff9cc0ae"). InnerVolumeSpecName "kube-api-access-4qh79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:14.896128 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.895244 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:14.896128 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.895278 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qh79\" (UniqueName: \"kubernetes.io/projected/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-kube-api-access-4qh79\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:14.896128 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.895294 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2175025-c963-4a01-9c1e-d8fdff9cc0ae-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:14.971812 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.971751 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp" event={"ID":"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7","Type":"ContainerStarted","Data":"280dc6bee98e4a2f25b348bd9b2597d560f29cce75f2330b6d89a08d0b15370b"} Oct 30 17:09:14.971996 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.971835 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp" event={"ID":"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7","Type":"ContainerStarted","Data":"631fa5778a7ba801011728f43b4ede5a41c60ce48b32f04306ae6162c5cba9c2"} Oct 30 17:09:14.973467 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.973438 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g" Oct 30 17:09:14.973467 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.973458 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g" event={"ID":"a339742e-1b29-4044-9908-0304b50ba5e2","Type":"ContainerDied","Data":"cf87869dbf2b30807c396e6ac55743b1ae19ffe4b9a35e00b03d33e70bea6efe"} Oct 30 17:09:14.973648 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.973503 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf87869dbf2b30807c396e6ac55743b1ae19ffe4b9a35e00b03d33e70bea6efe" Oct 30 17:09:14.975269 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.975241 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs" event={"ID":"c2175025-c963-4a01-9c1e-d8fdff9cc0ae","Type":"ContainerDied","Data":"21e49ddf3afabf61ca001fcf4540764954750ec7bedcf1bb65db0b2fa3273ba6"} Oct 30 17:09:14.975455 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.975439 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21e49ddf3afabf61ca001fcf4540764954750ec7bedcf1bb65db0b2fa3273ba6" Oct 30 17:09:14.975455 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:14.975323 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs" Oct 30 17:09:15.016117 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:15.016067 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp" podStartSLOduration=2.016053488 podStartE2EDuration="2.016053488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:15.011568405 +0000 UTC m=+2868.722023149" watchObservedRunningTime="2026-10-30 17:09:15.016053488 +0000 UTC m=+2868.726508232" Oct 30 17:09:15.985575 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:15.985540 6202 generic.go:356] "Generic (PLEG): container finished" podID="326044d6-f5fb-4f8d-b348-f2c4f41b5cc7" containerID="280dc6bee98e4a2f25b348bd9b2597d560f29cce75f2330b6d89a08d0b15370b" exitCode=0 Oct 30 17:09:15.985830 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:15.985627 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp" event={"ID":"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7","Type":"ContainerDied","Data":"280dc6bee98e4a2f25b348bd9b2597d560f29cce75f2330b6d89a08d0b15370b"} Oct 30 17:09:15.990104 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:15.990080 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:16.148812 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:16.113389 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:17.181134 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:17.181104 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp" Oct 30 17:09:17.263298 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:17.263196 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-maas-api-xks-ca\") pod \"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7\" (UID: \"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7\") " Oct 30 17:09:17.263298 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:17.263297 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-maas-api-openshift-ca\") pod \"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7\" (UID: \"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7\") " Oct 30 17:09:17.263555 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:17.263532 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-kube-api-access-6b7cz\" (UniqueName: \"kubernetes.io/projected/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-kube-api-access-6b7cz\") pod \"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7\" (UID: \"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7\") " Oct 30 17:09:17.264267 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:17.264230 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-maas-api-openshift-ca" pod "326044d6-f5fb-4f8d-b348-f2c4f41b5cc7" (UID: "326044d6-f5fb-4f8d-b348-f2c4f41b5cc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:17.272915 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:17.272891 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-maas-api-xks-ca" pod "326044d6-f5fb-4f8d-b348-f2c4f41b5cc7" (UID: "326044d6-f5fb-4f8d-b348-f2c4f41b5cc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:09:17.273054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:17.272949 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-kube-api-access-6b7cz" pod "326044d6-f5fb-4f8d-b348-f2c4f41b5cc7" (UID: "326044d6-f5fb-4f8d-b348-f2c4f41b5cc7"). InnerVolumeSpecName "kube-api-access-6b7cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:17.366148 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:17.366110 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:17.366148 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:17.366142 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b7cz\" (UniqueName: \"kubernetes.io/projected/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-kube-api-access-6b7cz\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:17.366148 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:17.366152 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:17.728387 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:17.728357 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:18.007372 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:18.007216 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp" event={"ID":"326044d6-f5fb-4f8d-b348-f2c4f41b5cc7","Type":"ContainerDied","Data":"631fa5778a7ba801011728f43b4ede5a41c60ce48b32f04306ae6162c5cba9c2"} Oct 30 17:09:18.007372 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:18.007266 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp" Oct 30 17:09:18.007372 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:18.007270 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="631fa5778a7ba801011728f43b4ede5a41c60ce48b32f04306ae6162c5cba9c2" Oct 30 17:09:19.321663 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:19.321632 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:19.433614 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:19.433578 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:19.556381 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:19.556056 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:21.404295 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:21.404263 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:22.045780 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:22.045738 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:22.177046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:22.177015 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:22.940757 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:22.940720 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:23.182724 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.182676 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:23.396241 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.396204 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6"] Oct 30 17:09:23.397203 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.397175 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c2175025-c963-4a01-9c1e-d8fdff9cc0ae" containerName="revoke-keys" Oct 30 17:09:23.397203 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.397202 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2175025-c963-4a01-9c1e-d8fdff9cc0ae" containerName="revoke-keys" Oct 30 17:09:23.397417 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.397266 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a339742e-1b29-4044-9908-0304b50ba5e2" containerName="revoke-keys" Oct 30 17:09:23.397417 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.397274 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a339742e-1b29-4044-9908-0304b50ba5e2" containerName="revoke-keys" Oct 30 17:09:23.397417 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.397306 6202 cpu_manager.go:414] "containerMap: removing container" podUID="326044d6-f5fb-4f8d-b348-f2c4f41b5cc7" containerName="revoke-keys" Oct 30 17:09:23.397417 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.397314 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="326044d6-f5fb-4f8d-b348-f2c4f41b5cc7" containerName="revoke-keys" Oct 30 17:09:23.397693 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.397487 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2175025-c963-4a01-9c1e-d8fdff9cc0ae" containerName="revoke-keys" Oct 30 17:09:23.397693 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.397515 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a339742e-1b29-4044-9908-0304b50ba5e2" containerName="revoke-keys" Oct 30 17:09:23.397693 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.397529 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="326044d6-f5fb-4f8d-b348-f2c4f41b5cc7" containerName="revoke-keys" Oct 30 17:09:23.409929 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.409901 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.422301 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.422272 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6"] Oct 30 17:09:23.431317 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.431290 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-698c71-openshift-default-dockercfg-sqv5p\"" Oct 30 17:09:23.436809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.436784 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:23.475628 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.475565 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-token\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.476004 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.475977 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-envoy\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.476188 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.476170 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-data\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.476387 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.476368 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-podinfo\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.476517 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.476501 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istiod-ca-cert\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.476725 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.476663 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-workload-certs\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.476915 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.476894 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjp4x\" (UniqueName: \"kubernetes.io/projected/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-kube-api-access-sjp4x\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.476985 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.476936 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-workload-socket\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.477053 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.477016 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-credential-socket\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.594792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.580746 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-podinfo\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.594792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.580819 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istiod-ca-cert\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.594792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.580899 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-workload-certs\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.594792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.581027 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjp4x\" (UniqueName: \"kubernetes.io/projected/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-kube-api-access-sjp4x\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.594792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.581058 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-workload-socket\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.594792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.581128 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-credential-socket\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.594792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.581239 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-token\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.594792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.581307 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-envoy\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.594792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.581351 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-data\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.594792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.581977 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-data\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.594792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.583102 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-workload-socket\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.594792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.583727 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istiod-ca-cert\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.594792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.594089 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-workload-certs\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.632794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.618942 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-credential-socket\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.676120 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.676029 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-envoy\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.708805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.707494 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-token\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.742978 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.725879 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-podinfo\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.742978 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.734556 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjp4x\" (UniqueName: \"kubernetes.io/projected/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-kube-api-access-sjp4x\") pod \"e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.742978 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.739118 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:23.772799 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:23.762262 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:24.146254 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:09:24.146218 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcdb58e9_32c9_4901_8878_b2e85c2a1ee9.slice/crio-643283fd2f68d8cda0892275545e23275fd21b8fde1a33f783be4dcf5fd07137 WatchSource:0}: Error finding container 643283fd2f68d8cda0892275545e23275fd21b8fde1a33f783be4dcf5fd07137: Status 404 returned error can't find the container with id 643283fd2f68d8cda0892275545e23275fd21b8fde1a33f783be4dcf5fd07137 Oct 30 17:09:24.148161 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:24.148119 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6"] Oct 30 17:09:24.149715 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:24.149676 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:09:24.149892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:24.149781 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:09:24.149892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:24.149831 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:09:24.900326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:24.900290 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:25.021362 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:25.021327 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:25.104417 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:25.104181 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" event={"ID":"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9","Type":"ContainerStarted","Data":"1ae008250328678a3eafc9e9c153fde7b99dba55cb6f0b19cbba29a784e1521b"} Oct 30 17:09:25.104417 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:25.104249 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" event={"ID":"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9","Type":"ContainerStarted","Data":"643283fd2f68d8cda0892275545e23275fd21b8fde1a33f783be4dcf5fd07137"} Oct 30 17:09:25.168852 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:25.166086 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" podStartSLOduration=2.166065272 podStartE2EDuration="2.166065272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:25.16016068 +0000 UTC m=+2878.870615429" watchObservedRunningTime="2026-10-30 17:09:25.166065272 +0000 UTC m=+2878.876520017" Oct 30 17:09:25.749706 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:25.749662 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:25.756553 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:25.756529 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:26.033212 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:26.033119 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6"] Oct 30 17:09:26.037518 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:26.037487 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580be196-rpdk6"] Oct 30 17:09:26.114494 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:26.114462 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:26.115452 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:26.115432 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:09:26.416730 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:26.416691 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24dd6920-1fd0-4644-892c-5c3d04d09c14" path="/var/lib/kubelet/pods/24dd6920-1fd0-4644-892c-5c3d04d09c14/volumes" Oct 30 17:09:27.044100 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:27.044062 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:28.141232 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:28.141184 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:28.268952 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:28.268919 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:28.984576 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:28.984540 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:29.813620 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:29.813576 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:29.943639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:29.943573 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:29.946183 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:29.946162 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk"] Oct 30 17:09:29.952501 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:29.952479 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk" Oct 30 17:09:29.971703 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:29.970134 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-698c71-dockercfg-8ld88\"" Oct 30 17:09:29.980363 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:29.978114 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-698c71\"" Oct 30 17:09:29.985340 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:29.985301 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk"] Oct 30 17:09:30.000029 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:29.999988 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl"] Oct 30 17:09:30.005430 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:30.005404 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl" Oct 30 17:09:30.024256 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:30.024221 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl"] Oct 30 17:09:30.068129 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:30.068032 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrc6b\" (UniqueName: \"kubernetes.io/projected/0d32e930-fdfe-4971-a4bb-b6eea8491790-kube-api-access-hrc6b\") pod \"payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk\" (UID: \"0d32e930-fdfe-4971-a4bb-b6eea8491790\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk" Oct 30 17:09:30.068447 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:30.068413 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d32e930-fdfe-4971-a4bb-b6eea8491790-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk\" (UID: \"0d32e930-fdfe-4971-a4bb-b6eea8491790\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk" Oct 30 17:09:30.260794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:30.236202 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrc6b\" (UniqueName: \"kubernetes.io/projected/0d32e930-fdfe-4971-a4bb-b6eea8491790-kube-api-access-hrc6b\") pod \"payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk\" (UID: \"0d32e930-fdfe-4971-a4bb-b6eea8491790\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk" Oct 30 17:09:30.346030 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:30.345996 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d32e930-fdfe-4971-a4bb-b6eea8491790-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk\" (UID: \"0d32e930-fdfe-4971-a4bb-b6eea8491790\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk" Oct 30 17:09:30.346250 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:30.346145 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4qqv\" (UniqueName: \"kubernetes.io/projected/4c775d0b-bde0-4f40-b429-536c484aeba9-kube-api-access-t4qqv\") pod \"payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl\" (UID: \"4c775d0b-bde0-4f40-b429-536c484aeba9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl" Oct 30 17:09:30.346431 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:30.346412 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c775d0b-bde0-4f40-b429-536c484aeba9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl\" (UID: \"4c775d0b-bde0-4f40-b429-536c484aeba9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl" Oct 30 17:09:30.348730 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:30.347388 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d32e930-fdfe-4971-a4bb-b6eea8491790-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk\" (UID: \"0d32e930-fdfe-4971-a4bb-b6eea8491790\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk" Oct 30 17:09:30.360548 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:30.357260 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrc6b\" (UniqueName: \"kubernetes.io/projected/0d32e930-fdfe-4971-a4bb-b6eea8491790-kube-api-access-hrc6b\") pod \"payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk\" (UID: \"0d32e930-fdfe-4971-a4bb-b6eea8491790\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk" Oct 30 17:09:30.447469 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:30.447433 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4qqv\" (UniqueName: \"kubernetes.io/projected/4c775d0b-bde0-4f40-b429-536c484aeba9-kube-api-access-t4qqv\") pod \"payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl\" (UID: \"4c775d0b-bde0-4f40-b429-536c484aeba9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl" Oct 30 17:09:30.447664 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:30.447629 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c775d0b-bde0-4f40-b429-536c484aeba9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl\" (UID: \"4c775d0b-bde0-4f40-b429-536c484aeba9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl" Oct 30 17:09:30.448555 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:30.448529 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c775d0b-bde0-4f40-b429-536c484aeba9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl\" (UID: \"4c775d0b-bde0-4f40-b429-536c484aeba9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl" Oct 30 17:09:30.472726 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:30.472688 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4qqv\" (UniqueName: \"kubernetes.io/projected/4c775d0b-bde0-4f40-b429-536c484aeba9-kube-api-access-t4qqv\") pod \"payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl\" (UID: \"4c775d0b-bde0-4f40-b429-536c484aeba9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl" Oct 30 17:09:30.569432 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:30.569394 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk" Oct 30 17:09:30.632357 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:30.632321 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl" Oct 30 17:09:30.880271 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:09:30.880177 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d32e930_fdfe_4971_a4bb_b6eea8491790.slice/crio-66dc2a2783f35187d5e0a5dc6eb3a4fc550dde94971476c6c1ae03784307e518 WatchSource:0}: Error finding container 66dc2a2783f35187d5e0a5dc6eb3a4fc550dde94971476c6c1ae03784307e518: Status 404 returned error can't find the container with id 66dc2a2783f35187d5e0a5dc6eb3a4fc550dde94971476c6c1ae03784307e518 Oct 30 17:09:30.882310 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:30.882281 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk"] Oct 30 17:09:30.978920 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:30.978890 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl"] Oct 30 17:09:31.080955 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:31.080699 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:31.197968 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:31.197884 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:31.341111 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:31.341069 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:31.342358 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:31.342329 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk" event={"ID":"0d32e930-fdfe-4971-a4bb-b6eea8491790","Type":"ContainerStarted","Data":"946e6fb9ed3c6526f62b6fe516cf70ab17319a32dd95f2f626d23118e56a4731"} Oct 30 17:09:31.342529 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:31.342395 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk" Oct 30 17:09:31.342529 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:31.342419 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk" event={"ID":"0d32e930-fdfe-4971-a4bb-b6eea8491790","Type":"ContainerStarted","Data":"66dc2a2783f35187d5e0a5dc6eb3a4fc550dde94971476c6c1ae03784307e518"} Oct 30 17:09:31.344455 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:31.344424 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl" event={"ID":"4c775d0b-bde0-4f40-b429-536c484aeba9","Type":"ContainerStarted","Data":"5b69c26f9a25fca64f6246febeea52459a689e2462da645598e6b2541e2d344a"} Oct 30 17:09:31.344586 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:31.344483 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl" event={"ID":"4c775d0b-bde0-4f40-b429-536c484aeba9","Type":"ContainerStarted","Data":"4505950cfd20e3553c011a7d81aa66bcdc29a4bb731587b3a96e6636a45a2bac"} Oct 30 17:09:31.344586 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:31.344515 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl" Oct 30 17:09:31.389645 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:31.389592 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk" podStartSLOduration=2.389577017 podStartE2EDuration="2.389577017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:31.385803697 +0000 UTC m=+2885.096258439" watchObservedRunningTime="2026-10-30 17:09:31.389577017 +0000 UTC m=+2885.100031761" Oct 30 17:09:31.435799 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:31.435726 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl" podStartSLOduration=2.435707325 podStartE2EDuration="2.435707325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:31.431512783 +0000 UTC m=+2885.141967526" watchObservedRunningTime="2026-10-30 17:09:31.435707325 +0000 UTC m=+2885.146162071" Oct 30 17:09:32.959267 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:32.959232 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:33.202366 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.202322 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:33.563789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.563442 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4"] Oct 30 17:09:33.593705 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.593662 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4"] Oct 30 17:09:33.594184 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.594090 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4" Oct 30 17:09:33.613223 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.611399 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e1e830-dockercfg-sswz6\"" Oct 30 17:09:33.619914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.619827 6202 scope.go:122] "RemoveContainer" containerID="6049d01c230b5f2608e81db63a56dffe38bb0b26517faab3b19bdf32476325e0" Oct 30 17:09:33.627420 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.626543 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e1e830\"" Oct 30 17:09:33.661740 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.661701 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggmrl\" (UniqueName: \"kubernetes.io/projected/d831ff6b-0124-40b7-aedb-70315823c044-kube-api-access-ggmrl\") pod \"payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4\" (UID: \"d831ff6b-0124-40b7-aedb-70315823c044\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4" Oct 30 17:09:33.662701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.662669 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d831ff6b-0124-40b7-aedb-70315823c044-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4\" (UID: \"d831ff6b-0124-40b7-aedb-70315823c044\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4" Oct 30 17:09:33.710175 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.708540 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl"] Oct 30 17:09:33.719356 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.719235 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl" Oct 30 17:09:33.731315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.731289 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl"] Oct 30 17:09:33.764983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.764945 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggmrl\" (UniqueName: \"kubernetes.io/projected/d831ff6b-0124-40b7-aedb-70315823c044-kube-api-access-ggmrl\") pod \"payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4\" (UID: \"d831ff6b-0124-40b7-aedb-70315823c044\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4" Oct 30 17:09:33.765200 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.765161 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bwr4\" (UniqueName: \"kubernetes.io/projected/f7bcdc3b-666e-4648-ace6-178f2c4718e5-kube-api-access-7bwr4\") pod \"payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl\" (UID: \"f7bcdc3b-666e-4648-ace6-178f2c4718e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl" Oct 30 17:09:33.765579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.765553 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d831ff6b-0124-40b7-aedb-70315823c044-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4\" (UID: \"d831ff6b-0124-40b7-aedb-70315823c044\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4" Oct 30 17:09:33.765700 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.765610 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bcdc3b-666e-4648-ace6-178f2c4718e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl\" (UID: \"f7bcdc3b-666e-4648-ace6-178f2c4718e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl" Oct 30 17:09:33.766320 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.766297 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d831ff6b-0124-40b7-aedb-70315823c044-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4\" (UID: \"d831ff6b-0124-40b7-aedb-70315823c044\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4" Oct 30 17:09:33.790700 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.790670 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggmrl\" (UniqueName: \"kubernetes.io/projected/d831ff6b-0124-40b7-aedb-70315823c044-kube-api-access-ggmrl\") pod \"payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4\" (UID: \"d831ff6b-0124-40b7-aedb-70315823c044\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4" Oct 30 17:09:33.867485 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.867448 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bwr4\" (UniqueName: \"kubernetes.io/projected/f7bcdc3b-666e-4648-ace6-178f2c4718e5-kube-api-access-7bwr4\") pod \"payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl\" (UID: \"f7bcdc3b-666e-4648-ace6-178f2c4718e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl" Oct 30 17:09:33.867718 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.867638 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bcdc3b-666e-4648-ace6-178f2c4718e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl\" (UID: \"f7bcdc3b-666e-4648-ace6-178f2c4718e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl" Oct 30 17:09:33.868574 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.868552 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bcdc3b-666e-4648-ace6-178f2c4718e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl\" (UID: \"f7bcdc3b-666e-4648-ace6-178f2c4718e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl" Oct 30 17:09:33.893131 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.893099 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bwr4\" (UniqueName: \"kubernetes.io/projected/f7bcdc3b-666e-4648-ace6-178f2c4718e5-kube-api-access-7bwr4\") pod \"payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl\" (UID: \"f7bcdc3b-666e-4648-ace6-178f2c4718e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl" Oct 30 17:09:33.941361 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:33.941326 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4" Oct 30 17:09:34.042333 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:34.042081 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl" Oct 30 17:09:34.255500 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:09:34.255469 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd831ff6b_0124_40b7_aedb_70315823c044.slice/crio-f9cdc451d3f70d10ed57f8c1ce426c7aed751e8d792bb69d5007c91b708401f5 WatchSource:0}: Error finding container f9cdc451d3f70d10ed57f8c1ce426c7aed751e8d792bb69d5007c91b708401f5: Status 404 returned error can't find the container with id f9cdc451d3f70d10ed57f8c1ce426c7aed751e8d792bb69d5007c91b708401f5 Oct 30 17:09:34.258958 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:34.258929 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4"] Oct 30 17:09:34.375442 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:34.375394 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4" event={"ID":"d831ff6b-0124-40b7-aedb-70315823c044","Type":"ContainerStarted","Data":"16332b758917699ce19cc949aca7689cadb5a4f29aa6bd1a18f5f51ea1f0622b"} Oct 30 17:09:34.375628 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:34.375459 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4" event={"ID":"d831ff6b-0124-40b7-aedb-70315823c044","Type":"ContainerStarted","Data":"f9cdc451d3f70d10ed57f8c1ce426c7aed751e8d792bb69d5007c91b708401f5"} Oct 30 17:09:34.375628 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:34.375504 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4" Oct 30 17:09:34.384668 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:09:34.384641 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bcdc3b_666e_4648_ace6_178f2c4718e5.slice/crio-e8eed958e6dc9c35b1e8c5936f65ad0371546dd7456acf817bf5050f4acac4e5 WatchSource:0}: Error finding container e8eed958e6dc9c35b1e8c5936f65ad0371546dd7456acf817bf5050f4acac4e5: Status 404 returned error can't find the container with id e8eed958e6dc9c35b1e8c5936f65ad0371546dd7456acf817bf5050f4acac4e5 Oct 30 17:09:34.386318 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:34.386292 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl"] Oct 30 17:09:34.422085 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:34.422039 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4" podStartSLOduration=1.422025954 podStartE2EDuration="1.422025954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:34.419400761 +0000 UTC m=+2888.129855505" watchObservedRunningTime="2026-10-30 17:09:34.422025954 +0000 UTC m=+2888.132480697" Oct 30 17:09:34.541586 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:34.541523 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:35.388109 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:35.388004 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl" event={"ID":"f7bcdc3b-666e-4648-ace6-178f2c4718e5","Type":"ContainerStarted","Data":"6ec249052e351e706bd0cff9edc063fe36c2c2f7d4e58e8fc729d850dc7893a0"} Oct 30 17:09:35.388109 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:35.388076 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl" event={"ID":"f7bcdc3b-666e-4648-ace6-178f2c4718e5","Type":"ContainerStarted","Data":"e8eed958e6dc9c35b1e8c5936f65ad0371546dd7456acf817bf5050f4acac4e5"} Oct 30 17:09:35.388583 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:35.388175 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl" Oct 30 17:09:35.432163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:35.432054 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl" podStartSLOduration=2.432040216 podStartE2EDuration="2.432040216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:35.430333719 +0000 UTC m=+2889.140788463" watchObservedRunningTime="2026-10-30 17:09:35.432040216 +0000 UTC m=+2889.142494959" Oct 30 17:09:36.366056 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:36.366014 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:37.039690 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:37.039647 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx"] Oct 30 17:09:37.054941 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:37.054902 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749"] Oct 30 17:09:37.059816 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:37.059749 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0992b63f-dtdmx"] Oct 30 17:09:37.063668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:37.063641 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e33204e-zx749"] Oct 30 17:09:38.227589 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:38.227555 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:38.413296 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:38.413260 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ded757e-4257-4ac4-9b06-15b34822ae38" path="/var/lib/kubelet/pods/2ded757e-4257-4ac4-9b06-15b34822ae38/volumes" Oct 30 17:09:38.413794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:38.413779 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53a30ace-ae20-4a45-a8a2-1ba3e0008486" path="/var/lib/kubelet/pods/53a30ace-ae20-4a45-a8a2-1ba3e0008486/volumes" Oct 30 17:09:39.891799 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:39.891738 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm"] Oct 30 17:09:39.898845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:39.898817 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:39.923322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:39.923293 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm"] Oct 30 17:09:39.930042 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:39.930009 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e4048c-openshift-default-dockercfg-ck5hk\"" Oct 30 17:09:39.969485 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:39.969447 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-workload-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:39.969703 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:39.969504 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-credential-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:39.969703 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:39.969592 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-istio-envoy\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:39.969703 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:39.969612 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ae3695-e79b-410d-86a3-06151a2c8038-istio-podinfo\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:39.969914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:39.969712 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ae3695-e79b-410d-86a3-06151a2c8038-istio-token\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:39.969914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:39.969855 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-workload-certs\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:39.970033 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:39.969926 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ae3695-e79b-410d-86a3-06151a2c8038-istiod-ca-cert\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:39.970033 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:39.969945 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-istio-data\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:39.970033 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:39.969984 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr9g4\" (UniqueName: \"kubernetes.io/projected/05ae3695-e79b-410d-86a3-06151a2c8038-kube-api-access-rr9g4\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:39.994880 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:39.994822 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:40.077149 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.077105 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ae3695-e79b-410d-86a3-06151a2c8038-istio-token\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.079705 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.079669 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-workload-certs\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.079961 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.079858 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ae3695-e79b-410d-86a3-06151a2c8038-istiod-ca-cert\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.079961 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.079894 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-istio-data\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.080113 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.079964 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr9g4\" (UniqueName: \"kubernetes.io/projected/05ae3695-e79b-410d-86a3-06151a2c8038-kube-api-access-rr9g4\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.080113 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.080019 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-workload-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.080113 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.080064 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-credential-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.080276 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.080124 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-istio-envoy\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.080276 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.080149 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ae3695-e79b-410d-86a3-06151a2c8038-istio-podinfo\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.081279 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.081248 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-workload-certs\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.081963 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.081937 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-credential-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.081963 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.081956 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ae3695-e79b-410d-86a3-06151a2c8038-istiod-ca-cert\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.082153 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.082137 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-workload-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.082240 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.082220 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-istio-data\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.084646 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.084623 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ae3695-e79b-410d-86a3-06151a2c8038-istio-podinfo\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.085202 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.085182 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-istio-envoy\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.105300 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.105273 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ae3695-e79b-410d-86a3-06151a2c8038-istio-token\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.105507 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.105486 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr9g4\" (UniqueName: \"kubernetes.io/projected/05ae3695-e79b-410d-86a3-06151a2c8038-kube-api-access-rr9g4\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.216119 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.216023 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:40.550120 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.549258 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm"] Oct 30 17:09:40.552517 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.552473 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:09:40.552636 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.552564 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:09:40.552728 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:40.552636 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:09:41.519587 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:41.519550 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" event={"ID":"05ae3695-e79b-410d-86a3-06151a2c8038","Type":"ContainerStarted","Data":"957c7bcfc148b69b7f84a985154b13481ae133011345956d2adac35d821817bf"} Oct 30 17:09:41.520146 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:41.519616 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" event={"ID":"05ae3695-e79b-410d-86a3-06151a2c8038","Type":"ContainerStarted","Data":"02bc56bf6460affc36bcf36ada31ea83d394f886690cb7ed542f2179ebbbcdb1"} Oct 30 17:09:41.598608 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:41.598550 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" podStartSLOduration=2.598535009 podStartE2EDuration="2.598535009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:41.583564896 +0000 UTC m=+2895.294019668" watchObservedRunningTime="2026-10-30 17:09:41.598535009 +0000 UTC m=+2895.308989941" Oct 30 17:09:41.810641 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:41.810560 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:42.082416 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:42.082287 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:42.217005 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:42.216970 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:42.219039 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:42.219017 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body= Oct 30 17:09:42.219297 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:42.219074 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" podUID="05ae3695-e79b-410d-86a3-06151a2c8038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" Oct 30 17:09:42.320236 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:42.320204 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:42.375012 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:42.374928 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl" Oct 30 17:09:42.375505 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:42.375481 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk" Oct 30 17:09:43.217317 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:43.217283 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body= Oct 30 17:09:43.217817 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:43.217338 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" podUID="05ae3695-e79b-410d-86a3-06151a2c8038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" Oct 30 17:09:43.969850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:43.969804 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:44.088796 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:44.088012 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:44.224869 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:44.217164 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body= Oct 30 17:09:44.224869 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:44.217230 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" podUID="05ae3695-e79b-410d-86a3-06151a2c8038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" Oct 30 17:09:44.370172 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:09:44.368956 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:09:45.012433 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:45.012385 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:45.218975 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:45.218920 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body= Oct 30 17:09:45.219290 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:45.219255 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" podUID="05ae3695-e79b-410d-86a3-06151a2c8038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" Oct 30 17:09:45.395906 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:45.395295 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4" Oct 30 17:09:45.876604 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:45.876130 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:46.217248 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:46.217158 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body= Oct 30 17:09:46.217248 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:46.217229 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" podUID="05ae3695-e79b-410d-86a3-06151a2c8038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" Oct 30 17:09:46.473592 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:46.473506 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl" Oct 30 17:09:46.667530 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:46.667495 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:47.218565 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:47.218527 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body= Oct 30 17:09:47.224259 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:47.222970 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" podUID="05ae3695-e79b-410d-86a3-06151a2c8038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" Oct 30 17:09:47.925687 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:47.925653 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:48.035699 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:48.035663 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht"] Oct 30 17:09:48.045419 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:48.045389 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc69f624-7l8ht"] Oct 30 17:09:48.292829 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:48.292722 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:48.293217 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:48.293190 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:48.294183 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:48.294151 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:09:48.365071 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:48.365001 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:48.419731 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:48.419694 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf16d38-2b07-4378-bc4e-4a91a978506f" path="/var/lib/kubelet/pods/eaf16d38-2b07-4378-bc4e-4a91a978506f/volumes" Oct 30 17:09:48.482935 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:48.482906 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:51.217895 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:51.217861 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:52.075882 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:52.075846 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt"] Oct 30 17:09:52.097562 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:52.097530 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt" Oct 30 17:09:52.107010 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:52.106971 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt"] Oct 30 17:09:52.114646 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:52.114618 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:09:52.127444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:52.127406 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:09:52.127600 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:52.127557 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:09:52.127706 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:52.127610 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:09:52.196741 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:52.196660 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vlc2\" (UniqueName: \"kubernetes.io/projected/fbbd574a-a73c-4765-a4c3-e93064c7983c-kube-api-access-5vlc2\") pod \"maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt\" (UID: \"fbbd574a-a73c-4765-a4c3-e93064c7983c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt" Oct 30 17:09:52.196741 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:52.196722 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbd574a-a73c-4765-a4c3-e93064c7983c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt\" (UID: \"fbbd574a-a73c-4765-a4c3-e93064c7983c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt" Oct 30 17:09:52.196952 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:52.196874 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbd574a-a73c-4765-a4c3-e93064c7983c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt\" (UID: \"fbbd574a-a73c-4765-a4c3-e93064c7983c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt" Oct 30 17:09:52.298230 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:52.298197 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbd574a-a73c-4765-a4c3-e93064c7983c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt\" (UID: \"fbbd574a-a73c-4765-a4c3-e93064c7983c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt" Oct 30 17:09:52.298712 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:52.298281 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbd574a-a73c-4765-a4c3-e93064c7983c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt\" (UID: \"fbbd574a-a73c-4765-a4c3-e93064c7983c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt" Oct 30 17:09:52.298712 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:52.298579 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vlc2\" (UniqueName: \"kubernetes.io/projected/fbbd574a-a73c-4765-a4c3-e93064c7983c-kube-api-access-5vlc2\") pod \"maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt\" (UID: \"fbbd574a-a73c-4765-a4c3-e93064c7983c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt" Oct 30 17:09:52.299071 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:52.299046 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbd574a-a73c-4765-a4c3-e93064c7983c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt\" (UID: \"fbbd574a-a73c-4765-a4c3-e93064c7983c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt" Oct 30 17:09:52.301060 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:52.301029 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbd574a-a73c-4765-a4c3-e93064c7983c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt\" (UID: \"fbbd574a-a73c-4765-a4c3-e93064c7983c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt" Oct 30 17:09:52.323302 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:52.323276 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vlc2\" (UniqueName: \"kubernetes.io/projected/fbbd574a-a73c-4765-a4c3-e93064c7983c-kube-api-access-5vlc2\") pod \"maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt\" (UID: \"fbbd574a-a73c-4765-a4c3-e93064c7983c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt" Oct 30 17:09:52.426460 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:52.426426 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt" Oct 30 17:09:52.735371 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:09:52.735333 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbd574a_a73c_4765_a4c3_e93064c7983c.slice/crio-ba74ac7e88029d5ae79dfdadb3f590fc930e79c8db89736dad89a346f10e95c7 WatchSource:0}: Error finding container ba74ac7e88029d5ae79dfdadb3f590fc930e79c8db89736dad89a346f10e95c7: Status 404 returned error can't find the container with id ba74ac7e88029d5ae79dfdadb3f590fc930e79c8db89736dad89a346f10e95c7 Oct 30 17:09:52.735672 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:52.735649 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt"] Oct 30 17:09:53.144988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:53.144947 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt" event={"ID":"fbbd574a-a73c-4765-a4c3-e93064c7983c","Type":"ContainerStarted","Data":"ba74ac7e88029d5ae79dfdadb3f590fc930e79c8db89736dad89a346f10e95c7"} Oct 30 17:09:53.358957 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:53.358930 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:53.609343 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:53.609305 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:53.756921 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:53.756878 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:54.157415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:54.157380 6202 generic.go:356] "Generic (PLEG): container finished" podID="fbbd574a-a73c-4765-a4c3-e93064c7983c" containerID="72bf8fbf06dc3f6563476b1e148ee169eb7940e98faebe3e75ac197ce35abae7" exitCode=0 Oct 30 17:09:54.157626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:54.157443 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt" event={"ID":"fbbd574a-a73c-4765-a4c3-e93064c7983c","Type":"ContainerDied","Data":"72bf8fbf06dc3f6563476b1e148ee169eb7940e98faebe3e75ac197ce35abae7"} Oct 30 17:09:55.235013 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:55.234933 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:55.355068 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:55.355035 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:55.415413 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:55.415380 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt" Oct 30 17:09:55.555983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:55.555307 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbbd574a-a73c-4765-a4c3-e93064c7983c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbd574a-a73c-4765-a4c3-e93064c7983c-maas-api-openshift-ca\") pod \"fbbd574a-a73c-4765-a4c3-e93064c7983c\" (UID: \"fbbd574a-a73c-4765-a4c3-e93064c7983c\") " Oct 30 17:09:55.555983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:55.555420 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbbd574a-a73c-4765-a4c3-e93064c7983c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbd574a-a73c-4765-a4c3-e93064c7983c-maas-api-xks-ca\") pod \"fbbd574a-a73c-4765-a4c3-e93064c7983c\" (UID: \"fbbd574a-a73c-4765-a4c3-e93064c7983c\") " Oct 30 17:09:55.555983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:55.555699 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbbd574a-a73c-4765-a4c3-e93064c7983c-kube-api-access-5vlc2\" (UniqueName: \"kubernetes.io/projected/fbbd574a-a73c-4765-a4c3-e93064c7983c-kube-api-access-5vlc2\") pod \"fbbd574a-a73c-4765-a4c3-e93064c7983c\" (UID: \"fbbd574a-a73c-4765-a4c3-e93064c7983c\") " Oct 30 17:09:55.583794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:55.566522 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbbd574a-a73c-4765-a4c3-e93064c7983c-maas-api-openshift-ca" pod "fbbd574a-a73c-4765-a4c3-e93064c7983c" (UID: "fbbd574a-a73c-4765-a4c3-e93064c7983c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:55.599000 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:55.593268 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbbd574a-a73c-4765-a4c3-e93064c7983c-maas-api-xks-ca" pod "fbbd574a-a73c-4765-a4c3-e93064c7983c" (UID: "fbbd574a-a73c-4765-a4c3-e93064c7983c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:09:55.652789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:55.640940 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbd574a-a73c-4765-a4c3-e93064c7983c-kube-api-access-5vlc2" pod "fbbd574a-a73c-4765-a4c3-e93064c7983c" (UID: "fbbd574a-a73c-4765-a4c3-e93064c7983c"). InnerVolumeSpecName "kube-api-access-5vlc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:55.689038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:55.688202 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vlc2\" (UniqueName: \"kubernetes.io/projected/fbbd574a-a73c-4765-a4c3-e93064c7983c-kube-api-access-5vlc2\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:55.689038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:55.688235 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbd574a-a73c-4765-a4c3-e93064c7983c-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:55.689038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:55.688251 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbd574a-a73c-4765-a4c3-e93064c7983c-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:09:56.167050 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:56.167001 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:56.178585 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:56.178556 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt" Oct 30 17:09:56.178820 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:56.178579 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt" event={"ID":"fbbd574a-a73c-4765-a4c3-e93064c7983c","Type":"ContainerDied","Data":"ba74ac7e88029d5ae79dfdadb3f590fc930e79c8db89736dad89a346f10e95c7"} Oct 30 17:09:56.178820 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:56.178636 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba74ac7e88029d5ae79dfdadb3f590fc930e79c8db89736dad89a346f10e95c7" Oct 30 17:09:57.930014 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:57.929976 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:58.047258 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:58.044343 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4"] Oct 30 17:09:58.052634 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:58.052524 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba4aaf11-r2zl4"] Oct 30 17:09:58.412395 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:58.412357 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551943de-3741-4bb0-82d3-5bb8cd23a8ef" path="/var/lib/kubelet/pods/551943de-3741-4bb0-82d3-5bb8cd23a8ef/volumes" Oct 30 17:09:59.697616 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:09:59.697581 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:01.502324 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:01.502282 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:03.687204 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:03.687169 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:05.030953 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:05.030917 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc"] Oct 30 17:10:05.044005 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:05.043972 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805ee993-vq8xc"] Oct 30 17:10:05.677484 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:05.677439 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:06.415745 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:06.415707 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28dd80f8-8d21-4d64-9327-104671fbbe0d" path="/var/lib/kubelet/pods/28dd80f8-8d21-4d64-9327-104671fbbe0d/volumes" Oct 30 17:10:07.534101 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:07.534061 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:09.381635 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:09.381597 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:10.832055 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:10.832004 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:11.501580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:11.501544 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:11.672991 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:11.672959 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:11.882301 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:11.882266 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q"] Oct 30 17:10:11.892472 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:11.892446 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d693d0-qtb2q"] Oct 30 17:10:12.075092 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:12.075056 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:12.354122 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:12.354087 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:12.411798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:12.411741 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c7da8ff-a3e0-42e2-a369-6c6351686979" path="/var/lib/kubelet/pods/8c7da8ff-a3e0-42e2-a369-6c6351686979/volumes" Oct 30 17:10:14.011613 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:14.011576 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:14.146627 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:10:14.146580 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:10:15.632942 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:15.632908 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:17.694525 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:17.694483 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:18.906213 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:18.906180 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:19.049792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:19.039416 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:19.063690 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:19.061328 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq"] Oct 30 17:10:19.072497 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:19.072452 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c58226f4-64tvq"] Oct 30 17:10:19.236096 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:19.235996 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c"] Oct 30 17:10:19.236550 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:19.236341 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" podUID="a81b2492-7d46-44ac-a92b-7cd698769695" containerName="istio-proxy" containerID="cri-o://fc2689089f56e955f2acb5c4baef04c6963c29e2fb519737a31bfd741045c755" gracePeriod=30 Oct 30 17:10:19.287314 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:19.287284 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:19.893195 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:19.893156 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:20.413037 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:20.413007 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff05222-a66b-4f7a-994e-4df8c8f2cac5" path="/var/lib/kubelet/pods/fff05222-a66b-4f7a-994e-4df8c8f2cac5/volumes" Oct 30 17:10:20.829671 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:20.829635 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj"] Oct 30 17:10:20.865161 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:20.865126 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft"] Oct 30 17:10:20.968786 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:20.968728 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:21.108645 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:21.108548 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:21.217936 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:21.217884 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:21.920441 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:21.920398 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:23.154966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:23.154929 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:23.351317 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:23.351282 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:24.304522 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.304490 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:24.428270 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.428242 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:24.585824 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.585786 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:24.635089 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.635053 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 17:10:24.698283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.696812 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a81b2492-7d46-44ac-a92b-7cd698769695-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81b2492-7d46-44ac-a92b-7cd698769695-istio-podinfo\") pod \"a81b2492-7d46-44ac-a92b-7cd698769695\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " Oct 30 17:10:24.698283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.696889 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a81b2492-7d46-44ac-a92b-7cd698769695-kube-api-access-trlp2\" (UniqueName: \"kubernetes.io/projected/a81b2492-7d46-44ac-a92b-7cd698769695-kube-api-access-trlp2\") pod \"a81b2492-7d46-44ac-a92b-7cd698769695\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " Oct 30 17:10:24.698283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.696922 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-istio-envoy\") pod \"a81b2492-7d46-44ac-a92b-7cd698769695\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " Oct 30 17:10:24.698283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.697013 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-workload-socket\") pod \"a81b2492-7d46-44ac-a92b-7cd698769695\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " Oct 30 17:10:24.698283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.697055 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-istio-data\") pod \"a81b2492-7d46-44ac-a92b-7cd698769695\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " Oct 30 17:10:24.698283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.697168 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a81b2492-7d46-44ac-a92b-7cd698769695-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81b2492-7d46-44ac-a92b-7cd698769695-istiod-ca-cert\") pod \"a81b2492-7d46-44ac-a92b-7cd698769695\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " Oct 30 17:10:24.698283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.697310 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a81b2492-7d46-44ac-a92b-7cd698769695-istio-token\" (UniqueName: \"kubernetes.io/projected/a81b2492-7d46-44ac-a92b-7cd698769695-istio-token\") pod \"a81b2492-7d46-44ac-a92b-7cd698769695\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " Oct 30 17:10:24.698283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.697412 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-workload-certs\") pod \"a81b2492-7d46-44ac-a92b-7cd698769695\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " Oct 30 17:10:24.698283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.697447 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-credential-socket\") pod \"a81b2492-7d46-44ac-a92b-7cd698769695\" (UID: \"a81b2492-7d46-44ac-a92b-7cd698769695\") " Oct 30 17:10:24.699061 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.698900 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-credential-socket" pod "a81b2492-7d46-44ac-a92b-7cd698769695" (UID: "a81b2492-7d46-44ac-a92b-7cd698769695"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:24.699147 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.699119 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-istio-data" pod "a81b2492-7d46-44ac-a92b-7cd698769695" (UID: "a81b2492-7d46-44ac-a92b-7cd698769695"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:24.699823 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.699562 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-workload-socket" pod "a81b2492-7d46-44ac-a92b-7cd698769695" (UID: "a81b2492-7d46-44ac-a92b-7cd698769695"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:24.699823 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.699782 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a81b2492-7d46-44ac-a92b-7cd698769695-istiod-ca-cert" pod "a81b2492-7d46-44ac-a92b-7cd698769695" (UID: "a81b2492-7d46-44ac-a92b-7cd698769695"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:24.700308 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.700287 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-workload-certs" pod "a81b2492-7d46-44ac-a92b-7cd698769695" (UID: "a81b2492-7d46-44ac-a92b-7cd698769695"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:24.702338 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.702315 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81b2492-7d46-44ac-a92b-7cd698769695-kube-api-access-trlp2" pod "a81b2492-7d46-44ac-a92b-7cd698769695" (UID: "a81b2492-7d46-44ac-a92b-7cd698769695"). InnerVolumeSpecName "kube-api-access-trlp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:24.702724 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.702675 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81b2492-7d46-44ac-a92b-7cd698769695-istio-token" pod "a81b2492-7d46-44ac-a92b-7cd698769695" (UID: "a81b2492-7d46-44ac-a92b-7cd698769695"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:24.702724 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.702687 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a81b2492-7d46-44ac-a92b-7cd698769695-istio-podinfo" pod "a81b2492-7d46-44ac-a92b-7cd698769695" (UID: "a81b2492-7d46-44ac-a92b-7cd698769695"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:10:24.702903 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.702811 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-istio-envoy" pod "a81b2492-7d46-44ac-a92b-7cd698769695" (UID: "a81b2492-7d46-44ac-a92b-7cd698769695"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:24.786243 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.786203 6202 generic.go:356] "Generic (PLEG): container finished" podID="a81b2492-7d46-44ac-a92b-7cd698769695" containerID="fc2689089f56e955f2acb5c4baef04c6963c29e2fb519737a31bfd741045c755" exitCode=0 Oct 30 17:10:24.786409 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.786261 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" Oct 30 17:10:24.786409 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.786291 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" event={"ID":"a81b2492-7d46-44ac-a92b-7cd698769695","Type":"ContainerDied","Data":"fc2689089f56e955f2acb5c4baef04c6963c29e2fb519737a31bfd741045c755"} Oct 30 17:10:24.786409 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.786346 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c" event={"ID":"a81b2492-7d46-44ac-a92b-7cd698769695","Type":"ContainerDied","Data":"3f6226b3f1317817da896e05ecd421dd91e94a7d67255e6fef3ad5a6dd46a27c"} Oct 30 17:10:24.786409 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.786377 6202 scope.go:122] "RemoveContainer" containerID="fc2689089f56e955f2acb5c4baef04c6963c29e2fb519737a31bfd741045c755" Oct 30 17:10:24.801054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.800915 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81b2492-7d46-44ac-a92b-7cd698769695-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.801054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.800941 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trlp2\" (UniqueName: \"kubernetes.io/projected/a81b2492-7d46-44ac-a92b-7cd698769695-kube-api-access-trlp2\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.801054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.800958 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.801054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.800974 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.801054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.800985 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.801054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.800997 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81b2492-7d46-44ac-a92b-7cd698769695-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.801054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.801010 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a81b2492-7d46-44ac-a92b-7cd698769695-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.801054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.801022 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.801054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.801034 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81b2492-7d46-44ac-a92b-7cd698769695-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.805717 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.805691 6202 scope.go:122] "RemoveContainer" containerID="ac89cb3f5cacedb924bfd0c405e25b2741c138215753220e7bc234e19dcfc803" Oct 30 17:10:24.816406 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.816390 6202 scope.go:122] "RemoveContainer" containerID="fc2689089f56e955f2acb5c4baef04c6963c29e2fb519737a31bfd741045c755" Oct 30 17:10:24.816682 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:10:24.816661 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc2689089f56e955f2acb5c4baef04c6963c29e2fb519737a31bfd741045c755\": container with ID starting with fc2689089f56e955f2acb5c4baef04c6963c29e2fb519737a31bfd741045c755 not found: ID does not exist" containerID="fc2689089f56e955f2acb5c4baef04c6963c29e2fb519737a31bfd741045c755" Oct 30 17:10:24.816743 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.816690 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc2689089f56e955f2acb5c4baef04c6963c29e2fb519737a31bfd741045c755"} err="failed to get container status \"fc2689089f56e955f2acb5c4baef04c6963c29e2fb519737a31bfd741045c755\": rpc error: code = NotFound desc = could not find container \"fc2689089f56e955f2acb5c4baef04c6963c29e2fb519737a31bfd741045c755\": container with ID starting with fc2689089f56e955f2acb5c4baef04c6963c29e2fb519737a31bfd741045c755 not found: ID does not exist" Oct 30 17:10:24.816743 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.816708 6202 scope.go:122] "RemoveContainer" containerID="ac89cb3f5cacedb924bfd0c405e25b2741c138215753220e7bc234e19dcfc803" Oct 30 17:10:24.816946 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:10:24.816931 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac89cb3f5cacedb924bfd0c405e25b2741c138215753220e7bc234e19dcfc803\": container with ID starting with ac89cb3f5cacedb924bfd0c405e25b2741c138215753220e7bc234e19dcfc803 not found: ID does not exist" containerID="ac89cb3f5cacedb924bfd0c405e25b2741c138215753220e7bc234e19dcfc803" Oct 30 17:10:24.817008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.816951 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac89cb3f5cacedb924bfd0c405e25b2741c138215753220e7bc234e19dcfc803"} err="failed to get container status \"ac89cb3f5cacedb924bfd0c405e25b2741c138215753220e7bc234e19dcfc803\": rpc error: code = NotFound desc = could not find container \"ac89cb3f5cacedb924bfd0c405e25b2741c138215753220e7bc234e19dcfc803\": container with ID starting with ac89cb3f5cacedb924bfd0c405e25b2741c138215753220e7bc234e19dcfc803 not found: ID does not exist" Oct 30 17:10:24.847410 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.847336 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c"] Oct 30 17:10:24.851752 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:24.851721 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-npv7c"] Oct 30 17:10:25.830303 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:25.830267 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj" podUID="c7b73230-da08-40d4-8ecb-93cf4c5a6b6f" containerName="payload-processing" containerID="cri-o://770e6a9a06b1032b42342d6c86512d92858b696ccff7ab3793f0404453d1870b" gracePeriod=25 Oct 30 17:10:25.866268 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:25.866231 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft" podUID="9bba5eae-bd9d-4134-8945-15ec55fe4391" containerName="payload-pre-processing" containerID="cri-o://4a56376d3426b735dc78b68e4974ebdd8c73d9d78c314a66201f38f21c7cb917" gracePeriod=25 Oct 30 17:10:26.032407 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.032372 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q"] Oct 30 17:10:26.044888 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.044862 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f76c6767-7x94q"] Oct 30 17:10:26.184902 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.184877 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj" Oct 30 17:10:26.207136 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.207107 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft" Oct 30 17:10:26.331515 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.331480 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b73230-da08-40d4-8ecb-93cf4c5a6b6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b73230-da08-40d4-8ecb-93cf4c5a6b6f-plugins-config-volume\") pod \"c7b73230-da08-40d4-8ecb-93cf4c5a6b6f\" (UID: \"c7b73230-da08-40d4-8ecb-93cf4c5a6b6f\") " Oct 30 17:10:26.331713 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.331627 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bba5eae-bd9d-4134-8945-15ec55fe4391-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bba5eae-bd9d-4134-8945-15ec55fe4391-plugins-config-volume\") pod \"9bba5eae-bd9d-4134-8945-15ec55fe4391\" (UID: \"9bba5eae-bd9d-4134-8945-15ec55fe4391\") " Oct 30 17:10:26.331713 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.331705 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b73230-da08-40d4-8ecb-93cf4c5a6b6f-kube-api-access-6q4kj\" (UniqueName: \"kubernetes.io/projected/c7b73230-da08-40d4-8ecb-93cf4c5a6b6f-kube-api-access-6q4kj\") pod \"c7b73230-da08-40d4-8ecb-93cf4c5a6b6f\" (UID: \"c7b73230-da08-40d4-8ecb-93cf4c5a6b6f\") " Oct 30 17:10:26.331925 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.331863 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bba5eae-bd9d-4134-8945-15ec55fe4391-kube-api-access-csvsh\" (UniqueName: \"kubernetes.io/projected/9bba5eae-bd9d-4134-8945-15ec55fe4391-kube-api-access-csvsh\") pod \"9bba5eae-bd9d-4134-8945-15ec55fe4391\" (UID: \"9bba5eae-bd9d-4134-8945-15ec55fe4391\") " Oct 30 17:10:26.331995 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.331968 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b73230-da08-40d4-8ecb-93cf4c5a6b6f-plugins-config-volume" pod "c7b73230-da08-40d4-8ecb-93cf4c5a6b6f" (UID: "c7b73230-da08-40d4-8ecb-93cf4c5a6b6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:26.332466 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.332439 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bba5eae-bd9d-4134-8945-15ec55fe4391-plugins-config-volume" pod "9bba5eae-bd9d-4134-8945-15ec55fe4391" (UID: "9bba5eae-bd9d-4134-8945-15ec55fe4391"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:26.332712 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.332692 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b73230-da08-40d4-8ecb-93cf4c5a6b6f-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:10:26.332840 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.332718 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bba5eae-bd9d-4134-8945-15ec55fe4391-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:10:26.334477 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.334453 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b73230-da08-40d4-8ecb-93cf4c5a6b6f-kube-api-access-6q4kj" pod "c7b73230-da08-40d4-8ecb-93cf4c5a6b6f" (UID: "c7b73230-da08-40d4-8ecb-93cf4c5a6b6f"). InnerVolumeSpecName "kube-api-access-6q4kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:26.334553 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.334488 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bba5eae-bd9d-4134-8945-15ec55fe4391-kube-api-access-csvsh" pod "9bba5eae-bd9d-4134-8945-15ec55fe4391" (UID: "9bba5eae-bd9d-4134-8945-15ec55fe4391"). InnerVolumeSpecName "kube-api-access-csvsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:26.371599 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.371570 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:26.425296 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.425218 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a" path="/var/lib/kubelet/pods/6c3964f6-35bd-4cbb-a70f-76b7eeff0d6a/volumes" Oct 30 17:10:26.426282 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.426256 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a81b2492-7d46-44ac-a92b-7cd698769695" path="/var/lib/kubelet/pods/a81b2492-7d46-44ac-a92b-7cd698769695/volumes" Oct 30 17:10:26.452081 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.452045 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q4kj\" (UniqueName: \"kubernetes.io/projected/c7b73230-da08-40d4-8ecb-93cf4c5a6b6f-kube-api-access-6q4kj\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:10:26.473110 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.473078 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csvsh\" (UniqueName: \"kubernetes.io/projected/9bba5eae-bd9d-4134-8945-15ec55fe4391-kube-api-access-csvsh\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:10:26.806617 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.806513 6202 generic.go:356] "Generic (PLEG): container finished" podID="9bba5eae-bd9d-4134-8945-15ec55fe4391" containerID="4a56376d3426b735dc78b68e4974ebdd8c73d9d78c314a66201f38f21c7cb917" exitCode=0 Oct 30 17:10:26.806828 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.806643 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft" Oct 30 17:10:26.806828 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.806686 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft" event={"ID":"9bba5eae-bd9d-4134-8945-15ec55fe4391","Type":"ContainerDied","Data":"4a56376d3426b735dc78b68e4974ebdd8c73d9d78c314a66201f38f21c7cb917"} Oct 30 17:10:26.806828 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.806794 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft" event={"ID":"9bba5eae-bd9d-4134-8945-15ec55fe4391","Type":"ContainerDied","Data":"9485e64d74f8756e712b60abd1626aebe1235e8c3b3056877d664e5e08973e1e"} Oct 30 17:10:26.807011 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.806833 6202 scope.go:122] "RemoveContainer" containerID="4a56376d3426b735dc78b68e4974ebdd8c73d9d78c314a66201f38f21c7cb917" Oct 30 17:10:26.808848 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.808824 6202 generic.go:356] "Generic (PLEG): container finished" podID="c7b73230-da08-40d4-8ecb-93cf4c5a6b6f" containerID="770e6a9a06b1032b42342d6c86512d92858b696ccff7ab3793f0404453d1870b" exitCode=0 Oct 30 17:10:26.808960 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.808888 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj" Oct 30 17:10:26.809039 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.808967 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj" event={"ID":"c7b73230-da08-40d4-8ecb-93cf4c5a6b6f","Type":"ContainerDied","Data":"770e6a9a06b1032b42342d6c86512d92858b696ccff7ab3793f0404453d1870b"} Oct 30 17:10:26.809103 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.809066 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj" event={"ID":"c7b73230-da08-40d4-8ecb-93cf4c5a6b6f","Type":"ContainerDied","Data":"4bafa4b0e31c670e2c5eba57d83bc13886992dc684ad96c9eadb5623f3155e3c"} Oct 30 17:10:26.818739 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.818720 6202 scope.go:122] "RemoveContainer" containerID="4a56376d3426b735dc78b68e4974ebdd8c73d9d78c314a66201f38f21c7cb917" Oct 30 17:10:26.819070 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:10:26.819050 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a56376d3426b735dc78b68e4974ebdd8c73d9d78c314a66201f38f21c7cb917\": container with ID starting with 4a56376d3426b735dc78b68e4974ebdd8c73d9d78c314a66201f38f21c7cb917 not found: ID does not exist" containerID="4a56376d3426b735dc78b68e4974ebdd8c73d9d78c314a66201f38f21c7cb917" Oct 30 17:10:26.819124 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.819080 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a56376d3426b735dc78b68e4974ebdd8c73d9d78c314a66201f38f21c7cb917"} err="failed to get container status \"4a56376d3426b735dc78b68e4974ebdd8c73d9d78c314a66201f38f21c7cb917\": rpc error: code = NotFound desc = could not find container \"4a56376d3426b735dc78b68e4974ebdd8c73d9d78c314a66201f38f21c7cb917\": container with ID starting with 4a56376d3426b735dc78b68e4974ebdd8c73d9d78c314a66201f38f21c7cb917 not found: ID does not exist" Oct 30 17:10:26.819124 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.819102 6202 scope.go:122] "RemoveContainer" containerID="770e6a9a06b1032b42342d6c86512d92858b696ccff7ab3793f0404453d1870b" Oct 30 17:10:26.830353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.830319 6202 scope.go:122] "RemoveContainer" containerID="770e6a9a06b1032b42342d6c86512d92858b696ccff7ab3793f0404453d1870b" Oct 30 17:10:26.830674 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:10:26.830631 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"770e6a9a06b1032b42342d6c86512d92858b696ccff7ab3793f0404453d1870b\": container with ID starting with 770e6a9a06b1032b42342d6c86512d92858b696ccff7ab3793f0404453d1870b not found: ID does not exist" containerID="770e6a9a06b1032b42342d6c86512d92858b696ccff7ab3793f0404453d1870b" Oct 30 17:10:26.830740 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.830669 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770e6a9a06b1032b42342d6c86512d92858b696ccff7ab3793f0404453d1870b"} err="failed to get container status \"770e6a9a06b1032b42342d6c86512d92858b696ccff7ab3793f0404453d1870b\": rpc error: code = NotFound desc = could not find container \"770e6a9a06b1032b42342d6c86512d92858b696ccff7ab3793f0404453d1870b\": container with ID starting with 770e6a9a06b1032b42342d6c86512d92858b696ccff7ab3793f0404453d1870b not found: ID does not exist" Oct 30 17:10:26.865847 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.865814 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft"] Oct 30 17:10:26.878749 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.878722 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-8b85799f-9wkft"] Oct 30 17:10:26.934312 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.934268 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj"] Oct 30 17:10:26.942722 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:26.941232 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97a47d-76cdc49c86-2rmlj"] Oct 30 17:10:28.064314 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:28.064283 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:28.413275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:28.413232 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bba5eae-bd9d-4134-8945-15ec55fe4391" path="/var/lib/kubelet/pods/9bba5eae-bd9d-4134-8945-15ec55fe4391/volumes" Oct 30 17:10:28.413829 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:28.413807 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b73230-da08-40d4-8ecb-93cf4c5a6b6f" path="/var/lib/kubelet/pods/c7b73230-da08-40d4-8ecb-93cf4c5a6b6f/volumes" Oct 30 17:10:29.795566 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:29.795533 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:32.187139 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:32.187093 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:33.717235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:33.717202 6202 scope.go:122] "RemoveContainer" containerID="ffe2cd96c4bd7c1c39410cb08cc3f44b67dadf697448f1e9e01e3181041054c9" Oct 30 17:10:33.728858 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:33.728833 6202 scope.go:122] "RemoveContainer" containerID="81197c09ff1c6d38b3b5d2211d7dce5a89939be1ca4572532f114bcce1ea9db7" Oct 30 17:10:33.739474 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:33.739453 6202 scope.go:122] "RemoveContainer" containerID="36e0c77db599f86dd23a18f17d3958812e1929e67bc9d457ec9f4ef9600beeb1" Oct 30 17:10:33.843186 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:33.843154 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:35.552610 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:35.552577 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:37.388384 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:37.388349 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:38.849486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:38.849445 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:39.160340 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.160258 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:39.308545 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.308512 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7"] Oct 30 17:10:39.309366 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.309347 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c7b73230-da08-40d4-8ecb-93cf4c5a6b6f" containerName="payload-processing" Oct 30 17:10:39.309423 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.309367 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b73230-da08-40d4-8ecb-93cf4c5a6b6f" containerName="payload-processing" Oct 30 17:10:39.309423 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.309376 6202 cpu_manager.go:414] "containerMap: removing container" podUID="fbbd574a-a73c-4765-a4c3-e93064c7983c" containerName="revoke-keys" Oct 30 17:10:39.309423 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.309382 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbd574a-a73c-4765-a4c3-e93064c7983c" containerName="revoke-keys" Oct 30 17:10:39.309423 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.309403 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a81b2492-7d46-44ac-a92b-7cd698769695" containerName="istio-proxy" Oct 30 17:10:39.309423 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.309411 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81b2492-7d46-44ac-a92b-7cd698769695" containerName="istio-proxy" Oct 30 17:10:39.309603 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.309437 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a81b2492-7d46-44ac-a92b-7cd698769695" containerName="istio-proxy" Oct 30 17:10:39.309603 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.309441 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81b2492-7d46-44ac-a92b-7cd698769695" containerName="istio-proxy" Oct 30 17:10:39.309603 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.309451 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9bba5eae-bd9d-4134-8945-15ec55fe4391" containerName="payload-pre-processing" Oct 30 17:10:39.309603 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.309455 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bba5eae-bd9d-4134-8945-15ec55fe4391" containerName="payload-pre-processing" Oct 30 17:10:39.309603 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.309574 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b73230-da08-40d4-8ecb-93cf4c5a6b6f" containerName="payload-processing" Oct 30 17:10:39.309603 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.309582 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbbd574a-a73c-4765-a4c3-e93064c7983c" containerName="revoke-keys" Oct 30 17:10:39.309603 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.309590 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81b2492-7d46-44ac-a92b-7cd698769695" containerName="istio-proxy" Oct 30 17:10:39.309603 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.309601 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bba5eae-bd9d-4134-8945-15ec55fe4391" containerName="payload-pre-processing" Oct 30 17:10:39.315857 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.315832 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.369197 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.369142 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-4def7e-openshift-default-dockercfg-vlpbf\"" Oct 30 17:10:39.370345 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.370313 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:39.374738 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.374704 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7"] Oct 30 17:10:39.533850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.530238 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-workload-certs\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.533850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.530414 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-podinfo\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.533850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.530497 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbcf3ac0-0197-4960-8672-2fd5d9053729-istiod-ca-cert\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.533850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.530722 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-data\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.533850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.530938 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-token\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.533850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.531115 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-credential-socket\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.533850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.531177 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-workload-socket\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.533850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.531206 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djvvw\" (UniqueName: \"kubernetes.io/projected/bbcf3ac0-0197-4960-8672-2fd5d9053729-kube-api-access-djvvw\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.533850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.531374 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-envoy\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.533850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.531949 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:39.633790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.632845 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-workload-certs\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.633790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.632938 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-podinfo\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.633790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.632971 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbcf3ac0-0197-4960-8672-2fd5d9053729-istiod-ca-cert\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.633790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.633081 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-data\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.633790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.633190 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-token\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.633790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.633312 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-credential-socket\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.633790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.633364 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-workload-socket\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.633790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.633392 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djvvw\" (UniqueName: \"kubernetes.io/projected/bbcf3ac0-0197-4960-8672-2fd5d9053729-kube-api-access-djvvw\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.633790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.633423 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-envoy\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.634528 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.634160 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-data\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.634528 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.634467 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-credential-socket\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.634863 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.634832 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbcf3ac0-0197-4960-8672-2fd5d9053729-istiod-ca-cert\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.635080 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.635056 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-workload-certs\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.648790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.635524 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-workload-socket\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.659790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.657365 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-podinfo\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.679501 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.679455 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-envoy\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.687418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.687334 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-token\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.699926 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.699504 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djvvw\" (UniqueName: \"kubernetes.io/projected/bbcf3ac0-0197-4960-8672-2fd5d9053729-kube-api-access-djvvw\") pod \"e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:39.940072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:39.940037 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:40.259600 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:10:40.259556 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbcf3ac0_0197_4960_8672_2fd5d9053729.slice/crio-6774cd66fac4e0c3903c2592811757f1133df4956a8212acad20694ba74542cd WatchSource:0}: Error finding container 6774cd66fac4e0c3903c2592811757f1133df4956a8212acad20694ba74542cd: Status 404 returned error can't find the container with id 6774cd66fac4e0c3903c2592811757f1133df4956a8212acad20694ba74542cd Oct 30 17:10:40.262873 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:40.262844 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7"] Oct 30 17:10:40.263645 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:40.263554 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:10:40.263780 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:40.263702 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:10:40.263780 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:40.263752 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:10:40.959952 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:40.959910 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" event={"ID":"bbcf3ac0-0197-4960-8672-2fd5d9053729","Type":"ContainerStarted","Data":"44f4a301993e7495cd2e75baae830b425b878d7daabe9e253fe16b8483cd50df"} Oct 30 17:10:40.960471 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:40.959984 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" event={"ID":"bbcf3ac0-0197-4960-8672-2fd5d9053729","Type":"ContainerStarted","Data":"6774cd66fac4e0c3903c2592811757f1133df4956a8212acad20694ba74542cd"} Oct 30 17:10:41.036051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:41.035996 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" podStartSLOduration=2.035976918 podStartE2EDuration="2.035976918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:41.02209964 +0000 UTC m=+2954.732554383" watchObservedRunningTime="2026-10-30 17:10:41.035976918 +0000 UTC m=+2954.746431664" Oct 30 17:10:41.181607 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:41.181570 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:41.940676 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:41.940644 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:41.947753 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:41.947724 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:41.967138 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:41.967096 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:41.969027 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:41.969002 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:41.969850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:41.969833 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:10:42.085811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:42.082728 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:42.197461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:42.197374 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:42.609547 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:42.609501 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:42.862415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:42.862298 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:43.467960 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:43.466903 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4"] Oct 30 17:10:43.468475 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:43.468428 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81b2492-7d46-44ac-a92b-7cd698769695" containerName="istio-proxy" Oct 30 17:10:43.475838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:43.473170 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4" Oct 30 17:10:43.506631 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:43.506599 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4ac125\"" Oct 30 17:10:43.507065 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:43.507047 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4ac125-dockercfg-d2qlt\"" Oct 30 17:10:43.514501 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:43.512390 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4"] Oct 30 17:10:43.517532 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:43.517494 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:43.580789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:43.579383 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16c3ab98-a5d7-4895-bdf0-5f584c1f510c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4\" (UID: \"16c3ab98-a5d7-4895-bdf0-5f584c1f510c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4" Oct 30 17:10:43.580789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:43.579469 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pfqz\" (UniqueName: \"kubernetes.io/projected/16c3ab98-a5d7-4895-bdf0-5f584c1f510c-kube-api-access-6pfqz\") pod \"payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4\" (UID: \"16c3ab98-a5d7-4895-bdf0-5f584c1f510c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4" Oct 30 17:10:43.665571 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:43.665534 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:43.681660 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:43.681617 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16c3ab98-a5d7-4895-bdf0-5f584c1f510c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4\" (UID: \"16c3ab98-a5d7-4895-bdf0-5f584c1f510c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4" Oct 30 17:10:43.681931 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:43.681709 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pfqz\" (UniqueName: \"kubernetes.io/projected/16c3ab98-a5d7-4895-bdf0-5f584c1f510c-kube-api-access-6pfqz\") pod \"payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4\" (UID: \"16c3ab98-a5d7-4895-bdf0-5f584c1f510c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4" Oct 30 17:10:43.682410 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:43.682390 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16c3ab98-a5d7-4895-bdf0-5f584c1f510c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4\" (UID: \"16c3ab98-a5d7-4895-bdf0-5f584c1f510c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4" Oct 30 17:10:43.710969 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:43.710931 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pfqz\" (UniqueName: \"kubernetes.io/projected/16c3ab98-a5d7-4895-bdf0-5f584c1f510c-kube-api-access-6pfqz\") pod \"payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4\" (UID: \"16c3ab98-a5d7-4895-bdf0-5f584c1f510c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4" Oct 30 17:10:43.830151 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:43.830120 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4" Oct 30 17:10:44.119802 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:10:44.119694 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:10:44.141802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:44.137749 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:44.285279 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:44.285201 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4"] Oct 30 17:10:44.289517 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:10:44.289486 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c3ab98_a5d7_4895_bdf0_5f584c1f510c.slice/crio-b82e9eb7da590565d403973f8e3f1b2118d1fdaadab8cc027a26a2ab5b644f89 WatchSource:0}: Error finding container b82e9eb7da590565d403973f8e3f1b2118d1fdaadab8cc027a26a2ab5b644f89: Status 404 returned error can't find the container with id b82e9eb7da590565d403973f8e3f1b2118d1fdaadab8cc027a26a2ab5b644f89 Oct 30 17:10:45.212235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:45.212200 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4" event={"ID":"16c3ab98-a5d7-4895-bdf0-5f584c1f510c","Type":"ContainerStarted","Data":"af61721dbb7cbb0643728dc5c046bc8f857ef1a71b054cc7c139f5308b143001"} Oct 30 17:10:45.213093 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:45.212954 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4" event={"ID":"16c3ab98-a5d7-4895-bdf0-5f584c1f510c","Type":"ContainerStarted","Data":"b82e9eb7da590565d403973f8e3f1b2118d1fdaadab8cc027a26a2ab5b644f89"} Oct 30 17:10:45.213646 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:45.213620 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4" Oct 30 17:10:45.289911 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:45.289862 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:45.293962 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:45.293912 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4" podStartSLOduration=2.293899623 podStartE2EDuration="2.293899623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:45.288157052 +0000 UTC m=+2958.998611822" watchObservedRunningTime="2026-10-30 17:10:45.293899623 +0000 UTC m=+2959.004354367" Oct 30 17:10:45.622793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:45.622314 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:46.679009 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:46.678915 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:48.477982 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:48.477902 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:48.592695 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:48.592610 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:50.323212 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:50.323173 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:52.538746 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:52.538707 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:53.333047 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:53.333010 6202 generic.go:356] "Generic (PLEG): container finished" podID="bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" containerID="1ae008250328678a3eafc9e9c153fde7b99dba55cb6f0b19cbba29a784e1521b" exitCode=0 Oct 30 17:10:53.333315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:53.333077 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" event={"ID":"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9","Type":"ContainerDied","Data":"1ae008250328678a3eafc9e9c153fde7b99dba55cb6f0b19cbba29a784e1521b"} Oct 30 17:10:53.333519 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:53.333499 6202 scope.go:122] "RemoveContainer" containerID="1ae008250328678a3eafc9e9c153fde7b99dba55cb6f0b19cbba29a784e1521b" Oct 30 17:10:53.335462 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:53.335431 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:10:53.335577 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:53.335505 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:10:53.335577 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:53.335539 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:10:54.377938 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:54.377899 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" event={"ID":"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9","Type":"ContainerStarted","Data":"0574e77ea7356cbbaf388ff42559d5378a52ea15581a759e3145645ce020b607"} Oct 30 17:10:54.391913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:54.391867 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:54.756852 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:54.754165 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:10:54.770153 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:54.770112 6202 patch_prober.go:29] interesting pod/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body= Oct 30 17:10:54.770380 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:54.770211 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" podUID="bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" Oct 30 17:10:55.753793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:55.753210 6202 patch_prober.go:29] interesting pod/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body= Oct 30 17:10:55.753793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:55.753274 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" podUID="bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" Oct 30 17:10:56.114947 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:56.114913 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:10:56.118946 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:56.118920 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:56.227863 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:56.227824 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4" Oct 30 17:10:56.753240 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:56.753199 6202 patch_prober.go:29] interesting pod/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body= Oct 30 17:10:56.753562 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:56.753532 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" podUID="bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" Oct 30 17:10:57.589899 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:57.589855 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:57.758729 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:57.758679 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:57.765844 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:57.765807 6202 patch_prober.go:29] interesting pod/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body= Oct 30 17:10:57.766147 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:57.766115 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" podUID="bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" Oct 30 17:10:57.875496 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:57.875364 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:58.275924 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:58.275811 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:58.347813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:58.340981 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx"] Oct 30 17:10:58.349729 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:58.349695 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-690e7e86-krkbx"] Oct 30 17:10:58.391524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:58.391467 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt"] Oct 30 17:10:58.392546 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:58.392499 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:58.398520 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:58.398492 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b16279-xmrdt"] Oct 30 17:10:58.415463 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:58.415431 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb" path="/var/lib/kubelet/pods/f2759d9a-b6e1-4b4f-8caa-50e4ae8864fb/volumes" Oct 30 17:10:58.416043 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:58.416022 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbbd574a-a73c-4765-a4c3-e93064c7983c" path="/var/lib/kubelet/pods/fbbd574a-a73c-4765-a4c3-e93064c7983c/volumes" Oct 30 17:10:58.581857 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:58.571978 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:58.753215 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:58.753175 6202 patch_prober.go:29] interesting pod/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body= Oct 30 17:10:58.753871 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:58.753832 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" podUID="bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" Oct 30 17:10:58.870232 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:58.870141 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:59.764791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:59.760194 6202 patch_prober.go:29] interesting pod/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body= Oct 30 17:10:59.764791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:10:59.760295 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" podUID="bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" Oct 30 17:11:00.613994 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:00.613962 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:00.755154 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:00.755124 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:11:00.756275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:00.756252 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:11:01.476641 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:01.476606 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:01.950176 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:01.950135 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:02.480320 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:02.480282 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:02.630783 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:02.630731 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:04.493688 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:04.493655 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:05.060442 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:05.060408 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:05.176483 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:05.176453 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:05.378097 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:05.378008 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t"] Oct 30 17:11:05.378545 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:05.378505 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" podUID="cd39addc-aef3-41d3-aae4-6358ffbdf53e" containerName="istio-proxy" containerID="cri-o://4530bfe5562f98ab30ee7a33dfef936b647d877c98f64832a0b845e0e8b8a007" gracePeriod=30 Oct 30 17:11:05.389629 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:05.389602 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:05.812317 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:05.812232 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:05.907488 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:05.907455 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd"] Oct 30 17:11:05.912576 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:05.912552 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd" Oct 30 17:11:05.917377 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:05.917336 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd\" (UID: \"fe01f458-6837-424c-a9cb-4a6a2a3ee5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd" Oct 30 17:11:05.918471 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:05.917553 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd\" (UID: \"fe01f458-6837-424c-a9cb-4a6a2a3ee5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd" Oct 30 17:11:05.918471 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:05.917787 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssb4v\" (UniqueName: \"kubernetes.io/projected/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-kube-api-access-ssb4v\") pod \"maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd\" (UID: \"fe01f458-6837-424c-a9cb-4a6a2a3ee5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd" Oct 30 17:11:05.934936 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:05.934903 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:05.940153 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:05.940096 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:11:05.941463 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:05.940268 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:11:05.942865 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:05.942837 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd"] Oct 30 17:11:05.949395 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:05.949364 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:11:05.949523 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:05.949434 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:11:06.019592 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:06.019553 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssb4v\" (UniqueName: \"kubernetes.io/projected/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-kube-api-access-ssb4v\") pod \"maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd\" (UID: \"fe01f458-6837-424c-a9cb-4a6a2a3ee5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd" Oct 30 17:11:06.019835 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:06.019656 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd\" (UID: \"fe01f458-6837-424c-a9cb-4a6a2a3ee5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd" Oct 30 17:11:06.019835 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:06.019811 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd\" (UID: \"fe01f458-6837-424c-a9cb-4a6a2a3ee5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd" Oct 30 17:11:06.020369 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:06.020346 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd\" (UID: \"fe01f458-6837-424c-a9cb-4a6a2a3ee5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd" Oct 30 17:11:06.022545 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:06.022516 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd\" (UID: \"fe01f458-6837-424c-a9cb-4a6a2a3ee5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd" Oct 30 17:11:06.047581 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:06.047549 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssb4v\" (UniqueName: \"kubernetes.io/projected/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-kube-api-access-ssb4v\") pod \"maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd\" (UID: \"fe01f458-6837-424c-a9cb-4a6a2a3ee5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd" Oct 30 17:11:06.241907 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:06.241876 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd" Oct 30 17:11:06.549177 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:11:06.549139 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe01f458_6837_424c_a9cb_4a6a2a3ee5f0.slice/crio-5b395b9339270ee9f66fa080f4f1a21fc8f727687691af823793d4c9df13be3d WatchSource:0}: Error finding container 5b395b9339270ee9f66fa080f4f1a21fc8f727687691af823793d4c9df13be3d: Status 404 returned error can't find the container with id 5b395b9339270ee9f66fa080f4f1a21fc8f727687691af823793d4c9df13be3d Oct 30 17:11:06.550621 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:06.550592 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd"] Oct 30 17:11:06.572570 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:06.572536 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd" event={"ID":"fe01f458-6837-424c-a9cb-4a6a2a3ee5f0","Type":"ContainerStarted","Data":"5b395b9339270ee9f66fa080f4f1a21fc8f727687691af823793d4c9df13be3d"} Oct 30 17:11:06.622929 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:06.622894 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:07.425737 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:07.425701 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:07.547711 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:07.547678 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:07.585569 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:07.585536 6202 generic.go:356] "Generic (PLEG): container finished" podID="fe01f458-6837-424c-a9cb-4a6a2a3ee5f0" containerID="273da7149749dd0e9902855ac586532004f13f2e6c96e5dfdbebb382ba0248af" exitCode=0 Oct 30 17:11:07.585754 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:07.585631 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd" event={"ID":"fe01f458-6837-424c-a9cb-4a6a2a3ee5f0","Type":"ContainerDied","Data":"273da7149749dd0e9902855ac586532004f13f2e6c96e5dfdbebb382ba0248af"} Oct 30 17:11:08.031110 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:08.031067 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m"] Oct 30 17:11:08.035071 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:08.035044 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f6b00e4-6jz7m"] Oct 30 17:11:08.414115 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:08.414082 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c003e7f-5973-4479-88e5-cacb12f92fba" path="/var/lib/kubelet/pods/3c003e7f-5973-4479-88e5-cacb12f92fba/volumes" Oct 30 17:11:08.581177 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:08.581143 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:08.821973 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:08.821948 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd" Oct 30 17:11:08.920005 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:08.919977 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-maas-api-xks-ca\") pod \"fe01f458-6837-424c-a9cb-4a6a2a3ee5f0\" (UID: \"fe01f458-6837-424c-a9cb-4a6a2a3ee5f0\") " Oct 30 17:11:08.920222 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:08.920047 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-kube-api-access-ssb4v\" (UniqueName: \"kubernetes.io/projected/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-kube-api-access-ssb4v\") pod \"fe01f458-6837-424c-a9cb-4a6a2a3ee5f0\" (UID: \"fe01f458-6837-424c-a9cb-4a6a2a3ee5f0\") " Oct 30 17:11:08.920353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:08.920335 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-maas-api-openshift-ca\") pod \"fe01f458-6837-424c-a9cb-4a6a2a3ee5f0\" (UID: \"fe01f458-6837-424c-a9cb-4a6a2a3ee5f0\") " Oct 30 17:11:08.920856 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:08.920829 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-maas-api-openshift-ca" pod "fe01f458-6837-424c-a9cb-4a6a2a3ee5f0" (UID: "fe01f458-6837-424c-a9cb-4a6a2a3ee5f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:08.921074 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:08.921017 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:08.922748 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:08.922722 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-kube-api-access-ssb4v" pod "fe01f458-6837-424c-a9cb-4a6a2a3ee5f0" (UID: "fe01f458-6837-424c-a9cb-4a6a2a3ee5f0"). InnerVolumeSpecName "kube-api-access-ssb4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:08.923283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:08.923246 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-maas-api-xks-ca" pod "fe01f458-6837-424c-a9cb-4a6a2a3ee5f0" (UID: "fe01f458-6837-424c-a9cb-4a6a2a3ee5f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:11:09.022620 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:09.022546 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:09.022620 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:09.022576 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssb4v\" (UniqueName: \"kubernetes.io/projected/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0-kube-api-access-ssb4v\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:09.632686 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:09.632648 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd" Oct 30 17:11:09.633306 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:09.632684 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd" event={"ID":"fe01f458-6837-424c-a9cb-4a6a2a3ee5f0","Type":"ContainerDied","Data":"5b395b9339270ee9f66fa080f4f1a21fc8f727687691af823793d4c9df13be3d"} Oct 30 17:11:09.633306 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:09.632756 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b395b9339270ee9f66fa080f4f1a21fc8f727687691af823793d4c9df13be3d" Oct 30 17:11:10.432317 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.432267 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:10.655837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.654001 6202 generic.go:356] "Generic (PLEG): container finished" podID="cd39addc-aef3-41d3-aae4-6358ffbdf53e" containerID="4530bfe5562f98ab30ee7a33dfef936b647d877c98f64832a0b845e0e8b8a007" exitCode=0 Oct 30 17:11:10.655837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.654240 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" event={"ID":"cd39addc-aef3-41d3-aae4-6358ffbdf53e","Type":"ContainerDied","Data":"4530bfe5562f98ab30ee7a33dfef936b647d877c98f64832a0b845e0e8b8a007"} Oct 30 17:11:10.655837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.654299 6202 scope.go:122] "RemoveContainer" containerID="2c0a61cf7c8464598a3b322eade405e2b9d16d04dcef15d59d6977914c279a8b" Oct 30 17:11:10.720002 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.719976 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:11:10.852222 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.852185 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-token\" (UniqueName: \"kubernetes.io/projected/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-token\") pod \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " Oct 30 17:11:10.852444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.852284 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istiod-ca-cert\") pod \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " Oct 30 17:11:10.852444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.852311 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-envoy\") pod \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " Oct 30 17:11:10.852444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.852381 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd39addc-aef3-41d3-aae4-6358ffbdf53e-kube-api-access-567xt\" (UniqueName: \"kubernetes.io/projected/cd39addc-aef3-41d3-aae4-6358ffbdf53e-kube-api-access-567xt\") pod \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " Oct 30 17:11:10.852444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.852409 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-workload-socket\") pod \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " Oct 30 17:11:10.854326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.852556 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-podinfo\") pod \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " Oct 30 17:11:10.854326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.852660 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-credential-socket\") pod \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " Oct 30 17:11:10.854326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.852710 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-workload-certs\") pod \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " Oct 30 17:11:10.854326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.852755 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-data\") pod \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\" (UID: \"cd39addc-aef3-41d3-aae4-6358ffbdf53e\") " Oct 30 17:11:10.854326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.852862 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istiod-ca-cert" pod "cd39addc-aef3-41d3-aae4-6358ffbdf53e" (UID: "cd39addc-aef3-41d3-aae4-6358ffbdf53e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:10.854326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.853087 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-workload-certs" pod "cd39addc-aef3-41d3-aae4-6358ffbdf53e" (UID: "cd39addc-aef3-41d3-aae4-6358ffbdf53e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:10.854326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.853191 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-credential-socket" pod "cd39addc-aef3-41d3-aae4-6358ffbdf53e" (UID: "cd39addc-aef3-41d3-aae4-6358ffbdf53e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:10.854326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.853376 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-workload-socket" pod "cd39addc-aef3-41d3-aae4-6358ffbdf53e" (UID: "cd39addc-aef3-41d3-aae4-6358ffbdf53e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:10.854326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.853930 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:10.854326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.853952 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:10.854326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.853966 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:10.854326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.853980 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:10.854326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.854041 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-data" pod "cd39addc-aef3-41d3-aae4-6358ffbdf53e" (UID: "cd39addc-aef3-41d3-aae4-6358ffbdf53e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:10.855161 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.854856 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-token" pod "cd39addc-aef3-41d3-aae4-6358ffbdf53e" (UID: "cd39addc-aef3-41d3-aae4-6358ffbdf53e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:10.855313 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.855278 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-envoy" pod "cd39addc-aef3-41d3-aae4-6358ffbdf53e" (UID: "cd39addc-aef3-41d3-aae4-6358ffbdf53e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:10.855438 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.855420 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-podinfo" pod "cd39addc-aef3-41d3-aae4-6358ffbdf53e" (UID: "cd39addc-aef3-41d3-aae4-6358ffbdf53e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:10.855732 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.855701 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd39addc-aef3-41d3-aae4-6358ffbdf53e-kube-api-access-567xt" pod "cd39addc-aef3-41d3-aae4-6358ffbdf53e" (UID: "cd39addc-aef3-41d3-aae4-6358ffbdf53e"). InnerVolumeSpecName "kube-api-access-567xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:10.955059 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.954982 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:10.955059 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.955008 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:10.955059 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.955018 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-567xt\" (UniqueName: \"kubernetes.io/projected/cd39addc-aef3-41d3-aae4-6358ffbdf53e-kube-api-access-567xt\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:10.955059 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.955028 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:10.955059 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:10.955037 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd39addc-aef3-41d3-aae4-6358ffbdf53e-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:11.665562 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:11.665530 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" Oct 30 17:11:11.666087 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:11.665563 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t" event={"ID":"cd39addc-aef3-41d3-aae4-6358ffbdf53e","Type":"ContainerDied","Data":"568c15a2165eab1089093886731459dff70484c1370a4c36312eb26d8a03e946"} Oct 30 17:11:11.666087 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:11.665619 6202 scope.go:122] "RemoveContainer" containerID="4530bfe5562f98ab30ee7a33dfef936b647d877c98f64832a0b845e0e8b8a007" Oct 30 17:11:11.683997 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:11.683973 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:11.739809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:11.739781 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t"] Oct 30 17:11:11.753689 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:11.753665 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b16279-openshift-default-cbc4db57c-j5d4t"] Oct 30 17:11:11.956846 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:11.955522 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:12.209853 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.208597 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f"] Oct 30 17:11:12.220789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.212332 6202 cpu_manager.go:414] "containerMap: removing container" podUID="cd39addc-aef3-41d3-aae4-6358ffbdf53e" containerName="istio-proxy" Oct 30 17:11:12.220789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.212362 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd39addc-aef3-41d3-aae4-6358ffbdf53e" containerName="istio-proxy" Oct 30 17:11:12.220789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.212401 6202 cpu_manager.go:414] "containerMap: removing container" podUID="fe01f458-6837-424c-a9cb-4a6a2a3ee5f0" containerName="revoke-keys" Oct 30 17:11:12.220789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.212409 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe01f458-6837-424c-a9cb-4a6a2a3ee5f0" containerName="revoke-keys" Oct 30 17:11:12.220789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.212647 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd39addc-aef3-41d3-aae4-6358ffbdf53e" containerName="istio-proxy" Oct 30 17:11:12.220789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.212660 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe01f458-6837-424c-a9cb-4a6a2a3ee5f0" containerName="revoke-keys" Oct 30 17:11:12.243713 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.243677 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f"] Oct 30 17:11:12.244146 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.244070 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f" Oct 30 17:11:12.266089 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.266055 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-97a47d-dockercfg-5g46f\"" Oct 30 17:11:12.266471 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.266454 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-97a47d\"" Oct 30 17:11:12.378187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.377037 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a4abfde-fb32-4553-930a-d415efa3b941-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f\" (UID: \"1a4abfde-fb32-4553-930a-d415efa3b941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f" Oct 30 17:11:12.378187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.377123 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddn4z\" (UniqueName: \"kubernetes.io/projected/1a4abfde-fb32-4553-930a-d415efa3b941-kube-api-access-ddn4z\") pod \"payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f\" (UID: \"1a4abfde-fb32-4553-930a-d415efa3b941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f" Oct 30 17:11:12.378187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.377492 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx"] Oct 30 17:11:12.381815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.378817 6202 cpu_manager.go:414] "containerMap: removing container" podUID="cd39addc-aef3-41d3-aae4-6358ffbdf53e" containerName="istio-proxy" Oct 30 17:11:12.381815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.378840 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd39addc-aef3-41d3-aae4-6358ffbdf53e" containerName="istio-proxy" Oct 30 17:11:12.381815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.379061 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd39addc-aef3-41d3-aae4-6358ffbdf53e" containerName="istio-proxy" Oct 30 17:11:12.392657 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.392506 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx" Oct 30 17:11:12.407794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.397684 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx"] Oct 30 17:11:12.420200 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.420172 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd39addc-aef3-41d3-aae4-6358ffbdf53e" path="/var/lib/kubelet/pods/cd39addc-aef3-41d3-aae4-6358ffbdf53e/volumes" Oct 30 17:11:12.478901 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.478690 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddn4z\" (UniqueName: \"kubernetes.io/projected/1a4abfde-fb32-4553-930a-d415efa3b941-kube-api-access-ddn4z\") pod \"payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f\" (UID: \"1a4abfde-fb32-4553-930a-d415efa3b941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f" Oct 30 17:11:12.479105 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.479070 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31841021-3c67-427d-9b3b-8ebfdcd657b6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx\" (UID: \"31841021-3c67-427d-9b3b-8ebfdcd657b6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx" Oct 30 17:11:12.479538 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.479513 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcgj9\" (UniqueName: \"kubernetes.io/projected/31841021-3c67-427d-9b3b-8ebfdcd657b6-kube-api-access-jcgj9\") pod \"payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx\" (UID: \"31841021-3c67-427d-9b3b-8ebfdcd657b6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx" Oct 30 17:11:12.479660 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.479625 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a4abfde-fb32-4553-930a-d415efa3b941-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f\" (UID: \"1a4abfde-fb32-4553-930a-d415efa3b941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f" Oct 30 17:11:12.480444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.480419 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a4abfde-fb32-4553-930a-d415efa3b941-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f\" (UID: \"1a4abfde-fb32-4553-930a-d415efa3b941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f" Oct 30 17:11:12.503559 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.503539 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddn4z\" (UniqueName: \"kubernetes.io/projected/1a4abfde-fb32-4553-930a-d415efa3b941-kube-api-access-ddn4z\") pod \"payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f\" (UID: \"1a4abfde-fb32-4553-930a-d415efa3b941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f" Oct 30 17:11:12.580997 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.580964 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcgj9\" (UniqueName: \"kubernetes.io/projected/31841021-3c67-427d-9b3b-8ebfdcd657b6-kube-api-access-jcgj9\") pod \"payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx\" (UID: \"31841021-3c67-427d-9b3b-8ebfdcd657b6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx" Oct 30 17:11:12.581232 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.581213 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31841021-3c67-427d-9b3b-8ebfdcd657b6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx\" (UID: \"31841021-3c67-427d-9b3b-8ebfdcd657b6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx" Oct 30 17:11:12.581910 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.581891 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31841021-3c67-427d-9b3b-8ebfdcd657b6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx\" (UID: \"31841021-3c67-427d-9b3b-8ebfdcd657b6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx" Oct 30 17:11:12.605430 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.605408 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcgj9\" (UniqueName: \"kubernetes.io/projected/31841021-3c67-427d-9b3b-8ebfdcd657b6-kube-api-access-jcgj9\") pod \"payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx\" (UID: \"31841021-3c67-427d-9b3b-8ebfdcd657b6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx" Oct 30 17:11:12.711638 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.711600 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f" Oct 30 17:11:12.722074 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.722049 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx" Oct 30 17:11:12.814808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:12.813491 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:13.036814 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:11:13.036755 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4abfde_fb32_4553_930a_d415efa3b941.slice/crio-6ce02159734a0f53e6f2ff6ad9767c054470559e9d68f6e70dc15f8153fd70a7 WatchSource:0}: Error finding container 6ce02159734a0f53e6f2ff6ad9767c054470559e9d68f6e70dc15f8153fd70a7: Status 404 returned error can't find the container with id 6ce02159734a0f53e6f2ff6ad9767c054470559e9d68f6e70dc15f8153fd70a7 Oct 30 17:11:13.039019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:13.038999 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f"] Oct 30 17:11:13.046889 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:13.046863 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx"] Oct 30 17:11:13.692136 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:13.692098 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f" event={"ID":"1a4abfde-fb32-4553-930a-d415efa3b941","Type":"ContainerStarted","Data":"56764dc14cd9ff11ba759e59bbed2f23b3fd4e162745c7684719dbf877a3485c"} Oct 30 17:11:13.692355 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:13.692165 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f" event={"ID":"1a4abfde-fb32-4553-930a-d415efa3b941","Type":"ContainerStarted","Data":"6ce02159734a0f53e6f2ff6ad9767c054470559e9d68f6e70dc15f8153fd70a7"} Oct 30 17:11:13.692355 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:13.692195 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f" Oct 30 17:11:13.706233 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:13.706184 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx" event={"ID":"31841021-3c67-427d-9b3b-8ebfdcd657b6","Type":"ContainerStarted","Data":"b1b16e1dbbe8f682366cb0f32f7cf72be41e0dc29be8ccf30cc4913b3644bf72"} Oct 30 17:11:13.706411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:13.706291 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx" event={"ID":"31841021-3c67-427d-9b3b-8ebfdcd657b6","Type":"ContainerStarted","Data":"30dba14689feece46ebddb3fc495e9e3eb5a29b7d84ae2e0720dfa92fecbf4e9"} Oct 30 17:11:13.706596 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:13.706559 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx" Oct 30 17:11:13.734650 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:13.734592 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f" podStartSLOduration=1.734580079 podStartE2EDuration="1.734580079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:13.734252181 +0000 UTC m=+2987.444706929" watchObservedRunningTime="2026-10-30 17:11:13.734580079 +0000 UTC m=+2987.445034822" Oct 30 17:11:13.781063 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:13.780996 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx" podStartSLOduration=1.7809787080000001 podStartE2EDuration="1.780978708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:13.776963223 +0000 UTC m=+2987.487417970" watchObservedRunningTime="2026-10-30 17:11:13.780978708 +0000 UTC m=+2987.491433453" Oct 30 17:11:14.086865 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:11:14.086821 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:11:14.473738 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:14.473636 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:16.310751 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:16.310715 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:18.568917 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:18.568880 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:20.235308 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:20.235265 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:22.223444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:22.223399 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:23.796346 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:23.796302 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:23.910779 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:23.910728 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4"] Oct 30 17:11:23.911040 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:23.910995 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4" podUID="d831ff6b-0124-40b7-aedb-70315823c044" containerName="payload-processing" containerID="cri-o://16332b758917699ce19cc949aca7689cadb5a4f29aa6bd1a18f5f51ea1f0622b" gracePeriod=30 Oct 30 17:11:23.934090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:23.934060 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl"] Oct 30 17:11:23.934341 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:23.934313 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl" podUID="f7bcdc3b-666e-4648-ace6-178f2c4718e5" containerName="payload-pre-processing" containerID="cri-o://6ec249052e351e706bd0cff9edc063fe36c2c2f7d4e58e8fc729d850dc7893a0" gracePeriod=30 Oct 30 17:11:24.007818 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.005999 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:24.127260 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.127224 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:24.456033 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.450607 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:24.493792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.488266 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4" Oct 30 17:11:24.528792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.523977 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl" Oct 30 17:11:24.668792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.662369 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7bcdc3b-666e-4648-ace6-178f2c4718e5-kube-api-access-7bwr4\" (UniqueName: \"kubernetes.io/projected/f7bcdc3b-666e-4648-ace6-178f2c4718e5-kube-api-access-7bwr4\") pod \"f7bcdc3b-666e-4648-ace6-178f2c4718e5\" (UID: \"f7bcdc3b-666e-4648-ace6-178f2c4718e5\") " Oct 30 17:11:24.668792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.662525 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d831ff6b-0124-40b7-aedb-70315823c044-kube-api-access-ggmrl\" (UniqueName: \"kubernetes.io/projected/d831ff6b-0124-40b7-aedb-70315823c044-kube-api-access-ggmrl\") pod \"d831ff6b-0124-40b7-aedb-70315823c044\" (UID: \"d831ff6b-0124-40b7-aedb-70315823c044\") " Oct 30 17:11:24.668792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.662709 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d831ff6b-0124-40b7-aedb-70315823c044-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d831ff6b-0124-40b7-aedb-70315823c044-plugins-config-volume\") pod \"d831ff6b-0124-40b7-aedb-70315823c044\" (UID: \"d831ff6b-0124-40b7-aedb-70315823c044\") " Oct 30 17:11:24.682793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.682144 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d831ff6b-0124-40b7-aedb-70315823c044-plugins-config-volume" pod "d831ff6b-0124-40b7-aedb-70315823c044" (UID: "d831ff6b-0124-40b7-aedb-70315823c044"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:24.700791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.699136 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d831ff6b-0124-40b7-aedb-70315823c044-kube-api-access-ggmrl" pod "d831ff6b-0124-40b7-aedb-70315823c044" (UID: "d831ff6b-0124-40b7-aedb-70315823c044"). InnerVolumeSpecName "kube-api-access-ggmrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:24.730790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.730185 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f" Oct 30 17:11:24.730790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.730557 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx" Oct 30 17:11:24.771336 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.771295 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7bcdc3b-666e-4648-ace6-178f2c4718e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bcdc3b-666e-4648-ace6-178f2c4718e5-plugins-config-volume\") pod \"f7bcdc3b-666e-4648-ace6-178f2c4718e5\" (UID: \"f7bcdc3b-666e-4648-ace6-178f2c4718e5\") " Oct 30 17:11:24.772941 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.772918 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d831ff6b-0124-40b7-aedb-70315823c044-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:24.773119 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.773105 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggmrl\" (UniqueName: \"kubernetes.io/projected/d831ff6b-0124-40b7-aedb-70315823c044-kube-api-access-ggmrl\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:24.773642 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.773620 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7bcdc3b-666e-4648-ace6-178f2c4718e5-plugins-config-volume" pod "f7bcdc3b-666e-4648-ace6-178f2c4718e5" (UID: "f7bcdc3b-666e-4648-ace6-178f2c4718e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:24.790314 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.790279 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bcdc3b-666e-4648-ace6-178f2c4718e5-kube-api-access-7bwr4" pod "f7bcdc3b-666e-4648-ace6-178f2c4718e5" (UID: "f7bcdc3b-666e-4648-ace6-178f2c4718e5"). InnerVolumeSpecName "kube-api-access-7bwr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:24.814224 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.814176 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:24.821962 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.821928 6202 generic.go:356] "Generic (PLEG): container finished" podID="f7bcdc3b-666e-4648-ace6-178f2c4718e5" containerID="6ec249052e351e706bd0cff9edc063fe36c2c2f7d4e58e8fc729d850dc7893a0" exitCode=0 Oct 30 17:11:24.822525 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.821968 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl" event={"ID":"f7bcdc3b-666e-4648-ace6-178f2c4718e5","Type":"ContainerDied","Data":"6ec249052e351e706bd0cff9edc063fe36c2c2f7d4e58e8fc729d850dc7893a0"} Oct 30 17:11:24.822639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.822572 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl" event={"ID":"f7bcdc3b-666e-4648-ace6-178f2c4718e5","Type":"ContainerDied","Data":"e8eed958e6dc9c35b1e8c5936f65ad0371546dd7456acf817bf5050f4acac4e5"} Oct 30 17:11:24.822639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.822611 6202 scope.go:122] "RemoveContainer" containerID="6ec249052e351e706bd0cff9edc063fe36c2c2f7d4e58e8fc729d850dc7893a0" Oct 30 17:11:24.822731 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.822010 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl" Oct 30 17:11:24.825443 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.825381 6202 generic.go:356] "Generic (PLEG): container finished" podID="d831ff6b-0124-40b7-aedb-70315823c044" containerID="16332b758917699ce19cc949aca7689cadb5a4f29aa6bd1a18f5f51ea1f0622b" exitCode=0 Oct 30 17:11:24.825934 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.825906 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4" event={"ID":"d831ff6b-0124-40b7-aedb-70315823c044","Type":"ContainerDied","Data":"16332b758917699ce19cc949aca7689cadb5a4f29aa6bd1a18f5f51ea1f0622b"} Oct 30 17:11:24.826049 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.825980 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4" event={"ID":"d831ff6b-0124-40b7-aedb-70315823c044","Type":"ContainerDied","Data":"f9cdc451d3f70d10ed57f8c1ce426c7aed751e8d792bb69d5007c91b708401f5"} Oct 30 17:11:24.826123 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.826078 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4" Oct 30 17:11:24.876427 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.876384 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bcdc3b-666e-4648-ace6-178f2c4718e5-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:24.876427 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.876429 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bwr4\" (UniqueName: \"kubernetes.io/projected/f7bcdc3b-666e-4648-ace6-178f2c4718e5-kube-api-access-7bwr4\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:24.877450 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.877431 6202 scope.go:122] "RemoveContainer" containerID="6ec249052e351e706bd0cff9edc063fe36c2c2f7d4e58e8fc729d850dc7893a0" Oct 30 17:11:24.878206 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:11:24.878174 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec249052e351e706bd0cff9edc063fe36c2c2f7d4e58e8fc729d850dc7893a0\": container with ID starting with 6ec249052e351e706bd0cff9edc063fe36c2c2f7d4e58e8fc729d850dc7893a0 not found: ID does not exist" containerID="6ec249052e351e706bd0cff9edc063fe36c2c2f7d4e58e8fc729d850dc7893a0" Oct 30 17:11:24.878329 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.878217 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec249052e351e706bd0cff9edc063fe36c2c2f7d4e58e8fc729d850dc7893a0"} err="failed to get container status \"6ec249052e351e706bd0cff9edc063fe36c2c2f7d4e58e8fc729d850dc7893a0\": rpc error: code = NotFound desc = could not find container \"6ec249052e351e706bd0cff9edc063fe36c2c2f7d4e58e8fc729d850dc7893a0\": container with ID starting with 6ec249052e351e706bd0cff9edc063fe36c2c2f7d4e58e8fc729d850dc7893a0 not found: ID does not exist" Oct 30 17:11:24.878329 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.878245 6202 scope.go:122] "RemoveContainer" containerID="16332b758917699ce19cc949aca7689cadb5a4f29aa6bd1a18f5f51ea1f0622b" Oct 30 17:11:24.903552 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.903521 6202 scope.go:122] "RemoveContainer" containerID="16332b758917699ce19cc949aca7689cadb5a4f29aa6bd1a18f5f51ea1f0622b" Oct 30 17:11:24.904013 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:11:24.903984 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16332b758917699ce19cc949aca7689cadb5a4f29aa6bd1a18f5f51ea1f0622b\": container with ID starting with 16332b758917699ce19cc949aca7689cadb5a4f29aa6bd1a18f5f51ea1f0622b not found: ID does not exist" containerID="16332b758917699ce19cc949aca7689cadb5a4f29aa6bd1a18f5f51ea1f0622b" Oct 30 17:11:24.904076 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.904025 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16332b758917699ce19cc949aca7689cadb5a4f29aa6bd1a18f5f51ea1f0622b"} err="failed to get container status \"16332b758917699ce19cc949aca7689cadb5a4f29aa6bd1a18f5f51ea1f0622b\": rpc error: code = NotFound desc = could not find container \"16332b758917699ce19cc949aca7689cadb5a4f29aa6bd1a18f5f51ea1f0622b\": container with ID starting with 16332b758917699ce19cc949aca7689cadb5a4f29aa6bd1a18f5f51ea1f0622b not found: ID does not exist" Oct 30 17:11:24.905896 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.905776 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd"] Oct 30 17:11:24.911868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.911842 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e1e830-tlbbd"] Oct 30 17:11:24.915857 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.915830 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl"] Oct 30 17:11:24.919726 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.919670 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e1e830-68cfc69b87-p7tpl"] Oct 30 17:11:24.928999 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.928970 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:24.955863 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:24.953664 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4"] Oct 30 17:11:25.007799 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:25.005735 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e1e830-644c9cd5dd-52rt4"] Oct 30 17:11:25.355690 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:25.355653 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk"] Oct 30 17:11:25.381959 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:25.381906 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl"] Oct 30 17:11:25.405933 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:25.405879 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:25.522607 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:25.522567 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:25.632869 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:25.632779 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:25.762443 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:25.762385 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:26.416520 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:26.416456 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d831ff6b-0124-40b7-aedb-70315823c044" path="/var/lib/kubelet/pods/d831ff6b-0124-40b7-aedb-70315823c044/volumes" Oct 30 17:11:26.417293 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:26.417273 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7bcdc3b-666e-4648-ace6-178f2c4718e5" path="/var/lib/kubelet/pods/f7bcdc3b-666e-4648-ace6-178f2c4718e5/volumes" Oct 30 17:11:26.417936 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:26.417919 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe01f458-6837-424c-a9cb-4a6a2a3ee5f0" path="/var/lib/kubelet/pods/fe01f458-6837-424c-a9cb-4a6a2a3ee5f0/volumes" Oct 30 17:11:27.174574 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:27.174533 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:29.079084 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:29.079050 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:30.356229 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.356173 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk" podUID="0d32e930-fdfe-4971-a4bb-b6eea8491790" containerName="payload-processing" containerID="cri-o://946e6fb9ed3c6526f62b6fe516cf70ab17319a32dd95f2f626d23118e56a4731" gracePeriod=25 Oct 30 17:11:30.382751 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.382699 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl" podUID="4c775d0b-bde0-4f40-b429-536c484aeba9" containerName="payload-pre-processing" containerID="cri-o://5b69c26f9a25fca64f6246febeea52459a689e2462da645598e6b2541e2d344a" gracePeriod=25 Oct 30 17:11:30.698912 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.698876 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk" Oct 30 17:11:30.735604 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.735579 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl" Oct 30 17:11:30.801122 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.801087 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d32e930-fdfe-4971-a4bb-b6eea8491790-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d32e930-fdfe-4971-a4bb-b6eea8491790-plugins-config-volume\") pod \"0d32e930-fdfe-4971-a4bb-b6eea8491790\" (UID: \"0d32e930-fdfe-4971-a4bb-b6eea8491790\") " Oct 30 17:11:30.801317 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.801188 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d32e930-fdfe-4971-a4bb-b6eea8491790-kube-api-access-hrc6b\" (UniqueName: \"kubernetes.io/projected/0d32e930-fdfe-4971-a4bb-b6eea8491790-kube-api-access-hrc6b\") pod \"0d32e930-fdfe-4971-a4bb-b6eea8491790\" (UID: \"0d32e930-fdfe-4971-a4bb-b6eea8491790\") " Oct 30 17:11:30.801433 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.801415 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c775d0b-bde0-4f40-b429-536c484aeba9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c775d0b-bde0-4f40-b429-536c484aeba9-plugins-config-volume\") pod \"4c775d0b-bde0-4f40-b429-536c484aeba9\" (UID: \"4c775d0b-bde0-4f40-b429-536c484aeba9\") " Oct 30 17:11:30.801500 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.801465 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c775d0b-bde0-4f40-b429-536c484aeba9-kube-api-access-t4qqv\" (UniqueName: \"kubernetes.io/projected/4c775d0b-bde0-4f40-b429-536c484aeba9-kube-api-access-t4qqv\") pod \"4c775d0b-bde0-4f40-b429-536c484aeba9\" (UID: \"4c775d0b-bde0-4f40-b429-536c484aeba9\") " Oct 30 17:11:30.802268 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.802234 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c775d0b-bde0-4f40-b429-536c484aeba9-plugins-config-volume" pod "4c775d0b-bde0-4f40-b429-536c484aeba9" (UID: "4c775d0b-bde0-4f40-b429-536c484aeba9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:30.802268 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.802239 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d32e930-fdfe-4971-a4bb-b6eea8491790-plugins-config-volume" pod "0d32e930-fdfe-4971-a4bb-b6eea8491790" (UID: "0d32e930-fdfe-4971-a4bb-b6eea8491790"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:30.804212 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.804185 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d32e930-fdfe-4971-a4bb-b6eea8491790-kube-api-access-hrc6b" pod "0d32e930-fdfe-4971-a4bb-b6eea8491790" (UID: "0d32e930-fdfe-4971-a4bb-b6eea8491790"). InnerVolumeSpecName "kube-api-access-hrc6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:30.804549 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.804531 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c775d0b-bde0-4f40-b429-536c484aeba9-kube-api-access-t4qqv" pod "4c775d0b-bde0-4f40-b429-536c484aeba9" (UID: "4c775d0b-bde0-4f40-b429-536c484aeba9"). InnerVolumeSpecName "kube-api-access-t4qqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:30.889701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.889606 6202 generic.go:356] "Generic (PLEG): container finished" podID="0d32e930-fdfe-4971-a4bb-b6eea8491790" containerID="946e6fb9ed3c6526f62b6fe516cf70ab17319a32dd95f2f626d23118e56a4731" exitCode=0 Oct 30 17:11:30.889701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.889673 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk" Oct 30 17:11:30.889701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.889691 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk" event={"ID":"0d32e930-fdfe-4971-a4bb-b6eea8491790","Type":"ContainerDied","Data":"946e6fb9ed3c6526f62b6fe516cf70ab17319a32dd95f2f626d23118e56a4731"} Oct 30 17:11:30.890041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.889753 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk" event={"ID":"0d32e930-fdfe-4971-a4bb-b6eea8491790","Type":"ContainerDied","Data":"66dc2a2783f35187d5e0a5dc6eb3a4fc550dde94971476c6c1ae03784307e518"} Oct 30 17:11:30.890041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.889822 6202 scope.go:122] "RemoveContainer" containerID="946e6fb9ed3c6526f62b6fe516cf70ab17319a32dd95f2f626d23118e56a4731" Oct 30 17:11:30.891753 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.891733 6202 generic.go:356] "Generic (PLEG): container finished" podID="4c775d0b-bde0-4f40-b429-536c484aeba9" containerID="5b69c26f9a25fca64f6246febeea52459a689e2462da645598e6b2541e2d344a" exitCode=0 Oct 30 17:11:30.892006 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.891987 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl" event={"ID":"4c775d0b-bde0-4f40-b429-536c484aeba9","Type":"ContainerDied","Data":"5b69c26f9a25fca64f6246febeea52459a689e2462da645598e6b2541e2d344a"} Oct 30 17:11:30.892103 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.892025 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl" event={"ID":"4c775d0b-bde0-4f40-b429-536c484aeba9","Type":"ContainerDied","Data":"4505950cfd20e3553c011a7d81aa66bcdc29a4bb731587b3a96e6636a45a2bac"} Oct 30 17:11:30.892103 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.892082 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl" Oct 30 17:11:30.902522 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.902500 6202 scope.go:122] "RemoveContainer" containerID="946e6fb9ed3c6526f62b6fe516cf70ab17319a32dd95f2f626d23118e56a4731" Oct 30 17:11:30.902841 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:11:30.902817 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"946e6fb9ed3c6526f62b6fe516cf70ab17319a32dd95f2f626d23118e56a4731\": container with ID starting with 946e6fb9ed3c6526f62b6fe516cf70ab17319a32dd95f2f626d23118e56a4731 not found: ID does not exist" containerID="946e6fb9ed3c6526f62b6fe516cf70ab17319a32dd95f2f626d23118e56a4731" Oct 30 17:11:30.902934 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.902848 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"946e6fb9ed3c6526f62b6fe516cf70ab17319a32dd95f2f626d23118e56a4731"} err="failed to get container status \"946e6fb9ed3c6526f62b6fe516cf70ab17319a32dd95f2f626d23118e56a4731\": rpc error: code = NotFound desc = could not find container \"946e6fb9ed3c6526f62b6fe516cf70ab17319a32dd95f2f626d23118e56a4731\": container with ID starting with 946e6fb9ed3c6526f62b6fe516cf70ab17319a32dd95f2f626d23118e56a4731 not found: ID does not exist" Oct 30 17:11:30.902934 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.902866 6202 scope.go:122] "RemoveContainer" containerID="5b69c26f9a25fca64f6246febeea52459a689e2462da645598e6b2541e2d344a" Oct 30 17:11:30.903458 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.903434 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4qqv\" (UniqueName: \"kubernetes.io/projected/4c775d0b-bde0-4f40-b429-536c484aeba9-kube-api-access-t4qqv\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.903524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.903464 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d32e930-fdfe-4971-a4bb-b6eea8491790-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.903524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.903479 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrc6b\" (UniqueName: \"kubernetes.io/projected/0d32e930-fdfe-4971-a4bb-b6eea8491790-kube-api-access-hrc6b\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.903524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.903493 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c775d0b-bde0-4f40-b429-536c484aeba9-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.913465 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.913444 6202 scope.go:122] "RemoveContainer" containerID="5b69c26f9a25fca64f6246febeea52459a689e2462da645598e6b2541e2d344a" Oct 30 17:11:30.913732 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:11:30.913711 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b69c26f9a25fca64f6246febeea52459a689e2462da645598e6b2541e2d344a\": container with ID starting with 5b69c26f9a25fca64f6246febeea52459a689e2462da645598e6b2541e2d344a not found: ID does not exist" containerID="5b69c26f9a25fca64f6246febeea52459a689e2462da645598e6b2541e2d344a" Oct 30 17:11:30.913799 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.913739 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b69c26f9a25fca64f6246febeea52459a689e2462da645598e6b2541e2d344a"} err="failed to get container status \"5b69c26f9a25fca64f6246febeea52459a689e2462da645598e6b2541e2d344a\": rpc error: code = NotFound desc = could not find container \"5b69c26f9a25fca64f6246febeea52459a689e2462da645598e6b2541e2d344a\": container with ID starting with 5b69c26f9a25fca64f6246febeea52459a689e2462da645598e6b2541e2d344a not found: ID does not exist" Oct 30 17:11:30.931796 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.931753 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:30.947449 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.947412 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk"] Oct 30 17:11:30.951644 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:30.951614 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-698c71-967dfc7b6-tfmhk"] Oct 30 17:11:31.030225 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:31.030181 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl"] Oct 30 17:11:31.036302 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:31.036267 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-7b867575f9-s6dpl"] Oct 30 17:11:31.719753 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:31.719653 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:31.833920 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:31.833885 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:32.078813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:32.078759 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:32.422324 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:32.422251 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d32e930-fdfe-4971-a4bb-b6eea8491790" path="/var/lib/kubelet/pods/0d32e930-fdfe-4971-a4bb-b6eea8491790/volumes" Oct 30 17:11:32.422737 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:32.422723 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c775d0b-bde0-4f40-b429-536c484aeba9" path="/var/lib/kubelet/pods/4c775d0b-bde0-4f40-b429-536c484aeba9/volumes" Oct 30 17:11:32.811250 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:32.811158 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:33.811793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:33.811732 6202 scope.go:122] "RemoveContainer" containerID="923e6fdbcf9aa00d7b67a06dd8d2940d25936bcf9dc3e604da014314c925b7c7" Oct 30 17:11:33.822483 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:33.822463 6202 scope.go:122] "RemoveContainer" containerID="52737995839a0d2e3a8a8e67d7edff7fdabc314931d2d34f6daa2ffc907e1364" Oct 30 17:11:33.836047 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:33.836022 6202 scope.go:122] "RemoveContainer" containerID="d6e1aff3172193db7cdeb855f86cdfb66a10f205b34a967a0434f8b475cc823f" Oct 30 17:11:33.849161 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:33.849137 6202 scope.go:122] "RemoveContainer" containerID="f49f4ce7186d5e83ea906a1da32107dbe1d9757ac9dae7a812d07078ecbbf7b7" Oct 30 17:11:33.861272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:33.861250 6202 scope.go:122] "RemoveContainer" containerID="e6df92ad4e31a156c30f6137263ef65f0d0093cc616ee831f61371c177ca4afb" Oct 30 17:11:34.695037 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:34.694997 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:36.934332 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:36.934299 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:37.046030 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:37.045992 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:37.168790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:37.165346 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:38.842619 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:38.842582 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:41.362415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:41.362352 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:41.473723 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:41.473674 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:42.313691 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.313653 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:42.562353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.562317 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x"] Oct 30 17:11:42.563184 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.563166 6202 cpu_manager.go:414] "containerMap: removing container" podUID="d831ff6b-0124-40b7-aedb-70315823c044" containerName="payload-processing" Oct 30 17:11:42.563239 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.563186 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d831ff6b-0124-40b7-aedb-70315823c044" containerName="payload-processing" Oct 30 17:11:42.563239 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.563211 6202 cpu_manager.go:414] "containerMap: removing container" podUID="f7bcdc3b-666e-4648-ace6-178f2c4718e5" containerName="payload-pre-processing" Oct 30 17:11:42.563239 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.563216 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7bcdc3b-666e-4648-ace6-178f2c4718e5" containerName="payload-pre-processing" Oct 30 17:11:42.563350 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.563261 6202 cpu_manager.go:414] "containerMap: removing container" podUID="0d32e930-fdfe-4971-a4bb-b6eea8491790" containerName="payload-processing" Oct 30 17:11:42.563350 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.563266 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d32e930-fdfe-4971-a4bb-b6eea8491790" containerName="payload-processing" Oct 30 17:11:42.563350 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.563277 6202 cpu_manager.go:414] "containerMap: removing container" podUID="4c775d0b-bde0-4f40-b429-536c484aeba9" containerName="payload-pre-processing" Oct 30 17:11:42.563350 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.563283 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c775d0b-bde0-4f40-b429-536c484aeba9" containerName="payload-pre-processing" Oct 30 17:11:42.563499 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.563402 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d32e930-fdfe-4971-a4bb-b6eea8491790" containerName="payload-processing" Oct 30 17:11:42.563499 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.563411 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c775d0b-bde0-4f40-b429-536c484aeba9" containerName="payload-pre-processing" Oct 30 17:11:42.563499 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.563417 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="d831ff6b-0124-40b7-aedb-70315823c044" containerName="payload-processing" Oct 30 17:11:42.563499 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.563428 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7bcdc3b-666e-4648-ace6-178f2c4718e5" containerName="payload-pre-processing" Oct 30 17:11:42.569927 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.569868 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x" Oct 30 17:11:42.586999 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.586973 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x"] Oct 30 17:11:42.594973 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.594946 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:11:42.595115 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.594954 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:11:42.598946 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.598921 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:11:42.599068 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.599016 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:11:42.629345 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.629311 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzwjg\" (UniqueName: \"kubernetes.io/projected/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-kube-api-access-qzwjg\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x\" (UID: \"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x" Oct 30 17:11:42.629482 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.629451 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x\" (UID: \"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x" Oct 30 17:11:42.629588 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.629568 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x\" (UID: \"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x" Oct 30 17:11:42.731401 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.731367 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwjg\" (UniqueName: \"kubernetes.io/projected/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-kube-api-access-qzwjg\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x\" (UID: \"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x" Oct 30 17:11:42.731632 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.731468 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x\" (UID: \"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x" Oct 30 17:11:42.731632 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.731549 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x\" (UID: \"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x" Oct 30 17:11:42.732526 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.732492 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x\" (UID: \"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x" Oct 30 17:11:42.734464 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.734442 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x\" (UID: \"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x" Oct 30 17:11:42.756296 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.756262 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwjg\" (UniqueName: \"kubernetes.io/projected/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-kube-api-access-qzwjg\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x\" (UID: \"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x" Oct 30 17:11:42.884380 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:42.884340 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x" Oct 30 17:11:43.193172 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:43.193135 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:43.193320 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:11:43.193149 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5dbd37_6315_4bb7_b3aa_929a68ba7ef8.slice/crio-ee2e0b3fb98ff6421e143f8a9d12c933ee9e0b7448470deada3a035219e4f5bb WatchSource:0}: Error finding container ee2e0b3fb98ff6421e143f8a9d12c933ee9e0b7448470deada3a035219e4f5bb: Status 404 returned error can't find the container with id ee2e0b3fb98ff6421e143f8a9d12c933ee9e0b7448470deada3a035219e4f5bb Oct 30 17:11:43.195558 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:43.195537 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x"] Oct 30 17:11:44.010256 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:44.010212 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x" event={"ID":"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8","Type":"ContainerStarted","Data":"2d4b5495fc88679cc7b6683ccb7a7aaecda9b31cfe91af22a84b6e0806173a9b"} Oct 30 17:11:44.010731 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:44.010302 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x" event={"ID":"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8","Type":"ContainerStarted","Data":"ee2e0b3fb98ff6421e143f8a9d12c933ee9e0b7448470deada3a035219e4f5bb"} Oct 30 17:11:44.056937 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:44.056884 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x" podStartSLOduration=2.056868404 podStartE2EDuration="2.056868404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:44.051828092 +0000 UTC m=+3017.762282838" watchObservedRunningTime="2026-10-30 17:11:44.056868404 +0000 UTC m=+3017.767323190" Oct 30 17:11:44.074460 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:11:44.074423 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:11:45.036284 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:45.036245 6202 generic.go:356] "Generic (PLEG): container finished" podID="7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8" containerID="2d4b5495fc88679cc7b6683ccb7a7aaecda9b31cfe91af22a84b6e0806173a9b" exitCode=0 Oct 30 17:11:45.036693 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:45.036384 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x" event={"ID":"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8","Type":"ContainerDied","Data":"2d4b5495fc88679cc7b6683ccb7a7aaecda9b31cfe91af22a84b6e0806173a9b"} Oct 30 17:11:45.230144 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:45.230105 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:46.232741 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:46.232715 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x" Oct 30 17:11:46.410958 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:46.410926 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-maas-api-xks-ca\") pod \"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8\" (UID: \"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8\") " Oct 30 17:11:46.410958 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:46.410963 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-maas-api-openshift-ca\") pod \"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8\" (UID: \"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8\") " Oct 30 17:11:46.411251 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:46.411055 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-kube-api-access-qzwjg\" (UniqueName: \"kubernetes.io/projected/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-kube-api-access-qzwjg\") pod \"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8\" (UID: \"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8\") " Oct 30 17:11:46.411673 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:46.411643 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-maas-api-openshift-ca" pod "7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8" (UID: "7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:46.414131 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:46.414102 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-maas-api-xks-ca" pod "7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8" (UID: "7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:11:46.415031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:46.415004 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-kube-api-access-qzwjg" pod "7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8" (UID: "7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8"). InnerVolumeSpecName "kube-api-access-qzwjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:46.474627 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:46.473199 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:46.513701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:46.513662 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:46.513701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:46.513686 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:46.513701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:46.513696 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzwjg\" (UniqueName: \"kubernetes.io/projected/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8-kube-api-access-qzwjg\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:46.613795 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:46.611655 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:46.733842 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:46.731396 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:47.138792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:47.136636 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x" event={"ID":"7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8","Type":"ContainerDied","Data":"ee2e0b3fb98ff6421e143f8a9d12c933ee9e0b7448470deada3a035219e4f5bb"} Oct 30 17:11:47.138792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:47.136700 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee2e0b3fb98ff6421e143f8a9d12c933ee9e0b7448470deada3a035219e4f5bb" Oct 30 17:11:47.138792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:47.136810 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x" Oct 30 17:11:47.790729 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:47.790698 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:49.671156 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:49.671109 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:51.464412 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:51.464371 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:52.680420 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:52.680377 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:52.984644 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:52.984550 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp"] Oct 30 17:11:53.088999 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:53.088966 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:53.217792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:53.214847 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:54.266818 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:54.266777 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4"] Oct 30 17:11:54.267250 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:54.266982 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" podUID="a227113a-3241-42d2-98d8-4c67c1ca8f16" containerName="istio-proxy" containerID="cri-o://c41e712a6e5049a241eb41fe82f4c2e8804f6f65fa256b460a01568c9a7bdd88" gracePeriod=30 Oct 30 17:11:54.982969 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:54.982928 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:56.639465 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:56.639427 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:57.985660 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:57.985614 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp" podUID="854625e6-cc7a-4f5d-9475-ec31b0258eef" containerName="payload-pre-processing" containerID="cri-o://acb2bdc25d36f042b59a9a7b2113959824b2a66f50d848e2ddec6274ef5dd1cd" gracePeriod=25 Oct 30 17:11:58.411132 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:58.411084 6202 generic.go:356] "Generic (PLEG): container finished" podID="854625e6-cc7a-4f5d-9475-ec31b0258eef" containerID="acb2bdc25d36f042b59a9a7b2113959824b2a66f50d848e2ddec6274ef5dd1cd" exitCode=0 Oct 30 17:11:58.429590 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:58.429551 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp" event={"ID":"854625e6-cc7a-4f5d-9475-ec31b0258eef","Type":"ContainerDied","Data":"acb2bdc25d36f042b59a9a7b2113959824b2a66f50d848e2ddec6274ef5dd1cd"} Oct 30 17:11:58.636214 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:58.636139 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp" Oct 30 17:11:58.813110 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:58.813074 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:58.864858 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:58.864806 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/854625e6-cc7a-4f5d-9475-ec31b0258eef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854625e6-cc7a-4f5d-9475-ec31b0258eef-plugins-config-volume\") pod \"854625e6-cc7a-4f5d-9475-ec31b0258eef\" (UID: \"854625e6-cc7a-4f5d-9475-ec31b0258eef\") " Oct 30 17:11:58.865196 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:58.864940 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854625e6-cc7a-4f5d-9475-ec31b0258eef-kube-api-access-wtc45\" (UniqueName: \"kubernetes.io/projected/854625e6-cc7a-4f5d-9475-ec31b0258eef-kube-api-access-wtc45\") pod \"854625e6-cc7a-4f5d-9475-ec31b0258eef\" (UID: \"854625e6-cc7a-4f5d-9475-ec31b0258eef\") " Oct 30 17:11:58.865315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:58.865272 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/854625e6-cc7a-4f5d-9475-ec31b0258eef-plugins-config-volume" pod "854625e6-cc7a-4f5d-9475-ec31b0258eef" (UID: "854625e6-cc7a-4f5d-9475-ec31b0258eef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:58.865863 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:58.865732 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854625e6-cc7a-4f5d-9475-ec31b0258eef-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:58.869806 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:58.869757 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854625e6-cc7a-4f5d-9475-ec31b0258eef-kube-api-access-wtc45" pod "854625e6-cc7a-4f5d-9475-ec31b0258eef" (UID: "854625e6-cc7a-4f5d-9475-ec31b0258eef"). InnerVolumeSpecName "kube-api-access-wtc45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:58.939850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:58.937391 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:58.972805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:58.970287 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtc45\" (UniqueName: \"kubernetes.io/projected/854625e6-cc7a-4f5d-9475-ec31b0258eef-kube-api-access-wtc45\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:11:59.051533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:59.051215 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:59.420856 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:59.420828 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp" Oct 30 17:11:59.421057 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:59.420890 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp" event={"ID":"854625e6-cc7a-4f5d-9475-ec31b0258eef","Type":"ContainerDied","Data":"d365d139e5966c84a3803a2d5e6f6bae404cf7a3b62c0e93cf4ff1cef8bbcbf1"} Oct 30 17:11:59.421057 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:59.420953 6202 scope.go:122] "RemoveContainer" containerID="acb2bdc25d36f042b59a9a7b2113959824b2a66f50d848e2ddec6274ef5dd1cd" Oct 30 17:11:59.423149 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:59.423126 6202 generic.go:356] "Generic (PLEG): container finished" podID="a227113a-3241-42d2-98d8-4c67c1ca8f16" containerID="c41e712a6e5049a241eb41fe82f4c2e8804f6f65fa256b460a01568c9a7bdd88" exitCode=0 Oct 30 17:11:59.423272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:59.423188 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" event={"ID":"a227113a-3241-42d2-98d8-4c67c1ca8f16","Type":"ContainerDied","Data":"c41e712a6e5049a241eb41fe82f4c2e8804f6f65fa256b460a01568c9a7bdd88"} Oct 30 17:11:59.478567 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:59.478536 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp"] Oct 30 17:11:59.491282 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:11:59.491251 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-e43400-595b985c75-lcqwp"] Oct 30 17:12:00.014327 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.014301 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:12:00.112808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.107167 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-credential-socket\") pod \"a227113a-3241-42d2-98d8-4c67c1ca8f16\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " Oct 30 17:12:00.112808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.107251 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a227113a-3241-42d2-98d8-4c67c1ca8f16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a227113a-3241-42d2-98d8-4c67c1ca8f16-istiod-ca-cert\") pod \"a227113a-3241-42d2-98d8-4c67c1ca8f16\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " Oct 30 17:12:00.112808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.107305 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-data\") pod \"a227113a-3241-42d2-98d8-4c67c1ca8f16\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " Oct 30 17:12:00.112808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.107361 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a227113a-3241-42d2-98d8-4c67c1ca8f16-kube-api-access-xxpn2\" (UniqueName: \"kubernetes.io/projected/a227113a-3241-42d2-98d8-4c67c1ca8f16-kube-api-access-xxpn2\") pod \"a227113a-3241-42d2-98d8-4c67c1ca8f16\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " Oct 30 17:12:00.112808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.107423 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-token\" (UniqueName: \"kubernetes.io/projected/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-token\") pod \"a227113a-3241-42d2-98d8-4c67c1ca8f16\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " Oct 30 17:12:00.112808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.107450 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-workload-socket\") pod \"a227113a-3241-42d2-98d8-4c67c1ca8f16\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " Oct 30 17:12:00.112808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.107501 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-envoy\") pod \"a227113a-3241-42d2-98d8-4c67c1ca8f16\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " Oct 30 17:12:00.112808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.107648 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-workload-certs\") pod \"a227113a-3241-42d2-98d8-4c67c1ca8f16\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " Oct 30 17:12:00.112808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.107675 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-podinfo\") pod \"a227113a-3241-42d2-98d8-4c67c1ca8f16\" (UID: \"a227113a-3241-42d2-98d8-4c67c1ca8f16\") " Oct 30 17:12:00.131785 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.131020 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-workload-socket" pod "a227113a-3241-42d2-98d8-4c67c1ca8f16" (UID: "a227113a-3241-42d2-98d8-4c67c1ca8f16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:00.131785 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.131317 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a227113a-3241-42d2-98d8-4c67c1ca8f16-istiod-ca-cert" pod "a227113a-3241-42d2-98d8-4c67c1ca8f16" (UID: "a227113a-3241-42d2-98d8-4c67c1ca8f16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:00.131785 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.131554 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-credential-socket" pod "a227113a-3241-42d2-98d8-4c67c1ca8f16" (UID: "a227113a-3241-42d2-98d8-4c67c1ca8f16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:00.131785 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.131701 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-data" pod "a227113a-3241-42d2-98d8-4c67c1ca8f16" (UID: "a227113a-3241-42d2-98d8-4c67c1ca8f16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:00.138593 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.136850 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-workload-certs" pod "a227113a-3241-42d2-98d8-4c67c1ca8f16" (UID: "a227113a-3241-42d2-98d8-4c67c1ca8f16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:00.164494 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.164423 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a227113a-3241-42d2-98d8-4c67c1ca8f16-kube-api-access-xxpn2" pod "a227113a-3241-42d2-98d8-4c67c1ca8f16" (UID: "a227113a-3241-42d2-98d8-4c67c1ca8f16"). InnerVolumeSpecName "kube-api-access-xxpn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:00.180150 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.180108 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-token" pod "a227113a-3241-42d2-98d8-4c67c1ca8f16" (UID: "a227113a-3241-42d2-98d8-4c67c1ca8f16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:00.253790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.251960 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-envoy" pod "a227113a-3241-42d2-98d8-4c67c1ca8f16" (UID: "a227113a-3241-42d2-98d8-4c67c1ca8f16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:00.287124 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.287052 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:00.287124 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.287087 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a227113a-3241-42d2-98d8-4c67c1ca8f16-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:00.287487 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.287424 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:00.287487 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.287451 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxpn2\" (UniqueName: \"kubernetes.io/projected/a227113a-3241-42d2-98d8-4c67c1ca8f16-kube-api-access-xxpn2\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:00.293874 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.287466 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:00.300884 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.293848 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:00.300884 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.294059 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:00.300884 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.294078 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a227113a-3241-42d2-98d8-4c67c1ca8f16-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:00.300884 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.299565 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-podinfo" pod "a227113a-3241-42d2-98d8-4c67c1ca8f16" (UID: "a227113a-3241-42d2-98d8-4c67c1ca8f16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:12:00.395728 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.395641 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a227113a-3241-42d2-98d8-4c67c1ca8f16-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:00.412902 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.412871 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="854625e6-cc7a-4f5d-9475-ec31b0258eef" path="/var/lib/kubelet/pods/854625e6-cc7a-4f5d-9475-ec31b0258eef/volumes" Oct 30 17:12:00.461034 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.460631 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" event={"ID":"a227113a-3241-42d2-98d8-4c67c1ca8f16","Type":"ContainerDied","Data":"d1a84471a87f20e289911c858abfd0a9b10ea4b24472164ac4c5e494de638f73"} Oct 30 17:12:00.461034 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.460695 6202 scope.go:122] "RemoveContainer" containerID="c41e712a6e5049a241eb41fe82f4c2e8804f6f65fa256b460a01568c9a7bdd88" Oct 30 17:12:00.461034 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.460871 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4" Oct 30 17:12:00.534461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.534427 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4"] Oct 30 17:12:00.575871 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.574291 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:00.579580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:00.579555 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-2nmv4"] Oct 30 17:12:01.317329 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:01.317295 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:01.498960 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:01.498917 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:01.537232 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:01.537204 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x"] Oct 30 17:12:01.570137 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:01.570025 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e43400-jp89x"] Oct 30 17:12:01.616065 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:01.616027 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:02.069651 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:02.069601 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:02.412511 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:02.412475 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8" path="/var/lib/kubelet/pods/7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8/volumes" Oct 30 17:12:02.413000 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:02.412982 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a227113a-3241-42d2-98d8-4c67c1ca8f16" path="/var/lib/kubelet/pods/a227113a-3241-42d2-98d8-4c67c1ca8f16/volumes" Oct 30 17:12:02.577536 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:02.577503 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq"] Oct 30 17:12:02.577812 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:02.577781 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" podUID="2b6965bb-5f5c-40ca-b098-9f67c7433381" containerName="istio-proxy" containerID="cri-o://63374dbb7df5d4ccf540a17f8b99f53e02ca35ef4f36eb8ccd7e7e6e4fbc0acf" gracePeriod=30 Oct 30 17:12:03.587999 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:03.587966 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:05.318473 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:05.318382 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:07.527867 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:07.527829 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:07.985758 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:07.985722 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 17:12:08.107811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.107758 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6965bb-5f5c-40ca-b098-9f67c7433381-kube-api-access-f4hwp\" (UniqueName: \"kubernetes.io/projected/2b6965bb-5f5c-40ca-b098-9f67c7433381-kube-api-access-f4hwp\") pod \"2b6965bb-5f5c-40ca-b098-9f67c7433381\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " Oct 30 17:12:08.108008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.107948 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b6965bb-5f5c-40ca-b098-9f67c7433381-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6965bb-5f5c-40ca-b098-9f67c7433381-istiod-ca-cert\") pod \"2b6965bb-5f5c-40ca-b098-9f67c7433381\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " Oct 30 17:12:08.108008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.107983 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-podinfo\") pod \"2b6965bb-5f5c-40ca-b098-9f67c7433381\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " Oct 30 17:12:08.108126 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.108029 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-workload-socket\") pod \"2b6965bb-5f5c-40ca-b098-9f67c7433381\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " Oct 30 17:12:08.108406 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.108171 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-workload-certs\") pod \"2b6965bb-5f5c-40ca-b098-9f67c7433381\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " Oct 30 17:12:08.108406 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.108259 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-envoy\") pod \"2b6965bb-5f5c-40ca-b098-9f67c7433381\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " Oct 30 17:12:08.108406 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.108308 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-data\") pod \"2b6965bb-5f5c-40ca-b098-9f67c7433381\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " Oct 30 17:12:08.108406 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.108365 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-token\" (UniqueName: \"kubernetes.io/projected/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-token\") pod \"2b6965bb-5f5c-40ca-b098-9f67c7433381\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " Oct 30 17:12:08.108671 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.108489 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-credential-socket\") pod \"2b6965bb-5f5c-40ca-b098-9f67c7433381\" (UID: \"2b6965bb-5f5c-40ca-b098-9f67c7433381\") " Oct 30 17:12:08.108942 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.108732 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6965bb-5f5c-40ca-b098-9f67c7433381-istiod-ca-cert" pod "2b6965bb-5f5c-40ca-b098-9f67c7433381" (UID: "2b6965bb-5f5c-40ca-b098-9f67c7433381"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:08.108942 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.108792 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-workload-socket" pod "2b6965bb-5f5c-40ca-b098-9f67c7433381" (UID: "2b6965bb-5f5c-40ca-b098-9f67c7433381"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:08.109363 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.109186 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-data" pod "2b6965bb-5f5c-40ca-b098-9f67c7433381" (UID: "2b6965bb-5f5c-40ca-b098-9f67c7433381"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:08.109626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.109470 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-credential-socket" pod "2b6965bb-5f5c-40ca-b098-9f67c7433381" (UID: "2b6965bb-5f5c-40ca-b098-9f67c7433381"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:08.109626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.109482 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6965bb-5f5c-40ca-b098-9f67c7433381-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:08.109626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.109506 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:08.109626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.109520 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:08.109937 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.109909 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-workload-certs" pod "2b6965bb-5f5c-40ca-b098-9f67c7433381" (UID: "2b6965bb-5f5c-40ca-b098-9f67c7433381"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:08.112717 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.112683 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-envoy" pod "2b6965bb-5f5c-40ca-b098-9f67c7433381" (UID: "2b6965bb-5f5c-40ca-b098-9f67c7433381"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:08.112975 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.112937 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-podinfo" pod "2b6965bb-5f5c-40ca-b098-9f67c7433381" (UID: "2b6965bb-5f5c-40ca-b098-9f67c7433381"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:12:08.114641 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.114610 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6965bb-5f5c-40ca-b098-9f67c7433381-kube-api-access-f4hwp" pod "2b6965bb-5f5c-40ca-b098-9f67c7433381" (UID: "2b6965bb-5f5c-40ca-b098-9f67c7433381"). InnerVolumeSpecName "kube-api-access-f4hwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:08.114946 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.114920 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-token" pod "2b6965bb-5f5c-40ca-b098-9f67c7433381" (UID: "2b6965bb-5f5c-40ca-b098-9f67c7433381"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:08.211449 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.211410 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:08.211449 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.211448 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:08.211664 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.211462 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4hwp\" (UniqueName: \"kubernetes.io/projected/2b6965bb-5f5c-40ca-b098-9f67c7433381-kube-api-access-f4hwp\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:08.211664 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.211476 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:08.211664 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.211488 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:08.211664 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.211498 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6965bb-5f5c-40ca-b098-9f67c7433381-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:08.543733 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.543692 6202 generic.go:356] "Generic (PLEG): container finished" podID="2b6965bb-5f5c-40ca-b098-9f67c7433381" containerID="63374dbb7df5d4ccf540a17f8b99f53e02ca35ef4f36eb8ccd7e7e6e4fbc0acf" exitCode=0 Oct 30 17:12:08.544288 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.543754 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" Oct 30 17:12:08.544288 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.543803 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" event={"ID":"2b6965bb-5f5c-40ca-b098-9f67c7433381","Type":"ContainerDied","Data":"63374dbb7df5d4ccf540a17f8b99f53e02ca35ef4f36eb8ccd7e7e6e4fbc0acf"} Oct 30 17:12:08.544288 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.543875 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq" event={"ID":"2b6965bb-5f5c-40ca-b098-9f67c7433381","Type":"ContainerDied","Data":"2c2438e43479bc5a8507577f8c3be898f50868106fcc413ed905bd5f149d1d74"} Oct 30 17:12:08.544288 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.543913 6202 scope.go:122] "RemoveContainer" containerID="63374dbb7df5d4ccf540a17f8b99f53e02ca35ef4f36eb8ccd7e7e6e4fbc0acf" Oct 30 17:12:08.560026 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.559040 6202 scope.go:122] "RemoveContainer" containerID="b59da60ecc764d40930bc401e680c533f6c1a12b9b5f935c27159b26ad047367" Oct 30 17:12:08.560026 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.559563 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:08.578311 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.578282 6202 scope.go:122] "RemoveContainer" containerID="63374dbb7df5d4ccf540a17f8b99f53e02ca35ef4f36eb8ccd7e7e6e4fbc0acf" Oct 30 17:12:08.578697 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:12:08.578662 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63374dbb7df5d4ccf540a17f8b99f53e02ca35ef4f36eb8ccd7e7e6e4fbc0acf\": container with ID starting with 63374dbb7df5d4ccf540a17f8b99f53e02ca35ef4f36eb8ccd7e7e6e4fbc0acf not found: ID does not exist" containerID="63374dbb7df5d4ccf540a17f8b99f53e02ca35ef4f36eb8ccd7e7e6e4fbc0acf" Oct 30 17:12:08.578873 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.578710 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63374dbb7df5d4ccf540a17f8b99f53e02ca35ef4f36eb8ccd7e7e6e4fbc0acf"} err="failed to get container status \"63374dbb7df5d4ccf540a17f8b99f53e02ca35ef4f36eb8ccd7e7e6e4fbc0acf\": rpc error: code = NotFound desc = could not find container \"63374dbb7df5d4ccf540a17f8b99f53e02ca35ef4f36eb8ccd7e7e6e4fbc0acf\": container with ID starting with 63374dbb7df5d4ccf540a17f8b99f53e02ca35ef4f36eb8ccd7e7e6e4fbc0acf not found: ID does not exist" Oct 30 17:12:08.578873 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.578738 6202 scope.go:122] "RemoveContainer" containerID="b59da60ecc764d40930bc401e680c533f6c1a12b9b5f935c27159b26ad047367" Oct 30 17:12:08.579092 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:12:08.579069 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b59da60ecc764d40930bc401e680c533f6c1a12b9b5f935c27159b26ad047367\": container with ID starting with b59da60ecc764d40930bc401e680c533f6c1a12b9b5f935c27159b26ad047367 not found: ID does not exist" containerID="b59da60ecc764d40930bc401e680c533f6c1a12b9b5f935c27159b26ad047367" Oct 30 17:12:08.579156 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.579098 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b59da60ecc764d40930bc401e680c533f6c1a12b9b5f935c27159b26ad047367"} err="failed to get container status \"b59da60ecc764d40930bc401e680c533f6c1a12b9b5f935c27159b26ad047367\": rpc error: code = NotFound desc = could not find container \"b59da60ecc764d40930bc401e680c533f6c1a12b9b5f935c27159b26ad047367\": container with ID starting with b59da60ecc764d40930bc401e680c533f6c1a12b9b5f935c27159b26ad047367 not found: ID does not exist" Oct 30 17:12:08.644397 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.644354 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq"] Oct 30 17:12:08.656237 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.656209 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-jhtrq"] Oct 30 17:12:08.672971 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.672939 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:08.878139 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:08.878103 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:09.343261 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.343227 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv"] Oct 30 17:12:09.345486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.344020 6202 cpu_manager.go:414] "containerMap: removing container" podUID="2b6965bb-5f5c-40ca-b098-9f67c7433381" containerName="istio-proxy" Oct 30 17:12:09.345486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.344035 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6965bb-5f5c-40ca-b098-9f67c7433381" containerName="istio-proxy" Oct 30 17:12:09.345486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.344075 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a227113a-3241-42d2-98d8-4c67c1ca8f16" containerName="istio-proxy" Oct 30 17:12:09.345486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.344080 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a227113a-3241-42d2-98d8-4c67c1ca8f16" containerName="istio-proxy" Oct 30 17:12:09.345486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.344100 6202 cpu_manager.go:414] "containerMap: removing container" podUID="854625e6-cc7a-4f5d-9475-ec31b0258eef" containerName="payload-pre-processing" Oct 30 17:12:09.345486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.344106 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="854625e6-cc7a-4f5d-9475-ec31b0258eef" containerName="payload-pre-processing" Oct 30 17:12:09.345486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.344124 6202 cpu_manager.go:414] "containerMap: removing container" podUID="7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8" containerName="revoke-keys" Oct 30 17:12:09.345486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.344129 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8" containerName="revoke-keys" Oct 30 17:12:09.345486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.344240 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a227113a-3241-42d2-98d8-4c67c1ca8f16" containerName="istio-proxy" Oct 30 17:12:09.345486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.344248 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6965bb-5f5c-40ca-b098-9f67c7433381" containerName="istio-proxy" Oct 30 17:12:09.345486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.344255 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="854625e6-cc7a-4f5d-9475-ec31b0258eef" containerName="payload-pre-processing" Oct 30 17:12:09.345486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.344262 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5dbd37-6315-4bb7-b3aa-929a68ba7ef8" containerName="revoke-keys" Oct 30 17:12:09.348091 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.348073 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv" Oct 30 17:12:09.360008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.359978 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28fd6512-645b-40c7-a82a-5cd01b2b2212-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv\" (UID: \"28fd6512-645b-40c7-a82a-5cd01b2b2212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv" Oct 30 17:12:09.360272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.360163 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28fd6512-645b-40c7-a82a-5cd01b2b2212-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv\" (UID: \"28fd6512-645b-40c7-a82a-5cd01b2b2212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv" Oct 30 17:12:09.360272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.360203 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdrbw\" (UniqueName: \"kubernetes.io/projected/28fd6512-645b-40c7-a82a-5cd01b2b2212-kube-api-access-gdrbw\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv\" (UID: \"28fd6512-645b-40c7-a82a-5cd01b2b2212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv" Oct 30 17:12:09.368397 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.368371 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv"] Oct 30 17:12:09.374644 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.374621 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:12:09.374877 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.374859 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:12:09.446066 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.446028 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:12:09.446066 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.446040 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:12:09.462059 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.462024 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28fd6512-645b-40c7-a82a-5cd01b2b2212-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv\" (UID: \"28fd6512-645b-40c7-a82a-5cd01b2b2212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv" Oct 30 17:12:09.462249 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.462230 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28fd6512-645b-40c7-a82a-5cd01b2b2212-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv\" (UID: \"28fd6512-645b-40c7-a82a-5cd01b2b2212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv" Oct 30 17:12:09.462314 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.462268 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdrbw\" (UniqueName: \"kubernetes.io/projected/28fd6512-645b-40c7-a82a-5cd01b2b2212-kube-api-access-gdrbw\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv\" (UID: \"28fd6512-645b-40c7-a82a-5cd01b2b2212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv" Oct 30 17:12:09.463063 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.463032 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28fd6512-645b-40c7-a82a-5cd01b2b2212-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv\" (UID: \"28fd6512-645b-40c7-a82a-5cd01b2b2212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv" Oct 30 17:12:09.464666 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.464635 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28fd6512-645b-40c7-a82a-5cd01b2b2212-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv\" (UID: \"28fd6512-645b-40c7-a82a-5cd01b2b2212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv" Oct 30 17:12:09.476471 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.476444 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:09.486906 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.486883 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdrbw\" (UniqueName: \"kubernetes.io/projected/28fd6512-645b-40c7-a82a-5cd01b2b2212-kube-api-access-gdrbw\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv\" (UID: \"28fd6512-645b-40c7-a82a-5cd01b2b2212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv" Oct 30 17:12:09.575495 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.575462 6202 generic.go:356] "Generic (PLEG): container finished" podID="bbcf3ac0-0197-4960-8672-2fd5d9053729" containerID="44f4a301993e7495cd2e75baae830b425b878d7daabe9e253fe16b8483cd50df" exitCode=0 Oct 30 17:12:09.576045 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.575813 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" event={"ID":"bbcf3ac0-0197-4960-8672-2fd5d9053729","Type":"ContainerDied","Data":"44f4a301993e7495cd2e75baae830b425b878d7daabe9e253fe16b8483cd50df"} Oct 30 17:12:09.576341 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.576316 6202 scope.go:122] "RemoveContainer" containerID="44f4a301993e7495cd2e75baae830b425b878d7daabe9e253fe16b8483cd50df" Oct 30 17:12:09.579812 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.579758 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:12:09.579971 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.579895 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:12:09.579971 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.579965 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:12:09.682925 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:09.682895 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv" Oct 30 17:12:10.040957 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:12:10.040915 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fd6512_645b_40c7_a82a_5cd01b2b2212.slice/crio-66a3e7fcbe89369ba2ee7f240697cd79e8ceb0fe84cf5cb89a68796b63e46dcd WatchSource:0}: Error finding container 66a3e7fcbe89369ba2ee7f240697cd79e8ceb0fe84cf5cb89a68796b63e46dcd: Status 404 returned error can't find the container with id 66a3e7fcbe89369ba2ee7f240697cd79e8ceb0fe84cf5cb89a68796b63e46dcd Oct 30 17:12:10.041219 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:10.041195 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv"] Oct 30 17:12:10.052033 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:10.052001 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd"] Oct 30 17:12:10.059000 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:10.058940 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c0624f-t7sqd"] Oct 30 17:12:10.420558 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:10.420500 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6965bb-5f5c-40ca-b098-9f67c7433381" path="/var/lib/kubelet/pods/2b6965bb-5f5c-40ca-b098-9f67c7433381/volumes" Oct 30 17:12:10.421288 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:10.421263 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43" path="/var/lib/kubelet/pods/a0fb4e7f-b0a3-45f6-bc7a-7ccc1d3edc43/volumes" Oct 30 17:12:10.589509 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:10.589467 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv" event={"ID":"28fd6512-645b-40c7-a82a-5cd01b2b2212","Type":"ContainerStarted","Data":"1f9e893d4bd5d69916944522b7d992d1f2d9646fe783639145fa0222defd40bf"} Oct 30 17:12:10.590035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:10.589534 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv" event={"ID":"28fd6512-645b-40c7-a82a-5cd01b2b2212","Type":"ContainerStarted","Data":"66a3e7fcbe89369ba2ee7f240697cd79e8ceb0fe84cf5cb89a68796b63e46dcd"} Oct 30 17:12:10.591794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:10.591739 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" event={"ID":"bbcf3ac0-0197-4960-8672-2fd5d9053729","Type":"ContainerStarted","Data":"829c0276f97159669765ababa07fcd992cc8472b9a4ffb144489de9795365a89"} Oct 30 17:12:10.634827 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:10.634743 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv" podStartSLOduration=1.634730306 podStartE2EDuration="1.634730306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:10.631033979 +0000 UTC m=+3044.341488725" watchObservedRunningTime="2026-10-30 17:12:10.634730306 +0000 UTC m=+3044.345185050" Oct 30 17:12:10.957791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:10.948180 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:12:10.958030 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:10.957994 6202 patch_prober.go:29] interesting pod/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body= Oct 30 17:12:10.958110 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:10.958068 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" podUID="bbcf3ac0-0197-4960-8672-2fd5d9053729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" Oct 30 17:12:11.032007 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:11.031972 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f"] Oct 30 17:12:11.054973 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:11.054934 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0efcbc66-pzn2f"] Oct 30 17:12:11.503665 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:11.503623 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:11.601619 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:11.601584 6202 generic.go:356] "Generic (PLEG): container finished" podID="28fd6512-645b-40c7-a82a-5cd01b2b2212" containerID="1f9e893d4bd5d69916944522b7d992d1f2d9646fe783639145fa0222defd40bf" exitCode=0 Oct 30 17:12:11.602083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:11.601673 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv" event={"ID":"28fd6512-645b-40c7-a82a-5cd01b2b2212","Type":"ContainerDied","Data":"1f9e893d4bd5d69916944522b7d992d1f2d9646fe783639145fa0222defd40bf"} Oct 30 17:12:11.945850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:11.942048 6202 patch_prober.go:29] interesting pod/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body= Oct 30 17:12:11.945850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:11.942103 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" podUID="bbcf3ac0-0197-4960-8672-2fd5d9053729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" Oct 30 17:12:11.970009 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:11.969974 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:12:12.415927 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:12.415891 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8" path="/var/lib/kubelet/pods/5ba6e18e-27a3-4fd7-946e-f8bb47d6a5c8/volumes" Oct 30 17:12:12.829690 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:12.829663 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv" Oct 30 17:12:12.969849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:12.968941 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28fd6512-645b-40c7-a82a-5cd01b2b2212-kube-api-access-gdrbw\" (UniqueName: \"kubernetes.io/projected/28fd6512-645b-40c7-a82a-5cd01b2b2212-kube-api-access-gdrbw\") pod \"28fd6512-645b-40c7-a82a-5cd01b2b2212\" (UID: \"28fd6512-645b-40c7-a82a-5cd01b2b2212\") " Oct 30 17:12:12.969849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:12.969141 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28fd6512-645b-40c7-a82a-5cd01b2b2212-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28fd6512-645b-40c7-a82a-5cd01b2b2212-maas-api-xks-ca\") pod \"28fd6512-645b-40c7-a82a-5cd01b2b2212\" (UID: \"28fd6512-645b-40c7-a82a-5cd01b2b2212\") " Oct 30 17:12:12.969849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:12.969407 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28fd6512-645b-40c7-a82a-5cd01b2b2212-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28fd6512-645b-40c7-a82a-5cd01b2b2212-maas-api-openshift-ca\") pod \"28fd6512-645b-40c7-a82a-5cd01b2b2212\" (UID: \"28fd6512-645b-40c7-a82a-5cd01b2b2212\") " Oct 30 17:12:12.974072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:12.973591 6202 patch_prober.go:29] interesting pod/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body= Oct 30 17:12:12.974072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:12.973647 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" podUID="bbcf3ac0-0197-4960-8672-2fd5d9053729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" Oct 30 17:12:12.974669 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:12.974630 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28fd6512-645b-40c7-a82a-5cd01b2b2212-maas-api-openshift-ca" pod "28fd6512-645b-40c7-a82a-5cd01b2b2212" (UID: "28fd6512-645b-40c7-a82a-5cd01b2b2212"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:12.976232 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:12.976201 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28fd6512-645b-40c7-a82a-5cd01b2b2212-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:12.977693 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:12.977657 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28fd6512-645b-40c7-a82a-5cd01b2b2212-maas-api-xks-ca" pod "28fd6512-645b-40c7-a82a-5cd01b2b2212" (UID: "28fd6512-645b-40c7-a82a-5cd01b2b2212"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:12:12.978021 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:12.977989 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fd6512-645b-40c7-a82a-5cd01b2b2212-kube-api-access-gdrbw" pod "28fd6512-645b-40c7-a82a-5cd01b2b2212" (UID: "28fd6512-645b-40c7-a82a-5cd01b2b2212"). InnerVolumeSpecName "kube-api-access-gdrbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:13.077938 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:13.077896 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdrbw\" (UniqueName: \"kubernetes.io/projected/28fd6512-645b-40c7-a82a-5cd01b2b2212-kube-api-access-gdrbw\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:13.077938 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:13.077930 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28fd6512-645b-40c7-a82a-5cd01b2b2212-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:13.620955 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:13.620919 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv" event={"ID":"28fd6512-645b-40c7-a82a-5cd01b2b2212","Type":"ContainerDied","Data":"66a3e7fcbe89369ba2ee7f240697cd79e8ceb0fe84cf5cb89a68796b63e46dcd"} Oct 30 17:12:13.620955 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:13.620948 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv" Oct 30 17:12:13.621175 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:13.620971 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66a3e7fcbe89369ba2ee7f240697cd79e8ceb0fe84cf5cb89a68796b63e46dcd" Oct 30 17:12:13.945860 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:13.944158 6202 patch_prober.go:29] interesting pod/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body= Oct 30 17:12:13.945860 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:13.944221 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" podUID="bbcf3ac0-0197-4960-8672-2fd5d9053729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" Oct 30 17:12:13.966494 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:13.966436 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:14.111802 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:12:14.110546 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:12:14.950794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:14.946134 6202 patch_prober.go:29] interesting pod/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body= Oct 30 17:12:14.950794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:14.946217 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" podUID="bbcf3ac0-0197-4960-8672-2fd5d9053729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" Oct 30 17:12:15.234870 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:15.234761 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:15.398792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:15.395359 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:15.753864 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:15.753828 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4"] Oct 30 17:12:15.755392 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:15.755361 6202 cpu_manager.go:414] "containerMap: removing container" podUID="2b6965bb-5f5c-40ca-b098-9f67c7433381" containerName="istio-proxy" Oct 30 17:12:15.755572 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:15.755558 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6965bb-5f5c-40ca-b098-9f67c7433381" containerName="istio-proxy" Oct 30 17:12:15.755692 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:15.755682 6202 cpu_manager.go:414] "containerMap: removing container" podUID="28fd6512-645b-40c7-a82a-5cd01b2b2212" containerName="revoke-keys" Oct 30 17:12:15.755812 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:15.755799 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28fd6512-645b-40c7-a82a-5cd01b2b2212" containerName="revoke-keys" Oct 30 17:12:15.756113 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:15.756097 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="28fd6512-645b-40c7-a82a-5cd01b2b2212" containerName="revoke-keys" Oct 30 17:12:15.756237 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:15.756226 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6965bb-5f5c-40ca-b098-9f67c7433381" containerName="istio-proxy" Oct 30 17:12:15.959845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:15.958579 6202 patch_prober.go:29] interesting pod/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body= Oct 30 17:12:15.959845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:15.958651 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" podUID="bbcf3ac0-0197-4960-8672-2fd5d9053729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" Oct 30 17:12:15.983789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:15.982640 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4"] Oct 30 17:12:15.983789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:15.982701 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv"] Oct 30 17:12:15.990282 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:15.984946 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4" Oct 30 17:12:15.998000 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:15.993355 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv" Oct 30 17:12:16.001606 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.001577 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv"] Oct 30 17:12:16.002606 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.002581 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-698c71-dockercfg-45tlr\"" Oct 30 17:12:16.004981 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.004380 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-698c71\"" Oct 30 17:12:16.093398 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.093362 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55546d54-81a4-402e-8ad5-7110cf03a17c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv\" (UID: \"55546d54-81a4-402e-8ad5-7110cf03a17c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv" Oct 30 17:12:16.093598 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.093434 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt7xj\" (UniqueName: \"kubernetes.io/projected/55546d54-81a4-402e-8ad5-7110cf03a17c-kube-api-access-kt7xj\") pod \"payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv\" (UID: \"55546d54-81a4-402e-8ad5-7110cf03a17c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv" Oct 30 17:12:16.093598 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.093557 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ed3178-5d8b-4cb5-9ec6-737aa118b73a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4\" (UID: \"66ed3178-5d8b-4cb5-9ec6-737aa118b73a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4" Oct 30 17:12:16.093730 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.093598 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmbq6\" (UniqueName: \"kubernetes.io/projected/66ed3178-5d8b-4cb5-9ec6-737aa118b73a-kube-api-access-lmbq6\") pod \"payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4\" (UID: \"66ed3178-5d8b-4cb5-9ec6-737aa118b73a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4" Oct 30 17:12:16.195102 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.195043 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ed3178-5d8b-4cb5-9ec6-737aa118b73a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4\" (UID: \"66ed3178-5d8b-4cb5-9ec6-737aa118b73a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4" Oct 30 17:12:16.195102 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.195095 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmbq6\" (UniqueName: \"kubernetes.io/projected/66ed3178-5d8b-4cb5-9ec6-737aa118b73a-kube-api-access-lmbq6\") pod \"payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4\" (UID: \"66ed3178-5d8b-4cb5-9ec6-737aa118b73a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4" Oct 30 17:12:16.195453 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.195267 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55546d54-81a4-402e-8ad5-7110cf03a17c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv\" (UID: \"55546d54-81a4-402e-8ad5-7110cf03a17c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv" Oct 30 17:12:16.195453 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.195335 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt7xj\" (UniqueName: \"kubernetes.io/projected/55546d54-81a4-402e-8ad5-7110cf03a17c-kube-api-access-kt7xj\") pod \"payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv\" (UID: \"55546d54-81a4-402e-8ad5-7110cf03a17c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv" Oct 30 17:12:16.196082 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.196050 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ed3178-5d8b-4cb5-9ec6-737aa118b73a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4\" (UID: \"66ed3178-5d8b-4cb5-9ec6-737aa118b73a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4" Oct 30 17:12:16.196189 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.196094 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55546d54-81a4-402e-8ad5-7110cf03a17c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv\" (UID: \"55546d54-81a4-402e-8ad5-7110cf03a17c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv" Oct 30 17:12:16.221040 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.221006 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt7xj\" (UniqueName: \"kubernetes.io/projected/55546d54-81a4-402e-8ad5-7110cf03a17c-kube-api-access-kt7xj\") pod \"payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv\" (UID: \"55546d54-81a4-402e-8ad5-7110cf03a17c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv" Oct 30 17:12:16.221248 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.221006 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmbq6\" (UniqueName: \"kubernetes.io/projected/66ed3178-5d8b-4cb5-9ec6-737aa118b73a-kube-api-access-lmbq6\") pod \"payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4\" (UID: \"66ed3178-5d8b-4cb5-9ec6-737aa118b73a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4" Oct 30 17:12:16.319536 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.319441 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4" Oct 30 17:12:16.325675 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.325650 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv" Oct 30 17:12:16.508200 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.508160 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:16.668985 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:12:16.668955 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55546d54_81a4_402e_8ad5_7110cf03a17c.slice/crio-6fcf90dae61c6752013272d6bc9899dbe4869e8424d3227d9bd7836bfa1c86ab WatchSource:0}: Error finding container 6fcf90dae61c6752013272d6bc9899dbe4869e8424d3227d9bd7836bfa1c86ab: Status 404 returned error can't find the container with id 6fcf90dae61c6752013272d6bc9899dbe4869e8424d3227d9bd7836bfa1c86ab Oct 30 17:12:16.670065 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.670034 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv"] Oct 30 17:12:16.675302 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.675271 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4"] Oct 30 17:12:16.677995 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:12:16.677966 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ed3178_5d8b_4cb5_9ec6_737aa118b73a.slice/crio-da26dad81747a804d1ed620467e41ae98ecf821680b5503cf336a86a62aa7523 WatchSource:0}: Error finding container da26dad81747a804d1ed620467e41ae98ecf821680b5503cf336a86a62aa7523: Status 404 returned error can't find the container with id da26dad81747a804d1ed620467e41ae98ecf821680b5503cf336a86a62aa7523 Oct 30 17:12:16.737748 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.737681 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv" event={"ID":"55546d54-81a4-402e-8ad5-7110cf03a17c","Type":"ContainerStarted","Data":"6fcf90dae61c6752013272d6bc9899dbe4869e8424d3227d9bd7836bfa1c86ab"} Oct 30 17:12:16.740139 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.740109 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4" event={"ID":"66ed3178-5d8b-4cb5-9ec6-737aa118b73a","Type":"ContainerStarted","Data":"da26dad81747a804d1ed620467e41ae98ecf821680b5503cf336a86a62aa7523"} Oct 30 17:12:16.948649 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.948559 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:12:16.950552 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:16.950523 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:12:17.031596 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:17.031552 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2"] Oct 30 17:12:17.035144 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:17.035119 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7c2cee-xdds2"] Oct 30 17:12:17.750326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:17.750283 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4" event={"ID":"66ed3178-5d8b-4cb5-9ec6-737aa118b73a","Type":"ContainerStarted","Data":"7338bbcc995e02b946317d450b09f450cf68b44ddbb457c05934fad8d40d5c22"} Oct 30 17:12:17.750533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:17.750377 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4" Oct 30 17:12:17.753019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:17.752986 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv" event={"ID":"55546d54-81a4-402e-8ad5-7110cf03a17c","Type":"ContainerStarted","Data":"b2ee111001cdebadb31555053db5e95ed6a1c04510ac7e068bea3711d64ceb87"} Oct 30 17:12:17.795368 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:17.795312 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4" podStartSLOduration=2.7952981 podStartE2EDuration="2.7952981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:17.79264118 +0000 UTC m=+3051.503096126" watchObservedRunningTime="2026-10-30 17:12:17.7952981 +0000 UTC m=+3051.505752878" Oct 30 17:12:17.836930 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:17.836876 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv" podStartSLOduration=2.83686044 podStartE2EDuration="2.83686044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:17.835157489 +0000 UTC m=+3051.545612233" watchObservedRunningTime="2026-10-30 17:12:17.83686044 +0000 UTC m=+3051.547315184" Oct 30 17:12:18.290622 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:18.290582 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:18.413541 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:18.413506 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eaa54db-afe1-4a37-ba9f-1e0745826d93" path="/var/lib/kubelet/pods/3eaa54db-afe1-4a37-ba9f-1e0745826d93/volumes" Oct 30 17:12:18.761342 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:18.761304 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv" Oct 30 17:12:19.032710 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:19.032626 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k"] Oct 30 17:12:19.045851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:19.045814 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4a7f8be-zw49k"] Oct 30 17:12:19.749602 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:19.749569 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6"] Oct 30 17:12:19.883577 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:19.883541 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:20.015318 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:20.015217 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:20.174832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:20.163176 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:20.459792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:20.458966 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="721fc1d7-6ad4-4f60-ae9f-6503de4d0331" path="/var/lib/kubelet/pods/721fc1d7-6ad4-4f60-ae9f-6503de4d0331/volumes" Oct 30 17:12:22.081077 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:22.081017 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:23.865190 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:23.865149 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:24.750838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:24.750798 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6" podUID="1f9bd673-68e5-4f3a-9c34-ffc286feaf04" containerName="payload-processing" containerID="cri-o://3a34f7c58155893ec8d2abf7b923b6aa1ec8098b1ecaee276f07537553dab5b6" gracePeriod=25 Oct 30 17:12:25.043444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:25.043417 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6" Oct 30 17:12:25.163710 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:25.163671 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9bd673-68e5-4f3a-9c34-ffc286feaf04-kube-api-access-kvq4h\" (UniqueName: \"kubernetes.io/projected/1f9bd673-68e5-4f3a-9c34-ffc286feaf04-kube-api-access-kvq4h\") pod \"1f9bd673-68e5-4f3a-9c34-ffc286feaf04\" (UID: \"1f9bd673-68e5-4f3a-9c34-ffc286feaf04\") " Oct 30 17:12:25.163967 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:25.163912 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f9bd673-68e5-4f3a-9c34-ffc286feaf04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f9bd673-68e5-4f3a-9c34-ffc286feaf04-plugins-config-volume\") pod \"1f9bd673-68e5-4f3a-9c34-ffc286feaf04\" (UID: \"1f9bd673-68e5-4f3a-9c34-ffc286feaf04\") " Oct 30 17:12:25.164316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:25.164285 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f9bd673-68e5-4f3a-9c34-ffc286feaf04-plugins-config-volume" pod "1f9bd673-68e5-4f3a-9c34-ffc286feaf04" (UID: "1f9bd673-68e5-4f3a-9c34-ffc286feaf04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:25.164684 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:25.164652 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f9bd673-68e5-4f3a-9c34-ffc286feaf04-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:25.166227 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:25.166197 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9bd673-68e5-4f3a-9c34-ffc286feaf04-kube-api-access-kvq4h" pod "1f9bd673-68e5-4f3a-9c34-ffc286feaf04" (UID: "1f9bd673-68e5-4f3a-9c34-ffc286feaf04"). InnerVolumeSpecName "kube-api-access-kvq4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:25.267017 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:25.266922 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvq4h\" (UniqueName: \"kubernetes.io/projected/1f9bd673-68e5-4f3a-9c34-ffc286feaf04-kube-api-access-kvq4h\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:25.615427 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:25.615391 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:25.824947 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:25.824903 6202 generic.go:356] "Generic (PLEG): container finished" podID="1f9bd673-68e5-4f3a-9c34-ffc286feaf04" containerID="3a34f7c58155893ec8d2abf7b923b6aa1ec8098b1ecaee276f07537553dab5b6" exitCode=0 Oct 30 17:12:25.825146 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:25.824964 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6" Oct 30 17:12:25.825146 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:25.824955 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6" event={"ID":"1f9bd673-68e5-4f3a-9c34-ffc286feaf04","Type":"ContainerDied","Data":"3a34f7c58155893ec8d2abf7b923b6aa1ec8098b1ecaee276f07537553dab5b6"} Oct 30 17:12:25.825146 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:25.825129 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6" event={"ID":"1f9bd673-68e5-4f3a-9c34-ffc286feaf04","Type":"ContainerDied","Data":"7bef02e01565b31b231706e64d95de51429308e0636eb36eb26928c2ebbf2427"} Oct 30 17:12:25.825341 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:25.825162 6202 scope.go:122] "RemoveContainer" containerID="3a34f7c58155893ec8d2abf7b923b6aa1ec8098b1ecaee276f07537553dab5b6" Oct 30 17:12:25.837239 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:25.837217 6202 scope.go:122] "RemoveContainer" containerID="3a34f7c58155893ec8d2abf7b923b6aa1ec8098b1ecaee276f07537553dab5b6" Oct 30 17:12:25.837477 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:12:25.837461 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a34f7c58155893ec8d2abf7b923b6aa1ec8098b1ecaee276f07537553dab5b6\": container with ID starting with 3a34f7c58155893ec8d2abf7b923b6aa1ec8098b1ecaee276f07537553dab5b6 not found: ID does not exist" containerID="3a34f7c58155893ec8d2abf7b923b6aa1ec8098b1ecaee276f07537553dab5b6" Oct 30 17:12:25.837527 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:25.837500 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a34f7c58155893ec8d2abf7b923b6aa1ec8098b1ecaee276f07537553dab5b6"} err="failed to get container status \"3a34f7c58155893ec8d2abf7b923b6aa1ec8098b1ecaee276f07537553dab5b6\": rpc error: code = NotFound desc = could not find container \"3a34f7c58155893ec8d2abf7b923b6aa1ec8098b1ecaee276f07537553dab5b6\": container with ID starting with 3a34f7c58155893ec8d2abf7b923b6aa1ec8098b1ecaee276f07537553dab5b6 not found: ID does not exist" Oct 30 17:12:25.884551 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:25.884474 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6"] Oct 30 17:12:25.895273 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:25.895242 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-1b7bb3-5fcfb94c78-22hx6"] Oct 30 17:12:26.413968 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:26.413935 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f9bd673-68e5-4f3a-9c34-ffc286feaf04" path="/var/lib/kubelet/pods/1f9bd673-68e5-4f3a-9c34-ffc286feaf04/volumes" Oct 30 17:12:27.369135 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:27.369101 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:28.763643 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:28.763612 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4" Oct 30 17:12:29.097041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:29.097011 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:29.773489 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:29.773455 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv" Oct 30 17:12:31.329037 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:31.329002 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:33.479450 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:33.478653 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:33.953235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:33.953196 6202 scope.go:122] "RemoveContainer" containerID="9fec06199b5da0002d0378c6365cd71905be72ab04e801ffb7395f4702059893" Oct 30 17:12:33.964188 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:33.964163 6202 scope.go:122] "RemoveContainer" containerID="40413c7ad35a7b14921d00b8634c1e61d3959f83160e47863d357e08e7ef7d2c" Oct 30 17:12:34.269888 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.269799 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk"] Oct 30 17:12:34.270622 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.270604 6202 cpu_manager.go:414] "containerMap: removing container" podUID="1f9bd673-68e5-4f3a-9c34-ffc286feaf04" containerName="payload-processing" Oct 30 17:12:34.270622 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.270621 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9bd673-68e5-4f3a-9c34-ffc286feaf04" containerName="payload-processing" Oct 30 17:12:34.270790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.270735 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9bd673-68e5-4f3a-9c34-ffc286feaf04" containerName="payload-processing" Oct 30 17:12:34.276356 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.276330 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk" Oct 30 17:12:34.295683 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.295643 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk"] Oct 30 17:12:34.303799 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.303742 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:12:34.303935 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.303840 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:12:34.303935 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.303928 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:12:34.305198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.305179 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:12:34.413160 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.413126 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8lx2\" (UniqueName: \"kubernetes.io/projected/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-kube-api-access-n8lx2\") pod \"maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk\" (UID: \"0bded893-57fb-4d38-ba0c-fa06f1ace2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk" Oct 30 17:12:34.413374 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.413355 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk\" (UID: \"0bded893-57fb-4d38-ba0c-fa06f1ace2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk" Oct 30 17:12:34.413432 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.413386 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk\" (UID: \"0bded893-57fb-4d38-ba0c-fa06f1ace2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk" Oct 30 17:12:34.515105 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.515076 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8lx2\" (UniqueName: \"kubernetes.io/projected/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-kube-api-access-n8lx2\") pod \"maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk\" (UID: \"0bded893-57fb-4d38-ba0c-fa06f1ace2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk" Oct 30 17:12:34.515687 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.515335 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk\" (UID: \"0bded893-57fb-4d38-ba0c-fa06f1ace2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk" Oct 30 17:12:34.515687 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.515373 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk\" (UID: \"0bded893-57fb-4d38-ba0c-fa06f1ace2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk" Oct 30 17:12:34.516040 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.516011 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk\" (UID: \"0bded893-57fb-4d38-ba0c-fa06f1ace2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk" Oct 30 17:12:34.518015 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.517991 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk\" (UID: \"0bded893-57fb-4d38-ba0c-fa06f1ace2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk" Oct 30 17:12:34.539968 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.539894 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8lx2\" (UniqueName: \"kubernetes.io/projected/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-kube-api-access-n8lx2\") pod \"maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk\" (UID: \"0bded893-57fb-4d38-ba0c-fa06f1ace2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk" Oct 30 17:12:34.591412 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.591381 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk" Oct 30 17:12:34.899483 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:12:34.899445 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bded893_57fb_4d38_ba0c_fa06f1ace2a2.slice/crio-a58f902e22dba39bd271006dcc50128832e97afabb5ccadc42c084895022479d WatchSource:0}: Error finding container a58f902e22dba39bd271006dcc50128832e97afabb5ccadc42c084895022479d: Status 404 returned error can't find the container with id a58f902e22dba39bd271006dcc50128832e97afabb5ccadc42c084895022479d Oct 30 17:12:34.901208 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:34.901179 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk"] Oct 30 17:12:35.080833 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:35.066287 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk" event={"ID":"0bded893-57fb-4d38-ba0c-fa06f1ace2a2","Type":"ContainerStarted","Data":"a58f902e22dba39bd271006dcc50128832e97afabb5ccadc42c084895022479d"} Oct 30 17:12:35.516153 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:35.516031 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:35.631868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:35.631823 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:35.752248 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:35.752195 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:36.159832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:36.159720 6202 generic.go:356] "Generic (PLEG): container finished" podID="0bded893-57fb-4d38-ba0c-fa06f1ace2a2" containerID="ad8df48270bc962a37e0181022ff5fe82e26babb2efed5189c271fb67463a728" exitCode=0 Oct 30 17:12:36.160400 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:36.160374 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk" event={"ID":"0bded893-57fb-4d38-ba0c-fa06f1ace2a2","Type":"ContainerDied","Data":"ad8df48270bc962a37e0181022ff5fe82e26babb2efed5189c271fb67463a728"} Oct 30 17:12:36.600173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:36.600135 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:37.363335 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:37.363309 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk" Oct 30 17:12:37.502654 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:37.502555 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-maas-api-xks-ca\") pod \"0bded893-57fb-4d38-ba0c-fa06f1ace2a2\" (UID: \"0bded893-57fb-4d38-ba0c-fa06f1ace2a2\") " Oct 30 17:12:37.502868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:37.502789 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-maas-api-openshift-ca\") pod \"0bded893-57fb-4d38-ba0c-fa06f1ace2a2\" (UID: \"0bded893-57fb-4d38-ba0c-fa06f1ace2a2\") " Oct 30 17:12:37.502960 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:37.502914 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-kube-api-access-n8lx2\" (UniqueName: \"kubernetes.io/projected/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-kube-api-access-n8lx2\") pod \"0bded893-57fb-4d38-ba0c-fa06f1ace2a2\" (UID: \"0bded893-57fb-4d38-ba0c-fa06f1ace2a2\") " Oct 30 17:12:37.503094 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:37.503066 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-maas-api-openshift-ca" pod "0bded893-57fb-4d38-ba0c-fa06f1ace2a2" (UID: "0bded893-57fb-4d38-ba0c-fa06f1ace2a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:37.504076 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:37.504047 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:37.505186 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:37.505158 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-maas-api-xks-ca" pod "0bded893-57fb-4d38-ba0c-fa06f1ace2a2" (UID: "0bded893-57fb-4d38-ba0c-fa06f1ace2a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:12:37.505512 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:37.505491 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-kube-api-access-n8lx2" pod "0bded893-57fb-4d38-ba0c-fa06f1ace2a2" (UID: "0bded893-57fb-4d38-ba0c-fa06f1ace2a2"). InnerVolumeSpecName "kube-api-access-n8lx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:37.605518 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:37.605469 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8lx2\" (UniqueName: \"kubernetes.io/projected/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-kube-api-access-n8lx2\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:37.605518 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:37.605513 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bded893-57fb-4d38-ba0c-fa06f1ace2a2-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:12:38.178924 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:38.178895 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk" Oct 30 17:12:38.179130 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:38.178891 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk" event={"ID":"0bded893-57fb-4d38-ba0c-fa06f1ace2a2","Type":"ContainerDied","Data":"a58f902e22dba39bd271006dcc50128832e97afabb5ccadc42c084895022479d"} Oct 30 17:12:38.179130 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:38.179022 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a58f902e22dba39bd271006dcc50128832e97afabb5ccadc42c084895022479d" Oct 30 17:12:38.613495 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:38.608167 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:39.877594 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:39.877559 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:40.063792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.063685 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:40.520789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.504366 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c"] Oct 30 17:12:40.523430 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.523397 6202 cpu_manager.go:414] "containerMap: removing container" podUID="0bded893-57fb-4d38-ba0c-fa06f1ace2a2" containerName="revoke-keys" Oct 30 17:12:40.523641 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.523627 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bded893-57fb-4d38-ba0c-fa06f1ace2a2" containerName="revoke-keys" Oct 30 17:12:40.523982 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.523967 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bded893-57fb-4d38-ba0c-fa06f1ace2a2" containerName="revoke-keys" Oct 30 17:12:40.547015 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.546981 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c" Oct 30 17:12:40.593434 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.593394 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-de18d5\"" Oct 30 17:12:40.593873 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.593840 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-de18d5-dockercfg-kcjnh\"" Oct 30 17:12:40.627111 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.626190 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5h8d\" (UniqueName: \"kubernetes.io/projected/adeec1f0-813d-4ade-bab6-b1f8d49a6f28-kube-api-access-f5h8d\") pod \"payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c\" (UID: \"adeec1f0-813d-4ade-bab6-b1f8d49a6f28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c" Oct 30 17:12:40.627111 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.626404 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adeec1f0-813d-4ade-bab6-b1f8d49a6f28-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c\" (UID: \"adeec1f0-813d-4ade-bab6-b1f8d49a6f28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c" Oct 30 17:12:40.633442 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.633409 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c"] Oct 30 17:12:40.669613 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.669576 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b"] Oct 30 17:12:40.689020 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.688986 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b"] Oct 30 17:12:40.689298 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.689138 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b" Oct 30 17:12:40.729181 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.729152 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5h8d\" (UniqueName: \"kubernetes.io/projected/adeec1f0-813d-4ade-bab6-b1f8d49a6f28-kube-api-access-f5h8d\") pod \"payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c\" (UID: \"adeec1f0-813d-4ade-bab6-b1f8d49a6f28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c" Oct 30 17:12:40.729338 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.729323 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cmws\" (UniqueName: \"kubernetes.io/projected/8f2f997e-e6fc-46d2-bb0f-68c947c8607f-kube-api-access-9cmws\") pod \"payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b\" (UID: \"8f2f997e-e6fc-46d2-bb0f-68c947c8607f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b" Oct 30 17:12:40.729406 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.729357 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adeec1f0-813d-4ade-bab6-b1f8d49a6f28-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c\" (UID: \"adeec1f0-813d-4ade-bab6-b1f8d49a6f28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c" Oct 30 17:12:40.729466 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.729448 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2f997e-e6fc-46d2-bb0f-68c947c8607f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b\" (UID: \"8f2f997e-e6fc-46d2-bb0f-68c947c8607f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b" Oct 30 17:12:40.730061 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.730037 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adeec1f0-813d-4ade-bab6-b1f8d49a6f28-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c\" (UID: \"adeec1f0-813d-4ade-bab6-b1f8d49a6f28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c" Oct 30 17:12:40.753975 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.753947 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5h8d\" (UniqueName: \"kubernetes.io/projected/adeec1f0-813d-4ade-bab6-b1f8d49a6f28-kube-api-access-f5h8d\") pod \"payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c\" (UID: \"adeec1f0-813d-4ade-bab6-b1f8d49a6f28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c" Oct 30 17:12:40.831110 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.831074 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cmws\" (UniqueName: \"kubernetes.io/projected/8f2f997e-e6fc-46d2-bb0f-68c947c8607f-kube-api-access-9cmws\") pod \"payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b\" (UID: \"8f2f997e-e6fc-46d2-bb0f-68c947c8607f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b" Oct 30 17:12:40.831335 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.831137 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2f997e-e6fc-46d2-bb0f-68c947c8607f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b\" (UID: \"8f2f997e-e6fc-46d2-bb0f-68c947c8607f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b" Oct 30 17:12:40.831863 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.831840 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2f997e-e6fc-46d2-bb0f-68c947c8607f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b\" (UID: \"8f2f997e-e6fc-46d2-bb0f-68c947c8607f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b" Oct 30 17:12:40.856100 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.856065 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cmws\" (UniqueName: \"kubernetes.io/projected/8f2f997e-e6fc-46d2-bb0f-68c947c8607f-kube-api-access-9cmws\") pod \"payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b\" (UID: \"8f2f997e-e6fc-46d2-bb0f-68c947c8607f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b" Oct 30 17:12:40.921533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:40.921501 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c" Oct 30 17:12:41.006453 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:41.006191 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b" Oct 30 17:12:41.226592 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:12:41.226547 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadeec1f0_813d_4ade_bab6_b1f8d49a6f28.slice/crio-dbe6029b3bad149515bafd84d77ed8b2bba6ab3db58671a7fd6f5b1c369fe916 WatchSource:0}: Error finding container dbe6029b3bad149515bafd84d77ed8b2bba6ab3db58671a7fd6f5b1c369fe916: Status 404 returned error can't find the container with id dbe6029b3bad149515bafd84d77ed8b2bba6ab3db58671a7fd6f5b1c369fe916 Oct 30 17:12:41.227855 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:41.227827 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c"] Oct 30 17:12:41.239655 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:41.239625 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c" event={"ID":"adeec1f0-813d-4ade-bab6-b1f8d49a6f28","Type":"ContainerStarted","Data":"dbe6029b3bad149515bafd84d77ed8b2bba6ab3db58671a7fd6f5b1c369fe916"} Oct 30 17:12:41.335911 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:41.335872 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b"] Oct 30 17:12:41.336065 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:12:41.335942 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2f997e_e6fc_46d2_bb0f_68c947c8607f.slice/crio-35732afc389625bea0373ce4d6f23dcc61f24428b1f3af1aabe5acec53ffb93e WatchSource:0}: Error finding container 35732afc389625bea0373ce4d6f23dcc61f24428b1f3af1aabe5acec53ffb93e: Status 404 returned error can't find the container with id 35732afc389625bea0373ce4d6f23dcc61f24428b1f3af1aabe5acec53ffb93e Oct 30 17:12:41.438392 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:41.438351 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:42.250871 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:42.250834 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b" event={"ID":"8f2f997e-e6fc-46d2-bb0f-68c947c8607f","Type":"ContainerStarted","Data":"71c57426929a633bc47406761c973ffeef018f3bab52ea70bb0befa874a15ebd"} Oct 30 17:12:42.251370 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:42.250897 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b" event={"ID":"8f2f997e-e6fc-46d2-bb0f-68c947c8607f","Type":"ContainerStarted","Data":"35732afc389625bea0373ce4d6f23dcc61f24428b1f3af1aabe5acec53ffb93e"} Oct 30 17:12:42.251370 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:42.250949 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b" Oct 30 17:12:42.253784 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:42.253738 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c" event={"ID":"adeec1f0-813d-4ade-bab6-b1f8d49a6f28","Type":"ContainerStarted","Data":"59c77a4e488404f2c9e79e9464b7575e6698d10051175a5bdebc1b9739a68110"} Oct 30 17:12:42.253939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:42.253834 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c" Oct 30 17:12:42.295039 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:42.294977 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b" podStartSLOduration=2.29495809 podStartE2EDuration="2.29495809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:42.293476104 +0000 UTC m=+3076.003930845" watchObservedRunningTime="2026-10-30 17:12:42.29495809 +0000 UTC m=+3076.005412835" Oct 30 17:12:42.337957 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:42.337900 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c" podStartSLOduration=2.337884795 podStartE2EDuration="2.337884795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:42.337238462 +0000 UTC m=+3076.047693205" watchObservedRunningTime="2026-10-30 17:12:42.337884795 +0000 UTC m=+3076.048339541" Oct 30 17:12:42.412273 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:42.412239 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:43.655214 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:43.655176 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:44.073811 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:12:44.073475 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:12:45.490414 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:45.490380 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:47.253441 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:47.253400 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:49.052035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:49.051998 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:50.871366 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:50.871334 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:52.684597 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:52.684561 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:53.265585 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:53.265556 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b" Oct 30 17:12:53.265835 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:53.265634 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c" Oct 30 17:12:54.374661 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:54.374626 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:54.751376 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:54.751242 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:55.227675 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:55.227593 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:55.341480 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:55.341441 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:56.485547 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:56.485502 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:58.095651 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:58.095616 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:58.221604 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:58.221529 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:58.337310 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:58.337230 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:58.509499 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:58.509420 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:59.370896 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:59.370852 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:59.483000 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:12:59.482964 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:00.393828 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:00.393757 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:01.028713 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:01.028665 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:01.146377 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:01.146342 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:02.735176 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:02.735145 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:04.670449 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:04.670411 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:06.458798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:06.452759 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:08.720085 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:08.720042 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:10.704146 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:10.704113 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:11.400886 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:11.400851 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:11.514899 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:11.514864 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:11.551335 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:11.551302 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv"] Oct 30 17:13:11.555181 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:11.555108 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-1b7bb3-hwsdv"] Oct 30 17:13:11.624209 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:11.624177 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:12.022631 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:12.022592 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:12.412426 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:12.412387 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28fd6512-645b-40c7-a82a-5cd01b2b2212" path="/var/lib/kubelet/pods/28fd6512-645b-40c7-a82a-5cd01b2b2212/volumes" Oct 30 17:13:13.470136 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:13.470087 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:14.119697 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:13:14.119665 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:13:15.445503 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:15.445468 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:16.195094 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:16.195063 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:17.076656 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:17.076599 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:18.201832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:18.201793 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:18.324084 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:18.324048 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:18.514322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:18.514238 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj"] Oct 30 17:13:18.514556 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:18.514527 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" podUID="ed723b1e-dfc2-441a-816b-1308b738215e" containerName="istio-proxy" containerID="cri-o://d61ca463996b4678c48ac6112d328f85565fc581d2ab72e3a2356a383c2ff35b" gracePeriod=30 Oct 30 17:13:18.548637 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:18.548599 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:18.823263 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:18.821496 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:19.009227 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.009195 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:19.164227 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.164153 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv"] Oct 30 17:13:19.170909 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.170885 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv" Oct 30 17:13:19.188321 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.187431 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:13:19.188321 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.187533 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:13:19.189554 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.189530 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv"] Oct 30 17:13:19.201293 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.201264 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:13:19.201455 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.201329 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" podUID="ed723b1e-dfc2-441a-816b-1308b738215e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:13:19.213700 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.213666 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv\" (UID: \"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv" Oct 30 17:13:19.214112 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.213739 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv\" (UID: \"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv" Oct 30 17:13:19.214112 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.213816 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnhp7\" (UniqueName: \"kubernetes.io/projected/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-kube-api-access-fnhp7\") pod \"maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv\" (UID: \"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv" Oct 30 17:13:19.246370 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.246342 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:13:19.246370 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.246367 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:13:19.315845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.315816 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv\" (UID: \"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv" Oct 30 17:13:19.316037 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.315862 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv\" (UID: \"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv" Oct 30 17:13:19.316037 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.315889 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnhp7\" (UniqueName: \"kubernetes.io/projected/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-kube-api-access-fnhp7\") pod \"maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv\" (UID: \"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv" Oct 30 17:13:19.316462 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.316441 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv\" (UID: \"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv" Oct 30 17:13:19.318359 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.318335 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv\" (UID: \"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv" Oct 30 17:13:19.340695 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.340664 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnhp7\" (UniqueName: \"kubernetes.io/projected/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-kube-api-access-fnhp7\") pod \"maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv\" (UID: \"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv" Oct 30 17:13:19.489935 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.489849 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv" Oct 30 17:13:19.793200 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:13:19.793165 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6af0aa_cdab_4e6b_bbc3_526caee0b65d.slice/crio-5e79ab165976c5f35d014cfd6ecc01b6bb404651f51fc4043ac3f97fb09e851f WatchSource:0}: Error finding container 5e79ab165976c5f35d014cfd6ecc01b6bb404651f51fc4043ac3f97fb09e851f: Status 404 returned error can't find the container with id 5e79ab165976c5f35d014cfd6ecc01b6bb404651f51fc4043ac3f97fb09e851f Oct 30 17:13:19.795411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:19.795389 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv"] Oct 30 17:13:20.784972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:20.784925 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv" event={"ID":"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d","Type":"ContainerStarted","Data":"f0f41d55174bc283484e836248bacca23701ef53eb7b40105ca811094102e8de"} Oct 30 17:13:20.785508 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:20.785032 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv" event={"ID":"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d","Type":"ContainerStarted","Data":"5e79ab165976c5f35d014cfd6ecc01b6bb404651f51fc4043ac3f97fb09e851f"} Oct 30 17:13:20.829580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:20.829517 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv" podStartSLOduration=1.829502528 podStartE2EDuration="1.829502528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:20.828296072 +0000 UTC m=+3114.538750816" watchObservedRunningTime="2026-10-30 17:13:20.829502528 +0000 UTC m=+3114.539957266" Oct 30 17:13:20.923982 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:20.923937 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:21.275721 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:21.275691 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx_31841021-3c67-427d-9b3b-8ebfdcd657b6/payload-processing/0.log" Oct 30 17:13:21.471934 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:21.471906 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx_31841021-3c67-427d-9b3b-8ebfdcd657b6/payload-processing/0.log" Oct 30 17:13:21.795857 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:21.795818 6202 generic.go:356] "Generic (PLEG): container finished" podID="5e6af0aa-cdab-4e6b-bbc3-526caee0b65d" containerID="f0f41d55174bc283484e836248bacca23701ef53eb7b40105ca811094102e8de" exitCode=0 Oct 30 17:13:21.796343 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:21.795851 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv" event={"ID":"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d","Type":"ContainerDied","Data":"f0f41d55174bc283484e836248bacca23701ef53eb7b40105ca811094102e8de"} Oct 30 17:13:22.007026 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:22.006991 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv"] Oct 30 17:13:22.011915 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:22.011892 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv" Oct 30 17:13:22.031234 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:22.031204 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv"] Oct 30 17:13:22.164972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:22.164928 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3537b1e-c21c-47d2-88de-821acfb3f34a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv\" (UID: \"c3537b1e-c21c-47d2-88de-821acfb3f34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv" Oct 30 17:13:22.165188 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:22.165139 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvbb6\" (UniqueName: \"kubernetes.io/projected/c3537b1e-c21c-47d2-88de-821acfb3f34a-kube-api-access-vvbb6\") pod \"maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv\" (UID: \"c3537b1e-c21c-47d2-88de-821acfb3f34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv" Oct 30 17:13:22.165275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:22.165250 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3537b1e-c21c-47d2-88de-821acfb3f34a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv\" (UID: \"c3537b1e-c21c-47d2-88de-821acfb3f34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv" Oct 30 17:13:22.267144 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:22.267105 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvbb6\" (UniqueName: \"kubernetes.io/projected/c3537b1e-c21c-47d2-88de-821acfb3f34a-kube-api-access-vvbb6\") pod \"maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv\" (UID: \"c3537b1e-c21c-47d2-88de-821acfb3f34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv" Oct 30 17:13:22.267323 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:22.267198 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3537b1e-c21c-47d2-88de-821acfb3f34a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv\" (UID: \"c3537b1e-c21c-47d2-88de-821acfb3f34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv" Oct 30 17:13:22.267411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:22.267382 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3537b1e-c21c-47d2-88de-821acfb3f34a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv\" (UID: \"c3537b1e-c21c-47d2-88de-821acfb3f34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv" Oct 30 17:13:22.268185 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:22.268155 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3537b1e-c21c-47d2-88de-821acfb3f34a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv\" (UID: \"c3537b1e-c21c-47d2-88de-821acfb3f34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv" Oct 30 17:13:22.269845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:22.269820 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3537b1e-c21c-47d2-88de-821acfb3f34a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv\" (UID: \"c3537b1e-c21c-47d2-88de-821acfb3f34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv" Oct 30 17:13:22.291939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:22.291908 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvbb6\" (UniqueName: \"kubernetes.io/projected/c3537b1e-c21c-47d2-88de-821acfb3f34a-kube-api-access-vvbb6\") pod \"maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv\" (UID: \"c3537b1e-c21c-47d2-88de-821acfb3f34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv" Oct 30 17:13:22.326777 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:22.326743 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv" Oct 30 17:13:22.633964 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:13:22.633923 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3537b1e_c21c_47d2_88de_821acfb3f34a.slice/crio-aff6ec391a344491b7bf133e1b1001cdb9e1e0e3285520fe6f81f3cf2b662ac0 WatchSource:0}: Error finding container aff6ec391a344491b7bf133e1b1001cdb9e1e0e3285520fe6f81f3cf2b662ac0: Status 404 returned error can't find the container with id aff6ec391a344491b7bf133e1b1001cdb9e1e0e3285520fe6f81f3cf2b662ac0 Oct 30 17:13:22.637030 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:22.636999 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv"] Oct 30 17:13:22.650185 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:22.650152 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:22.806464 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:22.806424 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv" event={"ID":"c3537b1e-c21c-47d2-88de-821acfb3f34a","Type":"ContainerStarted","Data":"aff6ec391a344491b7bf133e1b1001cdb9e1e0e3285520fe6f81f3cf2b662ac0"} Oct 30 17:13:22.988495 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:22.988467 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv" Oct 30 17:13:23.083756 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.082445 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-maas-api-xks-ca\") pod \"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d\" (UID: \"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d\") " Oct 30 17:13:23.083756 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.082630 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-kube-api-access-fnhp7\" (UniqueName: \"kubernetes.io/projected/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-kube-api-access-fnhp7\") pod \"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d\" (UID: \"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d\") " Oct 30 17:13:23.083756 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.082830 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-maas-api-openshift-ca\") pod \"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d\" (UID: \"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d\") " Oct 30 17:13:23.084607 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.084324 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-maas-api-openshift-ca" pod "5e6af0aa-cdab-4e6b-bbc3-526caee0b65d" (UID: "5e6af0aa-cdab-4e6b-bbc3-526caee0b65d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:23.085356 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.085323 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-maas-api-xks-ca" pod "5e6af0aa-cdab-4e6b-bbc3-526caee0b65d" (UID: "5e6af0aa-cdab-4e6b-bbc3-526caee0b65d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:23.088432 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.088406 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-kube-api-access-fnhp7" pod "5e6af0aa-cdab-4e6b-bbc3-526caee0b65d" (UID: "5e6af0aa-cdab-4e6b-bbc3-526caee0b65d"). InnerVolumeSpecName "kube-api-access-fnhp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:23.185093 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.185005 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:23.185093 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.185037 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnhp7\" (UniqueName: \"kubernetes.io/projected/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-kube-api-access-fnhp7\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:23.185093 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.185047 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:23.817846 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.817801 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv" event={"ID":"5e6af0aa-cdab-4e6b-bbc3-526caee0b65d","Type":"ContainerDied","Data":"5e79ab165976c5f35d014cfd6ecc01b6bb404651f51fc4043ac3f97fb09e851f"} Oct 30 17:13:23.818300 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.817864 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e79ab165976c5f35d014cfd6ecc01b6bb404651f51fc4043ac3f97fb09e851f" Oct 30 17:13:23.818300 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.817817 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv" Oct 30 17:13:23.819650 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.819623 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv" event={"ID":"c3537b1e-c21c-47d2-88de-821acfb3f34a","Type":"ContainerStarted","Data":"ec093f90000083dcaf68d34f7c4270e6b43175de8a1b3bb20dd54d399bb6b0b7"} Oct 30 17:13:23.821367 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.821344 6202 generic.go:356] "Generic (PLEG): container finished" podID="ed723b1e-dfc2-441a-816b-1308b738215e" containerID="d61ca463996b4678c48ac6112d328f85565fc581d2ab72e3a2356a383c2ff35b" exitCode=0 Oct 30 17:13:23.821525 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.821392 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" event={"ID":"ed723b1e-dfc2-441a-816b-1308b738215e","Type":"ContainerDied","Data":"d61ca463996b4678c48ac6112d328f85565fc581d2ab72e3a2356a383c2ff35b"} Oct 30 17:13:23.821525 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.821420 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" event={"ID":"ed723b1e-dfc2-441a-816b-1308b738215e","Type":"ContainerDied","Data":"d68b0ca4ccd766215916bb0afd806643e8d1b8b8f41e3f7a1dc593333abca43e"} Oct 30 17:13:23.821525 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.821435 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d68b0ca4ccd766215916bb0afd806643e8d1b8b8f41e3f7a1dc593333abca43e" Oct 30 17:13:23.829162 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.829143 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 17:13:23.869811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.866740 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv" podStartSLOduration=2.866720724 podStartE2EDuration="2.866720724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:23.861516656 +0000 UTC m=+3117.571971403" watchObservedRunningTime="2026-10-30 17:13:23.866720724 +0000 UTC m=+3117.577175469" Oct 30 17:13:23.897322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.897283 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed723b1e-dfc2-441a-816b-1308b738215e-istio-token\" (UniqueName: \"kubernetes.io/projected/ed723b1e-dfc2-441a-816b-1308b738215e-istio-token\") pod \"ed723b1e-dfc2-441a-816b-1308b738215e\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " Oct 30 17:13:23.897690 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.897391 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed723b1e-dfc2-441a-816b-1308b738215e-kube-api-access-6njgx\" (UniqueName: \"kubernetes.io/projected/ed723b1e-dfc2-441a-816b-1308b738215e-kube-api-access-6njgx\") pod \"ed723b1e-dfc2-441a-816b-1308b738215e\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " Oct 30 17:13:23.897690 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.897586 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-istio-envoy\") pod \"ed723b1e-dfc2-441a-816b-1308b738215e\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " Oct 30 17:13:23.898006 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.897969 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-workload-socket\") pod \"ed723b1e-dfc2-441a-816b-1308b738215e\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " Oct 30 17:13:23.898263 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.898230 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-workload-socket" pod "ed723b1e-dfc2-441a-816b-1308b738215e" (UID: "ed723b1e-dfc2-441a-816b-1308b738215e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:23.898368 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.898349 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed723b1e-dfc2-441a-816b-1308b738215e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed723b1e-dfc2-441a-816b-1308b738215e-istiod-ca-cert\") pod \"ed723b1e-dfc2-441a-816b-1308b738215e\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " Oct 30 17:13:23.898442 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.898427 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-credential-socket\") pod \"ed723b1e-dfc2-441a-816b-1308b738215e\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " Oct 30 17:13:23.898509 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.898492 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-istio-data\") pod \"ed723b1e-dfc2-441a-816b-1308b738215e\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " Oct 30 17:13:23.898576 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.898553 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed723b1e-dfc2-441a-816b-1308b738215e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed723b1e-dfc2-441a-816b-1308b738215e-istio-podinfo\") pod \"ed723b1e-dfc2-441a-816b-1308b738215e\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " Oct 30 17:13:23.898640 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.898586 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-workload-certs\") pod \"ed723b1e-dfc2-441a-816b-1308b738215e\" (UID: \"ed723b1e-dfc2-441a-816b-1308b738215e\") " Oct 30 17:13:23.899460 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.899435 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:23.900281 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.900251 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-workload-certs" pod "ed723b1e-dfc2-441a-816b-1308b738215e" (UID: "ed723b1e-dfc2-441a-816b-1308b738215e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:23.900512 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.900484 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed723b1e-dfc2-441a-816b-1308b738215e-istiod-ca-cert" pod "ed723b1e-dfc2-441a-816b-1308b738215e" (UID: "ed723b1e-dfc2-441a-816b-1308b738215e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:23.900586 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.900182 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-istio-data" pod "ed723b1e-dfc2-441a-816b-1308b738215e" (UID: "ed723b1e-dfc2-441a-816b-1308b738215e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:23.901270 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.901197 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed723b1e-dfc2-441a-816b-1308b738215e-kube-api-access-6njgx" pod "ed723b1e-dfc2-441a-816b-1308b738215e" (UID: "ed723b1e-dfc2-441a-816b-1308b738215e"). InnerVolumeSpecName "kube-api-access-6njgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:23.901270 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.901241 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed723b1e-dfc2-441a-816b-1308b738215e-istio-token" pod "ed723b1e-dfc2-441a-816b-1308b738215e" (UID: "ed723b1e-dfc2-441a-816b-1308b738215e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:23.901605 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.901578 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-credential-socket" pod "ed723b1e-dfc2-441a-816b-1308b738215e" (UID: "ed723b1e-dfc2-441a-816b-1308b738215e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:23.902670 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.902645 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-istio-envoy" pod "ed723b1e-dfc2-441a-816b-1308b738215e" (UID: "ed723b1e-dfc2-441a-816b-1308b738215e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:23.904131 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:23.904103 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed723b1e-dfc2-441a-816b-1308b738215e-istio-podinfo" pod "ed723b1e-dfc2-441a-816b-1308b738215e" (UID: "ed723b1e-dfc2-441a-816b-1308b738215e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:13:24.001359 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:24.001322 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6njgx\" (UniqueName: \"kubernetes.io/projected/ed723b1e-dfc2-441a-816b-1308b738215e-kube-api-access-6njgx\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.001359 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:24.001355 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.001359 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:24.001365 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed723b1e-dfc2-441a-816b-1308b738215e-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.001623 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:24.001373 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.001623 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:24.001381 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.001623 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:24.001389 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed723b1e-dfc2-441a-816b-1308b738215e-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.001623 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:24.001396 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed723b1e-dfc2-441a-816b-1308b738215e-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.001623 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:24.001404 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed723b1e-dfc2-441a-816b-1308b738215e-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.063537 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:24.063442 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:24.831671 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:24.831618 6202 generic.go:356] "Generic (PLEG): container finished" podID="c3537b1e-c21c-47d2-88de-821acfb3f34a" containerID="ec093f90000083dcaf68d34f7c4270e6b43175de8a1b3bb20dd54d399bb6b0b7" exitCode=0 Oct 30 17:13:24.832322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:24.831701 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv" event={"ID":"c3537b1e-c21c-47d2-88de-821acfb3f34a","Type":"ContainerDied","Data":"ec093f90000083dcaf68d34f7c4270e6b43175de8a1b3bb20dd54d399bb6b0b7"} Oct 30 17:13:24.832322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:24.831956 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj" Oct 30 17:13:24.916074 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:24.916039 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj"] Oct 30 17:13:24.920129 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:24.920103 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-gdxhj"] Oct 30 17:13:25.238911 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:25.238804 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:26.020180 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:26.020149 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv" Oct 30 17:13:26.099102 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:26.098994 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7"] Oct 30 17:13:26.099288 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:26.099241 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" podUID="d3e18c75-a227-4a6b-a998-3d3982ad16b2" containerName="istio-proxy" containerID="cri-o://cf867d002a6d035c46d839f83ebc7ccf5e160de175fae6e6a25066bb2e35a05a" gracePeriod=30 Oct 30 17:13:26.135155 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:26.135114 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3537b1e-c21c-47d2-88de-821acfb3f34a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3537b1e-c21c-47d2-88de-821acfb3f34a-maas-api-xks-ca\") pod \"c3537b1e-c21c-47d2-88de-821acfb3f34a\" (UID: \"c3537b1e-c21c-47d2-88de-821acfb3f34a\") " Oct 30 17:13:26.135339 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:26.135250 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3537b1e-c21c-47d2-88de-821acfb3f34a-kube-api-access-vvbb6\" (UniqueName: \"kubernetes.io/projected/c3537b1e-c21c-47d2-88de-821acfb3f34a-kube-api-access-vvbb6\") pod \"c3537b1e-c21c-47d2-88de-821acfb3f34a\" (UID: \"c3537b1e-c21c-47d2-88de-821acfb3f34a\") " Oct 30 17:13:26.135339 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:26.135278 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3537b1e-c21c-47d2-88de-821acfb3f34a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3537b1e-c21c-47d2-88de-821acfb3f34a-maas-api-openshift-ca\") pod \"c3537b1e-c21c-47d2-88de-821acfb3f34a\" (UID: \"c3537b1e-c21c-47d2-88de-821acfb3f34a\") " Oct 30 17:13:26.135801 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:26.135730 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3537b1e-c21c-47d2-88de-821acfb3f34a-maas-api-openshift-ca" pod "c3537b1e-c21c-47d2-88de-821acfb3f34a" (UID: "c3537b1e-c21c-47d2-88de-821acfb3f34a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:26.136168 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:26.136143 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3537b1e-c21c-47d2-88de-821acfb3f34a-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:26.137529 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:26.137504 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3537b1e-c21c-47d2-88de-821acfb3f34a-maas-api-xks-ca" pod "c3537b1e-c21c-47d2-88de-821acfb3f34a" (UID: "c3537b1e-c21c-47d2-88de-821acfb3f34a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:26.138077 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:26.138050 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3537b1e-c21c-47d2-88de-821acfb3f34a-kube-api-access-vvbb6" pod "c3537b1e-c21c-47d2-88de-821acfb3f34a" (UID: "c3537b1e-c21c-47d2-88de-821acfb3f34a"). InnerVolumeSpecName "kube-api-access-vvbb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:26.238207 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:26.238172 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3537b1e-c21c-47d2-88de-821acfb3f34a-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:26.238207 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:26.238203 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvbb6\" (UniqueName: \"kubernetes.io/projected/c3537b1e-c21c-47d2-88de-821acfb3f34a-kube-api-access-vvbb6\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:26.413149 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:26.413069 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed723b1e-dfc2-441a-816b-1308b738215e" path="/var/lib/kubelet/pods/ed723b1e-dfc2-441a-816b-1308b738215e/volumes" Oct 30 17:13:26.625138 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:26.625102 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:26.846791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:26.845587 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:26.878664 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:26.878638 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv" Oct 30 17:13:26.878866 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:26.878714 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv" event={"ID":"c3537b1e-c21c-47d2-88de-821acfb3f34a","Type":"ContainerDied","Data":"aff6ec391a344491b7bf133e1b1001cdb9e1e0e3285520fe6f81f3cf2b662ac0"} Oct 30 17:13:26.878866 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:26.878794 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aff6ec391a344491b7bf133e1b1001cdb9e1e0e3285520fe6f81f3cf2b662ac0" Oct 30 17:13:27.086791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.070379 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks"] Oct 30 17:13:27.086791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.071578 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5e6af0aa-cdab-4e6b-bbc3-526caee0b65d" containerName="revoke-keys" Oct 30 17:13:27.086791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.071602 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6af0aa-cdab-4e6b-bbc3-526caee0b65d" containerName="revoke-keys" Oct 30 17:13:27.086791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.071661 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c3537b1e-c21c-47d2-88de-821acfb3f34a" containerName="revoke-keys" Oct 30 17:13:27.086791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.071669 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3537b1e-c21c-47d2-88de-821acfb3f34a" containerName="revoke-keys" Oct 30 17:13:27.086791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.071703 6202 cpu_manager.go:414] "containerMap: removing container" podUID="ed723b1e-dfc2-441a-816b-1308b738215e" containerName="istio-proxy" Oct 30 17:13:27.086791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.071711 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed723b1e-dfc2-441a-816b-1308b738215e" containerName="istio-proxy" Oct 30 17:13:27.086791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.071915 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6af0aa-cdab-4e6b-bbc3-526caee0b65d" containerName="revoke-keys" Oct 30 17:13:27.086791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.071940 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3537b1e-c21c-47d2-88de-821acfb3f34a" containerName="revoke-keys" Oct 30 17:13:27.086791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.071956 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed723b1e-dfc2-441a-816b-1308b738215e" containerName="istio-proxy" Oct 30 17:13:27.086791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.085066 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks" Oct 30 17:13:27.107213 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.104214 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4def7e\"" Oct 30 17:13:27.107213 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.104361 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4def7e-dockercfg-cfbgg\"" Oct 30 17:13:27.170997 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.170564 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df7a2ad9-d72a-428a-b638-eac0d32e7751-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks\" (UID: \"df7a2ad9-d72a-428a-b638-eac0d32e7751\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks" Oct 30 17:13:27.170997 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.170755 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klhl8\" (UniqueName: \"kubernetes.io/projected/df7a2ad9-d72a-428a-b638-eac0d32e7751-kube-api-access-klhl8\") pod \"payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks\" (UID: \"df7a2ad9-d72a-428a-b638-eac0d32e7751\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks" Oct 30 17:13:27.238897 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.232944 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks"] Oct 30 17:13:27.271929 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.271858 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df7a2ad9-d72a-428a-b638-eac0d32e7751-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks\" (UID: \"df7a2ad9-d72a-428a-b638-eac0d32e7751\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks" Oct 30 17:13:27.272151 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.272110 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klhl8\" (UniqueName: \"kubernetes.io/projected/df7a2ad9-d72a-428a-b638-eac0d32e7751-kube-api-access-klhl8\") pod \"payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks\" (UID: \"df7a2ad9-d72a-428a-b638-eac0d32e7751\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks" Oct 30 17:13:27.272693 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.272671 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df7a2ad9-d72a-428a-b638-eac0d32e7751-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks\" (UID: \"df7a2ad9-d72a-428a-b638-eac0d32e7751\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks" Oct 30 17:13:27.296690 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.296653 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klhl8\" (UniqueName: \"kubernetes.io/projected/df7a2ad9-d72a-428a-b638-eac0d32e7751-kube-api-access-klhl8\") pod \"payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks\" (UID: \"df7a2ad9-d72a-428a-b638-eac0d32e7751\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks" Oct 30 17:13:27.425231 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.425153 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks" Oct 30 17:13:27.605090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.605057 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:27.707569 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.707486 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr"] Oct 30 17:13:27.707742 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.707711 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr" podUID="a1bab548-0f80-4d1b-ba3a-d63c8fff6eee" containerName="payload-processing" containerID="cri-o://ff7e6986241e394280180c8179aff6285ca1247243f0362bf3b2908a13635b58" gracePeriod=30 Oct 30 17:13:27.739824 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.728358 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv"] Oct 30 17:13:27.739824 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.728591 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv" podUID="81111551-33d9-4996-bc84-d7b27e6ab40c" containerName="payload-pre-processing" containerID="cri-o://32a0721c8542123e42a450bffc478c8bb61af4a1b680bb5d27efb642caae5302" gracePeriod=30 Oct 30 17:13:27.742462 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.742425 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks"] Oct 30 17:13:27.789517 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.789257 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:27.912530 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.912316 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:27.919142 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.918911 6202 generic.go:356] "Generic (PLEG): container finished" podID="a1bab548-0f80-4d1b-ba3a-d63c8fff6eee" containerID="ff7e6986241e394280180c8179aff6285ca1247243f0362bf3b2908a13635b58" exitCode=0 Oct 30 17:13:27.919142 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.919035 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr" event={"ID":"a1bab548-0f80-4d1b-ba3a-d63c8fff6eee","Type":"ContainerDied","Data":"ff7e6986241e394280180c8179aff6285ca1247243f0362bf3b2908a13635b58"} Oct 30 17:13:27.921953 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.921644 6202 generic.go:356] "Generic (PLEG): container finished" podID="81111551-33d9-4996-bc84-d7b27e6ab40c" containerID="32a0721c8542123e42a450bffc478c8bb61af4a1b680bb5d27efb642caae5302" exitCode=0 Oct 30 17:13:27.921953 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.921882 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv" event={"ID":"81111551-33d9-4996-bc84-d7b27e6ab40c","Type":"ContainerDied","Data":"32a0721c8542123e42a450bffc478c8bb61af4a1b680bb5d27efb642caae5302"} Oct 30 17:13:27.923885 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:27.923815 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks" event={"ID":"df7a2ad9-d72a-428a-b638-eac0d32e7751","Type":"ContainerStarted","Data":"15ac9a8162ca08fb8a775112999dc1f0f4a36b2263129ca07ede103e273f6986"} Oct 30 17:13:28.055495 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.055466 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr" Oct 30 17:13:28.086799 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.085449 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1bab548-0f80-4d1b-ba3a-d63c8fff6eee-kube-api-access-h26vc\" (UniqueName: \"kubernetes.io/projected/a1bab548-0f80-4d1b-ba3a-d63c8fff6eee-kube-api-access-h26vc\") pod \"a1bab548-0f80-4d1b-ba3a-d63c8fff6eee\" (UID: \"a1bab548-0f80-4d1b-ba3a-d63c8fff6eee\") " Oct 30 17:13:28.086799 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.085575 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1bab548-0f80-4d1b-ba3a-d63c8fff6eee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1bab548-0f80-4d1b-ba3a-d63c8fff6eee-plugins-config-volume\") pod \"a1bab548-0f80-4d1b-ba3a-d63c8fff6eee\" (UID: \"a1bab548-0f80-4d1b-ba3a-d63c8fff6eee\") " Oct 30 17:13:28.094798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.088988 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1bab548-0f80-4d1b-ba3a-d63c8fff6eee-plugins-config-volume" pod "a1bab548-0f80-4d1b-ba3a-d63c8fff6eee" (UID: "a1bab548-0f80-4d1b-ba3a-d63c8fff6eee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:28.163806 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.148634 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:28.206276 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.205873 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1bab548-0f80-4d1b-ba3a-d63c8fff6eee-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:28.287836 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.282002 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv" Oct 30 17:13:28.349356 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.349291 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bab548-0f80-4d1b-ba3a-d63c8fff6eee-kube-api-access-h26vc" pod "a1bab548-0f80-4d1b-ba3a-d63c8fff6eee" (UID: "a1bab548-0f80-4d1b-ba3a-d63c8fff6eee"). InnerVolumeSpecName "kube-api-access-h26vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:28.357872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.355790 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:28.411188 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.405818 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk"] Oct 30 17:13:28.411188 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.407476 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-330282-nf5fk"] Oct 30 17:13:28.421792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.414066 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81111551-33d9-4996-bc84-d7b27e6ab40c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81111551-33d9-4996-bc84-d7b27e6ab40c-plugins-config-volume\") pod \"81111551-33d9-4996-bc84-d7b27e6ab40c\" (UID: \"81111551-33d9-4996-bc84-d7b27e6ab40c\") " Oct 30 17:13:28.421792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.414192 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81111551-33d9-4996-bc84-d7b27e6ab40c-kube-api-access-dhxg5\" (UniqueName: \"kubernetes.io/projected/81111551-33d9-4996-bc84-d7b27e6ab40c-kube-api-access-dhxg5\") pod \"81111551-33d9-4996-bc84-d7b27e6ab40c\" (UID: \"81111551-33d9-4996-bc84-d7b27e6ab40c\") " Oct 30 17:13:28.421792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.414983 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h26vc\" (UniqueName: \"kubernetes.io/projected/a1bab548-0f80-4d1b-ba3a-d63c8fff6eee-kube-api-access-h26vc\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:28.421792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.416731 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81111551-33d9-4996-bc84-d7b27e6ab40c-plugins-config-volume" pod "81111551-33d9-4996-bc84-d7b27e6ab40c" (UID: "81111551-33d9-4996-bc84-d7b27e6ab40c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:28.450063 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.445166 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bded893-57fb-4d38-ba0c-fa06f1ace2a2" path="/var/lib/kubelet/pods/0bded893-57fb-4d38-ba0c-fa06f1ace2a2/volumes" Oct 30 17:13:28.456097 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.455954 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81111551-33d9-4996-bc84-d7b27e6ab40c-kube-api-access-dhxg5" pod "81111551-33d9-4996-bc84-d7b27e6ab40c" (UID: "81111551-33d9-4996-bc84-d7b27e6ab40c"). InnerVolumeSpecName "kube-api-access-dhxg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:28.483732 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.483683 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:28.516701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.516661 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81111551-33d9-4996-bc84-d7b27e6ab40c-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:28.516701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.516694 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhxg5\" (UniqueName: \"kubernetes.io/projected/81111551-33d9-4996-bc84-d7b27e6ab40c-kube-api-access-dhxg5\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:28.933673 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.933642 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr" Oct 30 17:13:28.933673 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.933664 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr" event={"ID":"a1bab548-0f80-4d1b-ba3a-d63c8fff6eee","Type":"ContainerDied","Data":"7d8f1fb4aff7ac612fa4acfcfb17bec8e3608dc93e96f5c615760e6d71f0fedd"} Oct 30 17:13:28.933968 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.933726 6202 scope.go:122] "RemoveContainer" containerID="ff7e6986241e394280180c8179aff6285ca1247243f0362bf3b2908a13635b58" Oct 30 17:13:28.935838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.935542 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv" Oct 30 17:13:28.935838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.935675 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv" event={"ID":"81111551-33d9-4996-bc84-d7b27e6ab40c","Type":"ContainerDied","Data":"da7ded078c64b0e9c01e12258698eec8c7af4e6d3b52d98bb199e5a33beea45a"} Oct 30 17:13:28.938057 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.938032 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks" event={"ID":"df7a2ad9-d72a-428a-b638-eac0d32e7751","Type":"ContainerStarted","Data":"b3ec86806cd8459ec395bff43395dc7d147f8dd8574ffe5bcf7593a922d13fa9"} Oct 30 17:13:28.938201 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.938186 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks" Oct 30 17:13:28.947427 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.947404 6202 scope.go:122] "RemoveContainer" containerID="32a0721c8542123e42a450bffc478c8bb61af4a1b680bb5d27efb642caae5302" Oct 30 17:13:28.993346 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:28.993309 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv"] Oct 30 17:13:29.006568 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:29.006538 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-330282-5fc46c6f9f-9wmkv"] Oct 30 17:13:29.039424 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:29.039364 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks" podStartSLOduration=2.039346533 podStartE2EDuration="2.039346533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:29.032847092 +0000 UTC m=+3122.743301836" watchObservedRunningTime="2026-10-30 17:13:29.039346533 +0000 UTC m=+3122.749801278" Oct 30 17:13:29.073228 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:29.073188 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr"] Oct 30 17:13:29.079606 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:29.079570 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-330282-85ff4bc8f4-jbzpr"] Oct 30 17:13:29.625796 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:29.625725 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m"] Oct 30 17:13:29.659057 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:29.659028 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6"] Oct 30 17:13:29.758953 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:29.758913 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:29.871954 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:29.871896 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:30.402950 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:30.402915 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:30.415319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:30.415288 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81111551-33d9-4996-bc84-d7b27e6ab40c" path="/var/lib/kubelet/pods/81111551-33d9-4996-bc84-d7b27e6ab40c/volumes" Oct 30 17:13:30.415732 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:30.415718 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1bab548-0f80-4d1b-ba3a-d63c8fff6eee" path="/var/lib/kubelet/pods/a1bab548-0f80-4d1b-ba3a-d63c8fff6eee/volumes" Oct 30 17:13:31.410871 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.410837 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:13:31.502435 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.502378 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-workload-socket\") pod \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " Oct 30 17:13:31.502649 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.502456 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-workload-certs\") pod \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " Oct 30 17:13:31.502649 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.502536 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e18c75-a227-4a6b-a998-3d3982ad16b2-kube-api-access-lhgwt\" (UniqueName: \"kubernetes.io/projected/d3e18c75-a227-4a6b-a998-3d3982ad16b2-kube-api-access-lhgwt\") pod \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " Oct 30 17:13:31.502649 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.502629 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istiod-ca-cert\") pod \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " Oct 30 17:13:31.503081 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.502717 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-token\" (UniqueName: \"kubernetes.io/projected/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-token\") pod \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " Oct 30 17:13:31.503081 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.502818 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-envoy\") pod \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " Oct 30 17:13:31.503081 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.502905 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-credential-socket\") pod \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " Oct 30 17:13:31.503081 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.502979 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-podinfo\") pod \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " Oct 30 17:13:31.503343 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.503083 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-data\") pod \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\" (UID: \"d3e18c75-a227-4a6b-a998-3d3982ad16b2\") " Oct 30 17:13:31.503810 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.503758 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-credential-socket" pod "d3e18c75-a227-4a6b-a998-3d3982ad16b2" (UID: "d3e18c75-a227-4a6b-a998-3d3982ad16b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:31.503984 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.503956 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-workload-certs" pod "d3e18c75-a227-4a6b-a998-3d3982ad16b2" (UID: "d3e18c75-a227-4a6b-a998-3d3982ad16b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:31.504096 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.504061 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-data" pod "d3e18c75-a227-4a6b-a998-3d3982ad16b2" (UID: "d3e18c75-a227-4a6b-a998-3d3982ad16b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:31.504173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.504145 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-workload-socket" pod "d3e18c75-a227-4a6b-a998-3d3982ad16b2" (UID: "d3e18c75-a227-4a6b-a998-3d3982ad16b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:31.504259 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.504233 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istiod-ca-cert" pod "d3e18c75-a227-4a6b-a998-3d3982ad16b2" (UID: "d3e18c75-a227-4a6b-a998-3d3982ad16b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:31.509809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.506721 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-token" pod "d3e18c75-a227-4a6b-a998-3d3982ad16b2" (UID: "d3e18c75-a227-4a6b-a998-3d3982ad16b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:31.509809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.506722 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-podinfo" pod "d3e18c75-a227-4a6b-a998-3d3982ad16b2" (UID: "d3e18c75-a227-4a6b-a998-3d3982ad16b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:13:31.509809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.506851 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-envoy" pod "d3e18c75-a227-4a6b-a998-3d3982ad16b2" (UID: "d3e18c75-a227-4a6b-a998-3d3982ad16b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:31.509809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.506958 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e18c75-a227-4a6b-a998-3d3982ad16b2-kube-api-access-lhgwt" pod "d3e18c75-a227-4a6b-a998-3d3982ad16b2" (UID: "d3e18c75-a227-4a6b-a998-3d3982ad16b2"). InnerVolumeSpecName "kube-api-access-lhgwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:31.604986 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.604947 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:31.604986 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.604980 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:31.604986 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.604992 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:31.605240 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.605001 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:31.605240 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.605009 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:31.605240 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.605017 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhgwt\" (UniqueName: \"kubernetes.io/projected/d3e18c75-a227-4a6b-a998-3d3982ad16b2-kube-api-access-lhgwt\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:31.605240 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.605027 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:31.605240 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.605035 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:31.605240 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.605043 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e18c75-a227-4a6b-a998-3d3982ad16b2-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:31.971455 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.971424 6202 generic.go:356] "Generic (PLEG): container finished" podID="d3e18c75-a227-4a6b-a998-3d3982ad16b2" containerID="cf867d002a6d035c46d839f83ebc7ccf5e160de175fae6e6a25066bb2e35a05a" exitCode=0 Oct 30 17:13:31.971646 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.971482 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" event={"ID":"d3e18c75-a227-4a6b-a998-3d3982ad16b2","Type":"ContainerDied","Data":"cf867d002a6d035c46d839f83ebc7ccf5e160de175fae6e6a25066bb2e35a05a"} Oct 30 17:13:31.971646 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.971489 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" Oct 30 17:13:31.971646 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.971542 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7" event={"ID":"d3e18c75-a227-4a6b-a998-3d3982ad16b2","Type":"ContainerDied","Data":"ce51480ce10f733420b4934e6bc76ce6d4e388395476a570a52822a16f3b48d9"} Oct 30 17:13:31.971646 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.971567 6202 scope.go:122] "RemoveContainer" containerID="cf867d002a6d035c46d839f83ebc7ccf5e160de175fae6e6a25066bb2e35a05a" Oct 30 17:13:31.983729 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.983710 6202 scope.go:122] "RemoveContainer" containerID="cf867d002a6d035c46d839f83ebc7ccf5e160de175fae6e6a25066bb2e35a05a" Oct 30 17:13:31.984094 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:13:31.984071 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf867d002a6d035c46d839f83ebc7ccf5e160de175fae6e6a25066bb2e35a05a\": container with ID starting with cf867d002a6d035c46d839f83ebc7ccf5e160de175fae6e6a25066bb2e35a05a not found: ID does not exist" containerID="cf867d002a6d035c46d839f83ebc7ccf5e160de175fae6e6a25066bb2e35a05a" Oct 30 17:13:31.984151 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:31.984102 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf867d002a6d035c46d839f83ebc7ccf5e160de175fae6e6a25066bb2e35a05a"} err="failed to get container status \"cf867d002a6d035c46d839f83ebc7ccf5e160de175fae6e6a25066bb2e35a05a\": rpc error: code = NotFound desc = could not find container \"cf867d002a6d035c46d839f83ebc7ccf5e160de175fae6e6a25066bb2e35a05a\": container with ID starting with cf867d002a6d035c46d839f83ebc7ccf5e160de175fae6e6a25066bb2e35a05a not found: ID does not exist" Oct 30 17:13:32.037126 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.037086 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7"] Oct 30 17:13:32.040858 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.040830 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-pdtl7"] Oct 30 17:13:32.412621 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.412579 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e18c75-a227-4a6b-a998-3d3982ad16b2" path="/var/lib/kubelet/pods/d3e18c75-a227-4a6b-a998-3d3982ad16b2/volumes" Oct 30 17:13:32.490067 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.490038 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8"] Oct 30 17:13:32.490828 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.490813 6202 cpu_manager.go:414] "containerMap: removing container" podUID="81111551-33d9-4996-bc84-d7b27e6ab40c" containerName="payload-pre-processing" Oct 30 17:13:32.490892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.490830 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81111551-33d9-4996-bc84-d7b27e6ab40c" containerName="payload-pre-processing" Oct 30 17:13:32.490892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.490875 6202 cpu_manager.go:414] "containerMap: removing container" podUID="d3e18c75-a227-4a6b-a998-3d3982ad16b2" containerName="istio-proxy" Oct 30 17:13:32.490892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.490880 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e18c75-a227-4a6b-a998-3d3982ad16b2" containerName="istio-proxy" Oct 30 17:13:32.490997 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.490897 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a1bab548-0f80-4d1b-ba3a-d63c8fff6eee" containerName="payload-processing" Oct 30 17:13:32.490997 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.490902 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1bab548-0f80-4d1b-ba3a-d63c8fff6eee" containerName="payload-processing" Oct 30 17:13:32.491065 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.491004 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="81111551-33d9-4996-bc84-d7b27e6ab40c" containerName="payload-pre-processing" Oct 30 17:13:32.491065 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.491020 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e18c75-a227-4a6b-a998-3d3982ad16b2" containerName="istio-proxy" Oct 30 17:13:32.491065 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.491030 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1bab548-0f80-4d1b-ba3a-d63c8fff6eee" containerName="payload-processing" Oct 30 17:13:32.496415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.496398 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8" Oct 30 17:13:32.522561 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.522530 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:13:32.522812 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.522663 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:13:32.522812 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.522739 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:13:32.522956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.522835 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:13:32.527405 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.527377 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8"] Oct 30 17:13:32.617243 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.617199 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63fd6c00-7mfd8\" (UID: \"f03c8fbb-68d9-49ec-8d1f-2e7affec093d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8" Oct 30 17:13:32.617451 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.617257 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxc69\" (UniqueName: \"kubernetes.io/projected/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-kube-api-access-sxc69\") pod \"maas-api-revoke-sub-63fd6c00-7mfd8\" (UID: \"f03c8fbb-68d9-49ec-8d1f-2e7affec093d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8" Oct 30 17:13:32.617554 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.617537 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63fd6c00-7mfd8\" (UID: \"f03c8fbb-68d9-49ec-8d1f-2e7affec093d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8" Oct 30 17:13:32.719604 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.719490 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63fd6c00-7mfd8\" (UID: \"f03c8fbb-68d9-49ec-8d1f-2e7affec093d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8" Oct 30 17:13:32.719899 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.719736 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63fd6c00-7mfd8\" (UID: \"f03c8fbb-68d9-49ec-8d1f-2e7affec093d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8" Oct 30 17:13:32.719899 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.719808 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxc69\" (UniqueName: \"kubernetes.io/projected/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-kube-api-access-sxc69\") pod \"maas-api-revoke-sub-63fd6c00-7mfd8\" (UID: \"f03c8fbb-68d9-49ec-8d1f-2e7affec093d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8" Oct 30 17:13:32.720358 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.720292 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63fd6c00-7mfd8\" (UID: \"f03c8fbb-68d9-49ec-8d1f-2e7affec093d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8" Oct 30 17:13:32.729619 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.729593 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63fd6c00-7mfd8\" (UID: \"f03c8fbb-68d9-49ec-8d1f-2e7affec093d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8" Oct 30 17:13:32.745208 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.745184 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxc69\" (UniqueName: \"kubernetes.io/projected/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-kube-api-access-sxc69\") pod \"maas-api-revoke-sub-63fd6c00-7mfd8\" (UID: \"f03c8fbb-68d9-49ec-8d1f-2e7affec093d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8" Oct 30 17:13:32.781268 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.781237 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:32.811951 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:32.811922 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8" Oct 30 17:13:33.161299 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:13:33.161267 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03c8fbb_68d9_49ec_8d1f_2e7affec093d.slice/crio-23df80f58b592d23cd81a4c8f494ce64a134473555874a716545301397de1e72 WatchSource:0}: Error finding container 23df80f58b592d23cd81a4c8f494ce64a134473555874a716545301397de1e72: Status 404 returned error can't find the container with id 23df80f58b592d23cd81a4c8f494ce64a134473555874a716545301397de1e72 Oct 30 17:13:33.164807 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:33.164740 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8"] Oct 30 17:13:34.000180 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:34.000142 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8" event={"ID":"f03c8fbb-68d9-49ec-8d1f-2e7affec093d","Type":"ContainerStarted","Data":"13e15c67845abf5ace4890ba9f9dc9cc5278afec306dd72e600a07e805f5a5c1"} Oct 30 17:13:34.000609 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:34.000223 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8" event={"ID":"f03c8fbb-68d9-49ec-8d1f-2e7affec093d","Type":"ContainerStarted","Data":"23df80f58b592d23cd81a4c8f494ce64a134473555874a716545301397de1e72"} Oct 30 17:13:34.024269 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:34.024235 6202 scope.go:122] "RemoveContainer" containerID="74fba84a0f56ddab2a8b49b23bdbe8379b4b37f0ca59517373a23c21d1a795f9" Oct 30 17:13:34.036879 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:34.036845 6202 scope.go:122] "RemoveContainer" containerID="2788e43573bdca91bb67fa73866e64780158ff7960de77aada5bc321cc0db56a" Oct 30 17:13:34.043163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:34.043120 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8" podStartSLOduration=2.04310109 podStartE2EDuration="2.04310109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:34.04207967 +0000 UTC m=+3127.752534414" watchObservedRunningTime="2026-10-30 17:13:34.04310109 +0000 UTC m=+3127.753555838" Oct 30 17:13:34.052711 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:34.052685 6202 scope.go:122] "RemoveContainer" containerID="f86aca6678efaeb8feb4bb0123c468d0260294319d089a528cafb3c3aba72f70" Oct 30 17:13:34.063187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:34.063168 6202 scope.go:122] "RemoveContainer" containerID="7737e3c9db9e6c4136fe4fa3ac279eb48ccc30e09896e2862f7d1d9d5496bd6f" Oct 30 17:13:34.073007 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:34.072987 6202 scope.go:122] "RemoveContainer" containerID="d61ca463996b4678c48ac6112d328f85565fc581d2ab72e3a2356a383c2ff35b" Oct 30 17:13:34.087155 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:34.087136 6202 scope.go:122] "RemoveContainer" containerID="b6e844c4fdae53503460b0bbd6b81baa9caee85b8d6dee1d355b75b50f26e4a0" Oct 30 17:13:34.539465 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:34.539429 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:34.631795 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:34.630850 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m" podUID="8e64229e-1a05-4bdd-8ebc-f01a3661584e" containerName="payload-processing" containerID="cri-o://130539f5138f7f496c0fc17cff039960d86e8c2a05ce4f11d318618608864845" gracePeriod=25 Oct 30 17:13:34.659803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:34.659727 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6" podUID="6733b6dd-929a-48c6-a8d8-ada293bc5079" containerName="payload-pre-processing" containerID="cri-o://1b9656704c2b23703d73f46621851b1e2b24bc016e8aa3ffa6c52369d3f8e759" gracePeriod=25 Oct 30 17:13:34.980140 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:34.980101 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m" Oct 30 17:13:35.008590 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.008559 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6" Oct 30 17:13:35.009026 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.009005 6202 generic.go:356] "Generic (PLEG): container finished" podID="8e64229e-1a05-4bdd-8ebc-f01a3661584e" containerID="130539f5138f7f496c0fc17cff039960d86e8c2a05ce4f11d318618608864845" exitCode=0 Oct 30 17:13:35.009078 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.009036 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m" Oct 30 17:13:35.009115 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.009081 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m" event={"ID":"8e64229e-1a05-4bdd-8ebc-f01a3661584e","Type":"ContainerDied","Data":"130539f5138f7f496c0fc17cff039960d86e8c2a05ce4f11d318618608864845"} Oct 30 17:13:35.009163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.009124 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m" event={"ID":"8e64229e-1a05-4bdd-8ebc-f01a3661584e","Type":"ContainerDied","Data":"7f45730e72738319bf55b16d8a0f177acd6039cdf7682c95894caa27dfcf2b80"} Oct 30 17:13:35.009163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.009150 6202 scope.go:122] "RemoveContainer" containerID="130539f5138f7f496c0fc17cff039960d86e8c2a05ce4f11d318618608864845" Oct 30 17:13:35.010882 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.010852 6202 generic.go:356] "Generic (PLEG): container finished" podID="f03c8fbb-68d9-49ec-8d1f-2e7affec093d" containerID="13e15c67845abf5ace4890ba9f9dc9cc5278afec306dd72e600a07e805f5a5c1" exitCode=0 Oct 30 17:13:35.010987 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.010899 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8" event={"ID":"f03c8fbb-68d9-49ec-8d1f-2e7affec093d","Type":"ContainerDied","Data":"13e15c67845abf5ace4890ba9f9dc9cc5278afec306dd72e600a07e805f5a5c1"} Oct 30 17:13:35.012493 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.012474 6202 generic.go:356] "Generic (PLEG): container finished" podID="6733b6dd-929a-48c6-a8d8-ada293bc5079" containerID="1b9656704c2b23703d73f46621851b1e2b24bc016e8aa3ffa6c52369d3f8e759" exitCode=0 Oct 30 17:13:35.012584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.012509 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6" event={"ID":"6733b6dd-929a-48c6-a8d8-ada293bc5079","Type":"ContainerDied","Data":"1b9656704c2b23703d73f46621851b1e2b24bc016e8aa3ffa6c52369d3f8e759"} Oct 30 17:13:35.012584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.012519 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6" Oct 30 17:13:35.012584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.012536 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6" event={"ID":"6733b6dd-929a-48c6-a8d8-ada293bc5079","Type":"ContainerDied","Data":"c234be4059fe1487d97af1fe926364c58f8eb6db3b0705dccdf4c7179985bd30"} Oct 30 17:13:35.020552 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.020527 6202 scope.go:122] "RemoveContainer" containerID="130539f5138f7f496c0fc17cff039960d86e8c2a05ce4f11d318618608864845" Oct 30 17:13:35.020876 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:13:35.020854 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"130539f5138f7f496c0fc17cff039960d86e8c2a05ce4f11d318618608864845\": container with ID starting with 130539f5138f7f496c0fc17cff039960d86e8c2a05ce4f11d318618608864845 not found: ID does not exist" containerID="130539f5138f7f496c0fc17cff039960d86e8c2a05ce4f11d318618608864845" Oct 30 17:13:35.021000 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.020897 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"130539f5138f7f496c0fc17cff039960d86e8c2a05ce4f11d318618608864845"} err="failed to get container status \"130539f5138f7f496c0fc17cff039960d86e8c2a05ce4f11d318618608864845\": rpc error: code = NotFound desc = could not find container \"130539f5138f7f496c0fc17cff039960d86e8c2a05ce4f11d318618608864845\": container with ID starting with 130539f5138f7f496c0fc17cff039960d86e8c2a05ce4f11d318618608864845 not found: ID does not exist" Oct 30 17:13:35.021000 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.020915 6202 scope.go:122] "RemoveContainer" containerID="1b9656704c2b23703d73f46621851b1e2b24bc016e8aa3ffa6c52369d3f8e759" Oct 30 17:13:35.040680 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.040446 6202 scope.go:122] "RemoveContainer" containerID="1b9656704c2b23703d73f46621851b1e2b24bc016e8aa3ffa6c52369d3f8e759" Oct 30 17:13:35.040947 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:13:35.040915 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9656704c2b23703d73f46621851b1e2b24bc016e8aa3ffa6c52369d3f8e759\": container with ID starting with 1b9656704c2b23703d73f46621851b1e2b24bc016e8aa3ffa6c52369d3f8e759 not found: ID does not exist" containerID="1b9656704c2b23703d73f46621851b1e2b24bc016e8aa3ffa6c52369d3f8e759" Oct 30 17:13:35.041041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.040960 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9656704c2b23703d73f46621851b1e2b24bc016e8aa3ffa6c52369d3f8e759"} err="failed to get container status \"1b9656704c2b23703d73f46621851b1e2b24bc016e8aa3ffa6c52369d3f8e759\": rpc error: code = NotFound desc = could not find container \"1b9656704c2b23703d73f46621851b1e2b24bc016e8aa3ffa6c52369d3f8e759\": container with ID starting with 1b9656704c2b23703d73f46621851b1e2b24bc016e8aa3ffa6c52369d3f8e759 not found: ID does not exist" Oct 30 17:13:35.069492 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.069448 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6733b6dd-929a-48c6-a8d8-ada293bc5079-kube-api-access-f97td\" (UniqueName: \"kubernetes.io/projected/6733b6dd-929a-48c6-a8d8-ada293bc5079-kube-api-access-f97td\") pod \"6733b6dd-929a-48c6-a8d8-ada293bc5079\" (UID: \"6733b6dd-929a-48c6-a8d8-ada293bc5079\") " Oct 30 17:13:35.069702 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.069577 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e64229e-1a05-4bdd-8ebc-f01a3661584e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e64229e-1a05-4bdd-8ebc-f01a3661584e-plugins-config-volume\") pod \"8e64229e-1a05-4bdd-8ebc-f01a3661584e\" (UID: \"8e64229e-1a05-4bdd-8ebc-f01a3661584e\") " Oct 30 17:13:35.069702 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.069628 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6733b6dd-929a-48c6-a8d8-ada293bc5079-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6733b6dd-929a-48c6-a8d8-ada293bc5079-plugins-config-volume\") pod \"6733b6dd-929a-48c6-a8d8-ada293bc5079\" (UID: \"6733b6dd-929a-48c6-a8d8-ada293bc5079\") " Oct 30 17:13:35.069880 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.069788 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e64229e-1a05-4bdd-8ebc-f01a3661584e-kube-api-access-d42pl\" (UniqueName: \"kubernetes.io/projected/8e64229e-1a05-4bdd-8ebc-f01a3661584e-kube-api-access-d42pl\") pod \"8e64229e-1a05-4bdd-8ebc-f01a3661584e\" (UID: \"8e64229e-1a05-4bdd-8ebc-f01a3661584e\") " Oct 30 17:13:35.071621 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.071586 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6733b6dd-929a-48c6-a8d8-ada293bc5079-plugins-config-volume" pod "6733b6dd-929a-48c6-a8d8-ada293bc5079" (UID: "6733b6dd-929a-48c6-a8d8-ada293bc5079"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:35.072250 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.072222 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e64229e-1a05-4bdd-8ebc-f01a3661584e-plugins-config-volume" pod "8e64229e-1a05-4bdd-8ebc-f01a3661584e" (UID: "8e64229e-1a05-4bdd-8ebc-f01a3661584e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:35.073653 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.073625 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e64229e-1a05-4bdd-8ebc-f01a3661584e-kube-api-access-d42pl" pod "8e64229e-1a05-4bdd-8ebc-f01a3661584e" (UID: "8e64229e-1a05-4bdd-8ebc-f01a3661584e"). InnerVolumeSpecName "kube-api-access-d42pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:35.083451 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.083420 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6733b6dd-929a-48c6-a8d8-ada293bc5079-kube-api-access-f97td" pod "6733b6dd-929a-48c6-a8d8-ada293bc5079" (UID: "6733b6dd-929a-48c6-a8d8-ada293bc5079"). InnerVolumeSpecName "kube-api-access-f97td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:35.171892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.171845 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f97td\" (UniqueName: \"kubernetes.io/projected/6733b6dd-929a-48c6-a8d8-ada293bc5079-kube-api-access-f97td\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:35.171892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.171891 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e64229e-1a05-4bdd-8ebc-f01a3661584e-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:35.172158 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.171906 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6733b6dd-929a-48c6-a8d8-ada293bc5079-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:35.172158 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.171920 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d42pl\" (UniqueName: \"kubernetes.io/projected/8e64229e-1a05-4bdd-8ebc-f01a3661584e-kube-api-access-d42pl\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:35.359070 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.358987 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m"] Oct 30 17:13:35.363015 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.362990 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-967fac-5cf77fd764-4fb8m"] Oct 30 17:13:35.400374 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.400343 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6"] Oct 30 17:13:35.406038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:35.406009 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-967fac-7f9755bc55-bhlv6"] Oct 30 17:13:36.217468 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:36.217441 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8" Oct 30 17:13:36.286785 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:36.286656 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-maas-api-xks-ca\") pod \"f03c8fbb-68d9-49ec-8d1f-2e7affec093d\" (UID: \"f03c8fbb-68d9-49ec-8d1f-2e7affec093d\") " Oct 30 17:13:36.286987 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:36.286822 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-kube-api-access-sxc69\" (UniqueName: \"kubernetes.io/projected/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-kube-api-access-sxc69\") pod \"f03c8fbb-68d9-49ec-8d1f-2e7affec093d\" (UID: \"f03c8fbb-68d9-49ec-8d1f-2e7affec093d\") " Oct 30 17:13:36.286987 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:36.286974 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-maas-api-openshift-ca\") pod \"f03c8fbb-68d9-49ec-8d1f-2e7affec093d\" (UID: \"f03c8fbb-68d9-49ec-8d1f-2e7affec093d\") " Oct 30 17:13:36.287433 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:36.287393 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-maas-api-openshift-ca" pod "f03c8fbb-68d9-49ec-8d1f-2e7affec093d" (UID: "f03c8fbb-68d9-49ec-8d1f-2e7affec093d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:36.288079 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:36.287910 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:36.290152 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:36.290113 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-kube-api-access-sxc69" pod "f03c8fbb-68d9-49ec-8d1f-2e7affec093d" (UID: "f03c8fbb-68d9-49ec-8d1f-2e7affec093d"). InnerVolumeSpecName "kube-api-access-sxc69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:36.290274 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:36.290180 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-maas-api-xks-ca" pod "f03c8fbb-68d9-49ec-8d1f-2e7affec093d" (UID: "f03c8fbb-68d9-49ec-8d1f-2e7affec093d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:36.357925 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:36.357882 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:36.389937 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:36.389900 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:36.389937 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:36.389936 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxc69\" (UniqueName: \"kubernetes.io/projected/f03c8fbb-68d9-49ec-8d1f-2e7affec093d-kube-api-access-sxc69\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:36.447821 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:36.447353 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6733b6dd-929a-48c6-a8d8-ada293bc5079" path="/var/lib/kubelet/pods/6733b6dd-929a-48c6-a8d8-ada293bc5079/volumes" Oct 30 17:13:36.451789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:36.448141 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e64229e-1a05-4bdd-8ebc-f01a3661584e" path="/var/lib/kubelet/pods/8e64229e-1a05-4bdd-8ebc-f01a3661584e/volumes" Oct 30 17:13:37.032064 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:37.032027 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8" event={"ID":"f03c8fbb-68d9-49ec-8d1f-2e7affec093d","Type":"ContainerDied","Data":"23df80f58b592d23cd81a4c8f494ce64a134473555874a716545301397de1e72"} Oct 30 17:13:37.032247 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:37.032104 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23df80f58b592d23cd81a4c8f494ce64a134473555874a716545301397de1e72" Oct 30 17:13:37.032247 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:37.032038 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8" Oct 30 17:13:38.085112 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:38.085065 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:38.875660 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:38.875616 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:39.597019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.596990 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp"] Oct 30 17:13:39.597809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.597793 6202 cpu_manager.go:414] "containerMap: removing container" podUID="f03c8fbb-68d9-49ec-8d1f-2e7affec093d" containerName="revoke-keys" Oct 30 17:13:39.597872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.597810 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f03c8fbb-68d9-49ec-8d1f-2e7affec093d" containerName="revoke-keys" Oct 30 17:13:39.597872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.597826 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8e64229e-1a05-4bdd-8ebc-f01a3661584e" containerName="payload-processing" Oct 30 17:13:39.597872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.597832 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e64229e-1a05-4bdd-8ebc-f01a3661584e" containerName="payload-processing" Oct 30 17:13:39.597872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.597841 6202 cpu_manager.go:414] "containerMap: removing container" podUID="6733b6dd-929a-48c6-a8d8-ada293bc5079" containerName="payload-pre-processing" Oct 30 17:13:39.597872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.597847 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6733b6dd-929a-48c6-a8d8-ada293bc5079" containerName="payload-pre-processing" Oct 30 17:13:39.598060 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.597970 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="f03c8fbb-68d9-49ec-8d1f-2e7affec093d" containerName="revoke-keys" Oct 30 17:13:39.598060 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.597979 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e64229e-1a05-4bdd-8ebc-f01a3661584e" containerName="payload-processing" Oct 30 17:13:39.598060 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.597987 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="6733b6dd-929a-48c6-a8d8-ada293bc5079" containerName="payload-pre-processing" Oct 30 17:13:39.602816 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.602798 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp" Oct 30 17:13:39.619098 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.619075 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7"] Oct 30 17:13:39.623842 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.623820 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7" Oct 30 17:13:39.624353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.624336 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp"] Oct 30 17:13:39.628500 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.628477 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:13:39.632819 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.632798 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:13:39.632966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.632815 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:13:39.633068 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.632969 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:13:39.636113 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.636088 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2e38ffa-flbnp\" (UID: \"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp" Oct 30 17:13:39.636802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.636737 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2e38ffa-flbnp\" (UID: \"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp" Oct 30 17:13:39.636924 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.636870 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcw8x\" (UniqueName: \"kubernetes.io/projected/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-kube-api-access-rcw8x\") pod \"maas-api-revoke-sub-c2e38ffa-flbnp\" (UID: \"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp" Oct 30 17:13:39.645366 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.645340 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7"] Oct 30 17:13:39.738425 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.738387 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc8zm\" (UniqueName: \"kubernetes.io/projected/c9c93bf2-386e-4a57-b52d-5c12f0da7671-kube-api-access-sc8zm\") pod \"maas-api-revoke-sub-44d721b6-h7dw7\" (UID: \"c9c93bf2-386e-4a57-b52d-5c12f0da7671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7" Oct 30 17:13:39.738584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.738450 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2e38ffa-flbnp\" (UID: \"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp" Oct 30 17:13:39.738584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.738563 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2e38ffa-flbnp\" (UID: \"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp" Oct 30 17:13:39.738679 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.738645 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c93bf2-386e-4a57-b52d-5c12f0da7671-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44d721b6-h7dw7\" (UID: \"c9c93bf2-386e-4a57-b52d-5c12f0da7671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7" Oct 30 17:13:39.738732 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.738715 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcw8x\" (UniqueName: \"kubernetes.io/projected/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-kube-api-access-rcw8x\") pod \"maas-api-revoke-sub-c2e38ffa-flbnp\" (UID: \"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp" Oct 30 17:13:39.738869 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.738841 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c93bf2-386e-4a57-b52d-5c12f0da7671-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44d721b6-h7dw7\" (UID: \"c9c93bf2-386e-4a57-b52d-5c12f0da7671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7" Oct 30 17:13:39.739427 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.739347 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2e38ffa-flbnp\" (UID: \"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp" Oct 30 17:13:39.741308 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.741280 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2e38ffa-flbnp\" (UID: \"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp" Oct 30 17:13:39.763549 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.763515 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcw8x\" (UniqueName: \"kubernetes.io/projected/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-kube-api-access-rcw8x\") pod \"maas-api-revoke-sub-c2e38ffa-flbnp\" (UID: \"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp" Oct 30 17:13:39.840642 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.840604 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c93bf2-386e-4a57-b52d-5c12f0da7671-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44d721b6-h7dw7\" (UID: \"c9c93bf2-386e-4a57-b52d-5c12f0da7671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7" Oct 30 17:13:39.840868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.840827 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc8zm\" (UniqueName: \"kubernetes.io/projected/c9c93bf2-386e-4a57-b52d-5c12f0da7671-kube-api-access-sc8zm\") pod \"maas-api-revoke-sub-44d721b6-h7dw7\" (UID: \"c9c93bf2-386e-4a57-b52d-5c12f0da7671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7" Oct 30 17:13:39.840961 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.840951 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c93bf2-386e-4a57-b52d-5c12f0da7671-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44d721b6-h7dw7\" (UID: \"c9c93bf2-386e-4a57-b52d-5c12f0da7671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7" Oct 30 17:13:39.841334 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.841309 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c93bf2-386e-4a57-b52d-5c12f0da7671-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44d721b6-h7dw7\" (UID: \"c9c93bf2-386e-4a57-b52d-5c12f0da7671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7" Oct 30 17:13:39.843433 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.843411 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c93bf2-386e-4a57-b52d-5c12f0da7671-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44d721b6-h7dw7\" (UID: \"c9c93bf2-386e-4a57-b52d-5c12f0da7671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7" Oct 30 17:13:39.865356 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.865300 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc8zm\" (UniqueName: \"kubernetes.io/projected/c9c93bf2-386e-4a57-b52d-5c12f0da7671-kube-api-access-sc8zm\") pod \"maas-api-revoke-sub-44d721b6-h7dw7\" (UID: \"c9c93bf2-386e-4a57-b52d-5c12f0da7671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7" Oct 30 17:13:39.918217 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.918187 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp" Oct 30 17:13:39.955191 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.955165 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks" Oct 30 17:13:39.956211 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.956190 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7" Oct 30 17:13:39.988812 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:39.979340 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:40.237961 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:13:40.237905 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f07f3a4_50a6_4dcf_9528_bc03555ce3d4.slice/crio-3d06b7e2c4b543a34ba8bfeb9d0e95b9dce1368301b0a3e335cbb2eefbe71277 WatchSource:0}: Error finding container 3d06b7e2c4b543a34ba8bfeb9d0e95b9dce1368301b0a3e335cbb2eefbe71277: Status 404 returned error can't find the container with id 3d06b7e2c4b543a34ba8bfeb9d0e95b9dce1368301b0a3e335cbb2eefbe71277 Oct 30 17:13:40.240974 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:40.240945 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp"] Oct 30 17:13:40.280139 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:13:40.280112 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c93bf2_386e_4a57_b52d_5c12f0da7671.slice/crio-43ec7e278eb9ba194d487b0aafc38677a9527b2077f8c73a6943078ed0da0491 WatchSource:0}: Error finding container 43ec7e278eb9ba194d487b0aafc38677a9527b2077f8c73a6943078ed0da0491: Status 404 returned error can't find the container with id 43ec7e278eb9ba194d487b0aafc38677a9527b2077f8c73a6943078ed0da0491 Oct 30 17:13:40.282271 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:40.282249 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7"] Oct 30 17:13:41.077349 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:41.077306 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7" event={"ID":"c9c93bf2-386e-4a57-b52d-5c12f0da7671","Type":"ContainerStarted","Data":"9fa7ab85b2b0ba87688e0f74b148559e5ee037496851db80276c48341ad1bae2"} Oct 30 17:13:41.077880 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:41.077375 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7" event={"ID":"c9c93bf2-386e-4a57-b52d-5c12f0da7671","Type":"ContainerStarted","Data":"43ec7e278eb9ba194d487b0aafc38677a9527b2077f8c73a6943078ed0da0491"} Oct 30 17:13:41.079415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:41.079385 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp" event={"ID":"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4","Type":"ContainerStarted","Data":"04c57318e3c5036fbac4debfd93f9b64e27d22929f2394f2013097ccbe6cf692"} Oct 30 17:13:41.079535 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:41.079441 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp" event={"ID":"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4","Type":"ContainerStarted","Data":"3d06b7e2c4b543a34ba8bfeb9d0e95b9dce1368301b0a3e335cbb2eefbe71277"} Oct 30 17:13:41.120938 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:41.120834 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7" podStartSLOduration=2.120818175 podStartE2EDuration="2.120818175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:41.119485264 +0000 UTC m=+3134.829940009" watchObservedRunningTime="2026-10-30 17:13:41.120818175 +0000 UTC m=+3134.831272919" Oct 30 17:13:41.167191 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:41.167138 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp" podStartSLOduration=2.167124274 podStartE2EDuration="2.167124274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:41.163618581 +0000 UTC m=+3134.874073342" watchObservedRunningTime="2026-10-30 17:13:41.167124274 +0000 UTC m=+3134.877579018" Oct 30 17:13:41.930948 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:41.930908 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:42.089883 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:42.089852 6202 generic.go:356] "Generic (PLEG): container finished" podID="c9c93bf2-386e-4a57-b52d-5c12f0da7671" containerID="9fa7ab85b2b0ba87688e0f74b148559e5ee037496851db80276c48341ad1bae2" exitCode=0 Oct 30 17:13:42.090347 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:42.089943 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7" event={"ID":"c9c93bf2-386e-4a57-b52d-5c12f0da7671","Type":"ContainerDied","Data":"9fa7ab85b2b0ba87688e0f74b148559e5ee037496851db80276c48341ad1bae2"} Oct 30 17:13:42.091819 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:42.091758 6202 generic.go:356] "Generic (PLEG): container finished" podID="1f07f3a4-50a6-4dcf-9528-bc03555ce3d4" containerID="04c57318e3c5036fbac4debfd93f9b64e27d22929f2394f2013097ccbe6cf692" exitCode=0 Oct 30 17:13:42.091819 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:42.091806 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp" event={"ID":"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4","Type":"ContainerDied","Data":"04c57318e3c5036fbac4debfd93f9b64e27d22929f2394f2013097ccbe6cf692"} Oct 30 17:13:42.543584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:42.543547 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:43.078116 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.078085 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:43.198866 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.196237 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:43.409095 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.408872 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:43.430909 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.430878 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp" Oct 30 17:13:43.495487 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.495457 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7" Oct 30 17:13:43.632841 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.594495 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-kube-api-access-rcw8x\" (UniqueName: \"kubernetes.io/projected/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-kube-api-access-rcw8x\") pod \"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4\" (UID: \"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4\") " Oct 30 17:13:43.632841 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.594550 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-maas-api-xks-ca\") pod \"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4\" (UID: \"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4\") " Oct 30 17:13:43.632841 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.594690 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-maas-api-openshift-ca\") pod \"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4\" (UID: \"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4\") " Oct 30 17:13:43.639788 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.634273 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-maas-api-openshift-ca" pod "1f07f3a4-50a6-4dcf-9528-bc03555ce3d4" (UID: "1f07f3a4-50a6-4dcf-9528-bc03555ce3d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:43.668988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.668950 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-kube-api-access-rcw8x" pod "1f07f3a4-50a6-4dcf-9528-bc03555ce3d4" (UID: "1f07f3a4-50a6-4dcf-9528-bc03555ce3d4"). InnerVolumeSpecName "kube-api-access-rcw8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:43.669271 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.669255 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-maas-api-xks-ca" pod "1f07f3a4-50a6-4dcf-9528-bc03555ce3d4" (UID: "1f07f3a4-50a6-4dcf-9528-bc03555ce3d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:43.684940 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.684898 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d"] Oct 30 17:13:43.703787 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.686430 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c9c93bf2-386e-4a57-b52d-5c12f0da7671" containerName="revoke-keys" Oct 30 17:13:43.704020 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.703995 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c93bf2-386e-4a57-b52d-5c12f0da7671" containerName="revoke-keys" Oct 30 17:13:43.704171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.704160 6202 cpu_manager.go:414] "containerMap: removing container" podUID="1f07f3a4-50a6-4dcf-9528-bc03555ce3d4" containerName="revoke-keys" Oct 30 17:13:43.704245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.704235 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f07f3a4-50a6-4dcf-9528-bc03555ce3d4" containerName="revoke-keys" Oct 30 17:13:43.704576 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.704561 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c93bf2-386e-4a57-b52d-5c12f0da7671" containerName="revoke-keys" Oct 30 17:13:43.704685 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.704674 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f07f3a4-50a6-4dcf-9528-bc03555ce3d4" containerName="revoke-keys" Oct 30 17:13:43.740113 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.723127 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c93bf2-386e-4a57-b52d-5c12f0da7671-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c93bf2-386e-4a57-b52d-5c12f0da7671-maas-api-openshift-ca\") pod \"c9c93bf2-386e-4a57-b52d-5c12f0da7671\" (UID: \"c9c93bf2-386e-4a57-b52d-5c12f0da7671\") " Oct 30 17:13:43.744332 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.744298 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c93bf2-386e-4a57-b52d-5c12f0da7671-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c93bf2-386e-4a57-b52d-5c12f0da7671-maas-api-xks-ca\") pod \"c9c93bf2-386e-4a57-b52d-5c12f0da7671\" (UID: \"c9c93bf2-386e-4a57-b52d-5c12f0da7671\") " Oct 30 17:13:43.779617 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.779583 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c93bf2-386e-4a57-b52d-5c12f0da7671-kube-api-access-sc8zm\" (UniqueName: \"kubernetes.io/projected/c9c93bf2-386e-4a57-b52d-5c12f0da7671-kube-api-access-sc8zm\") pod \"c9c93bf2-386e-4a57-b52d-5c12f0da7671\" (UID: \"c9c93bf2-386e-4a57-b52d-5c12f0da7671\") " Oct 30 17:13:43.781116 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.781084 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcw8x\" (UniqueName: \"kubernetes.io/projected/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-kube-api-access-rcw8x\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:43.781329 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.781315 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:43.797866 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.797827 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:43.798053 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.789574 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb"] Oct 30 17:13:43.801673 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.789666 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d" Oct 30 17:13:43.803399 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.729116 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c93bf2-386e-4a57-b52d-5c12f0da7671-maas-api-openshift-ca" pod "c9c93bf2-386e-4a57-b52d-5c12f0da7671" (UID: "c9c93bf2-386e-4a57-b52d-5c12f0da7671"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:43.803560 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.756911 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c93bf2-386e-4a57-b52d-5c12f0da7671-maas-api-xks-ca" pod "c9c93bf2-386e-4a57-b52d-5c12f0da7671" (UID: "c9c93bf2-386e-4a57-b52d-5c12f0da7671"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:43.803631 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.791753 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c93bf2-386e-4a57-b52d-5c12f0da7671-kube-api-access-sc8zm" pod "c9c93bf2-386e-4a57-b52d-5c12f0da7671" (UID: "c9c93bf2-386e-4a57-b52d-5c12f0da7671"). InnerVolumeSpecName "kube-api-access-sc8zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:43.829866 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.822840 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0ae3fe\"" Oct 30 17:13:43.829866 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.822953 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0ae3fe-dockercfg-xcc47\"" Oct 30 17:13:43.856057 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.856027 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d"] Oct 30 17:13:43.856300 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.856284 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb"] Oct 30 17:13:43.856434 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.856422 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn"] Oct 30 17:13:43.857079 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.857052 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb" Oct 30 17:13:43.871273 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.870951 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn"] Oct 30 17:13:43.871273 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.871072 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn" Oct 30 17:13:43.899522 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.899491 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9gt6\" (UniqueName: \"kubernetes.io/projected/1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a-kube-api-access-v9gt6\") pod \"payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d\" (UID: \"1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d" Oct 30 17:13:43.899668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.899642 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d\" (UID: \"1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d" Oct 30 17:13:43.899834 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.899815 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c93bf2-386e-4a57-b52d-5c12f0da7671-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:43.899902 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.899841 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c93bf2-386e-4a57-b52d-5c12f0da7671-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:43.899902 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:43.899855 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc8zm\" (UniqueName: \"kubernetes.io/projected/c9c93bf2-386e-4a57-b52d-5c12f0da7671-kube-api-access-sc8zm\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:44.001225 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.001195 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnns5\" (UniqueName: \"kubernetes.io/projected/6a0d47fd-b562-4eb1-b283-23bd21abc026-kube-api-access-cnns5\") pod \"payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn\" (UID: \"6a0d47fd-b562-4eb1-b283-23bd21abc026\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn" Oct 30 17:13:44.001400 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.001240 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb\" (UID: \"c0bcf171-31dc-4ee9-bbab-95ea9843eab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb" Oct 30 17:13:44.001456 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.001430 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb\" (UID: \"c0bcf171-31dc-4ee9-bbab-95ea9843eab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb" Oct 30 17:13:44.001505 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.001488 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-httww\" (UniqueName: \"kubernetes.io/projected/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-kube-api-access-httww\") pod \"maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb\" (UID: \"c0bcf171-31dc-4ee9-bbab-95ea9843eab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb" Oct 30 17:13:44.001574 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.001558 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0d47fd-b562-4eb1-b283-23bd21abc026-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn\" (UID: \"6a0d47fd-b562-4eb1-b283-23bd21abc026\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn" Oct 30 17:13:44.001639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.001626 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9gt6\" (UniqueName: \"kubernetes.io/projected/1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a-kube-api-access-v9gt6\") pod \"payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d\" (UID: \"1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d" Oct 30 17:13:44.001814 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.001788 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d\" (UID: \"1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d" Oct 30 17:13:44.002403 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.002384 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d\" (UID: \"1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d" Oct 30 17:13:44.026175 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.026147 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9gt6\" (UniqueName: \"kubernetes.io/projected/1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a-kube-api-access-v9gt6\") pod \"payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d\" (UID: \"1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d" Oct 30 17:13:44.071600 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:13:44.071565 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:13:44.102788 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.102742 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb\" (UID: \"c0bcf171-31dc-4ee9-bbab-95ea9843eab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb" Oct 30 17:13:44.102995 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.102825 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-httww\" (UniqueName: \"kubernetes.io/projected/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-kube-api-access-httww\") pod \"maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb\" (UID: \"c0bcf171-31dc-4ee9-bbab-95ea9843eab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb" Oct 30 17:13:44.102995 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.102878 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0d47fd-b562-4eb1-b283-23bd21abc026-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn\" (UID: \"6a0d47fd-b562-4eb1-b283-23bd21abc026\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn" Oct 30 17:13:44.103117 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.103044 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnns5\" (UniqueName: \"kubernetes.io/projected/6a0d47fd-b562-4eb1-b283-23bd21abc026-kube-api-access-cnns5\") pod \"payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn\" (UID: \"6a0d47fd-b562-4eb1-b283-23bd21abc026\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn" Oct 30 17:13:44.103117 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.103091 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb\" (UID: \"c0bcf171-31dc-4ee9-bbab-95ea9843eab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb" Oct 30 17:13:44.103863 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.103833 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0d47fd-b562-4eb1-b283-23bd21abc026-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn\" (UID: \"6a0d47fd-b562-4eb1-b283-23bd21abc026\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn" Oct 30 17:13:44.104032 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.103833 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb\" (UID: \"c0bcf171-31dc-4ee9-bbab-95ea9843eab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb" Oct 30 17:13:44.105401 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.105381 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb\" (UID: \"c0bcf171-31dc-4ee9-bbab-95ea9843eab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb" Oct 30 17:13:44.110453 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.110432 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7" Oct 30 17:13:44.110557 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.110462 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7" event={"ID":"c9c93bf2-386e-4a57-b52d-5c12f0da7671","Type":"ContainerDied","Data":"43ec7e278eb9ba194d487b0aafc38677a9527b2077f8c73a6943078ed0da0491"} Oct 30 17:13:44.110557 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.110514 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43ec7e278eb9ba194d487b0aafc38677a9527b2077f8c73a6943078ed0da0491" Oct 30 17:13:44.112000 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.111981 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp" Oct 30 17:13:44.112101 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.111968 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp" event={"ID":"1f07f3a4-50a6-4dcf-9528-bc03555ce3d4","Type":"ContainerDied","Data":"3d06b7e2c4b543a34ba8bfeb9d0e95b9dce1368301b0a3e335cbb2eefbe71277"} Oct 30 17:13:44.112101 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.112092 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d06b7e2c4b543a34ba8bfeb9d0e95b9dce1368301b0a3e335cbb2eefbe71277" Oct 30 17:13:44.127188 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.127163 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnns5\" (UniqueName: \"kubernetes.io/projected/6a0d47fd-b562-4eb1-b283-23bd21abc026-kube-api-access-cnns5\") pod \"payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn\" (UID: \"6a0d47fd-b562-4eb1-b283-23bd21abc026\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn" Oct 30 17:13:44.127279 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.127217 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-httww\" (UniqueName: \"kubernetes.io/projected/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-kube-api-access-httww\") pod \"maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb\" (UID: \"c0bcf171-31dc-4ee9-bbab-95ea9843eab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb" Oct 30 17:13:44.174685 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.174596 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d" Oct 30 17:13:44.197973 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.197945 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn" Oct 30 17:13:44.201043 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.201023 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb" Oct 30 17:13:44.471326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.471221 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:44.492666 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:13:44.492638 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fdc1f9b_8e02_4deb_9bf3_7941d8b6047a.slice/crio-7bf1675b34c03cd39bca0d9f1ac5eb4d62214906ee6f22eca773ed0165dbc6e4 WatchSource:0}: Error finding container 7bf1675b34c03cd39bca0d9f1ac5eb4d62214906ee6f22eca773ed0165dbc6e4: Status 404 returned error can't find the container with id 7bf1675b34c03cd39bca0d9f1ac5eb4d62214906ee6f22eca773ed0165dbc6e4 Oct 30 17:13:44.494603 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.494579 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d"] Oct 30 17:13:44.560054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.560011 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn"] Oct 30 17:13:44.571815 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:13:44.571662 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0d47fd_b562_4eb1_b283_23bd21abc026.slice/crio-5388e643ad5301321ddd2b6a627adddb23f0e7766954a3a22fb647eb11b3222e WatchSource:0}: Error finding container 5388e643ad5301321ddd2b6a627adddb23f0e7766954a3a22fb647eb11b3222e: Status 404 returned error can't find the container with id 5388e643ad5301321ddd2b6a627adddb23f0e7766954a3a22fb647eb11b3222e Oct 30 17:13:44.602259 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:44.602236 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb"] Oct 30 17:13:45.041270 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:45.041236 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn"] Oct 30 17:13:45.046810 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:45.046784 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1601645-4qpgn"] Oct 30 17:13:45.123989 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:45.123950 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb" event={"ID":"c0bcf171-31dc-4ee9-bbab-95ea9843eab7","Type":"ContainerStarted","Data":"da5a898c5fa3a3298e58c47a64614a52b7660c05eecb6b943a1f027ce338ad59"} Oct 30 17:13:45.124184 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:45.124011 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb" event={"ID":"c0bcf171-31dc-4ee9-bbab-95ea9843eab7","Type":"ContainerStarted","Data":"b5ea43fba62578ea3ff55b86762ad84c023ac45b1808b118e34354fc779386da"} Oct 30 17:13:45.125972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:45.125943 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d" event={"ID":"1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a","Type":"ContainerStarted","Data":"7fa91a5f3fe2fa7631292c931a2c932dfea0dff88260059ccfbccd9e9f739769"} Oct 30 17:13:45.126106 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:45.125997 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d" event={"ID":"1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a","Type":"ContainerStarted","Data":"7bf1675b34c03cd39bca0d9f1ac5eb4d62214906ee6f22eca773ed0165dbc6e4"} Oct 30 17:13:45.126106 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:45.126066 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d" Oct 30 17:13:45.127759 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:45.127735 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn" event={"ID":"6a0d47fd-b562-4eb1-b283-23bd21abc026","Type":"ContainerStarted","Data":"2aaccceff6899cac6c28cf12aa3252ee21765ce9a8a8406a82c87e139d018ab1"} Oct 30 17:13:45.127910 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:45.127798 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn" event={"ID":"6a0d47fd-b562-4eb1-b283-23bd21abc026","Type":"ContainerStarted","Data":"5388e643ad5301321ddd2b6a627adddb23f0e7766954a3a22fb647eb11b3222e"} Oct 30 17:13:45.127910 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:45.127828 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn" Oct 30 17:13:45.172997 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:45.172934 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb" podStartSLOduration=2.172915688 podStartE2EDuration="2.172915688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:45.166307896 +0000 UTC m=+3138.876762638" watchObservedRunningTime="2026-10-30 17:13:45.172915688 +0000 UTC m=+3138.883370432" Oct 30 17:13:45.217602 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:45.217551 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn" podStartSLOduration=2.21753838 podStartE2EDuration="2.21753838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:45.216849359 +0000 UTC m=+3138.927304105" watchObservedRunningTime="2026-10-30 17:13:45.21753838 +0000 UTC m=+3138.927993124" Oct 30 17:13:45.280073 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:45.279995 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d" podStartSLOduration=2.279973965 podStartE2EDuration="2.279973965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:45.273079777 +0000 UTC m=+3138.983534523" watchObservedRunningTime="2026-10-30 17:13:45.279973965 +0000 UTC m=+3138.990428710" Oct 30 17:13:46.142831 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:46.142741 6202 generic.go:356] "Generic (PLEG): container finished" podID="c0bcf171-31dc-4ee9-bbab-95ea9843eab7" containerID="da5a898c5fa3a3298e58c47a64614a52b7660c05eecb6b943a1f027ce338ad59" exitCode=0 Oct 30 17:13:46.143144 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:46.142827 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb" event={"ID":"c0bcf171-31dc-4ee9-bbab-95ea9843eab7","Type":"ContainerDied","Data":"da5a898c5fa3a3298e58c47a64614a52b7660c05eecb6b943a1f027ce338ad59"} Oct 30 17:13:46.157747 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:46.157713 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:46.269785 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:46.269730 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:46.414455 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:46.414375 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43be2c46-9b99-4174-a409-c67eb7563505" path="/var/lib/kubelet/pods/43be2c46-9b99-4174-a409-c67eb7563505/volumes" Oct 30 17:13:46.537495 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:46.537459 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:46.656104 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:46.656068 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:46.767415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:46.767326 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:47.341901 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:47.341876 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb" Oct 30 17:13:47.391830 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:47.391792 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-maas-api-xks-ca\") pod \"c0bcf171-31dc-4ee9-bbab-95ea9843eab7\" (UID: \"c0bcf171-31dc-4ee9-bbab-95ea9843eab7\") " Oct 30 17:13:47.392009 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:47.391903 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-maas-api-openshift-ca\") pod \"c0bcf171-31dc-4ee9-bbab-95ea9843eab7\" (UID: \"c0bcf171-31dc-4ee9-bbab-95ea9843eab7\") " Oct 30 17:13:47.392009 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:47.391983 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-kube-api-access-httww\" (UniqueName: \"kubernetes.io/projected/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-kube-api-access-httww\") pod \"c0bcf171-31dc-4ee9-bbab-95ea9843eab7\" (UID: \"c0bcf171-31dc-4ee9-bbab-95ea9843eab7\") " Oct 30 17:13:47.392286 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:47.392262 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-maas-api-openshift-ca" pod "c0bcf171-31dc-4ee9-bbab-95ea9843eab7" (UID: "c0bcf171-31dc-4ee9-bbab-95ea9843eab7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:47.392565 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:47.392547 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:47.394305 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:47.394280 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-maas-api-xks-ca" pod "c0bcf171-31dc-4ee9-bbab-95ea9843eab7" (UID: "c0bcf171-31dc-4ee9-bbab-95ea9843eab7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:47.394877 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:47.394843 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-kube-api-access-httww" pod "c0bcf171-31dc-4ee9-bbab-95ea9843eab7" (UID: "c0bcf171-31dc-4ee9-bbab-95ea9843eab7"). InnerVolumeSpecName "kube-api-access-httww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:47.494001 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:47.493963 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:47.494193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:47.494055 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-httww\" (UniqueName: \"kubernetes.io/projected/c0bcf171-31dc-4ee9-bbab-95ea9843eab7-kube-api-access-httww\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:13:48.164174 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:48.164144 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb" Oct 30 17:13:48.164174 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:48.164156 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb" event={"ID":"c0bcf171-31dc-4ee9-bbab-95ea9843eab7","Type":"ContainerDied","Data":"b5ea43fba62578ea3ff55b86762ad84c023ac45b1808b118e34354fc779386da"} Oct 30 17:13:48.164376 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:48.164212 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5ea43fba62578ea3ff55b86762ad84c023ac45b1808b118e34354fc779386da" Oct 30 17:13:48.584254 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:48.584220 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:50.259720 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:50.259681 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:52.490126 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:52.490086 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:54.196878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:54.196842 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:55.751246 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:55.751215 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:55.951072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:55.951042 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:56.037473 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:56.037376 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k"] Oct 30 17:13:56.051331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:56.051296 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725f3b09-qk25k"] Oct 30 17:13:56.066088 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:56.066058 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:56.146347 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:56.146315 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn" Oct 30 17:13:56.146641 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:56.146607 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d" Oct 30 17:13:56.322130 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:56.322090 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:56.462797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:56.452276 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:56.664842 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:56.654799 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:56.710805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:56.707993 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63c6cc3-5fb1-46d2-8418-fd5fb737e585" path="/var/lib/kubelet/pods/c63c6cc3-5fb1-46d2-8418-fd5fb737e585/volumes" Oct 30 17:13:56.799076 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:56.799041 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:56.816925 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:56.805861 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb"] Oct 30 17:13:56.845050 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:56.845014 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e4048c-fp6lb"] Oct 30 17:13:58.412136 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:58.412104 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0bcf171-31dc-4ee9-bbab-95ea9843eab7" path="/var/lib/kubelet/pods/c0bcf171-31dc-4ee9-bbab-95ea9843eab7/volumes" Oct 30 17:13:59.421206 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:13:59.421160 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:00.280831 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:00.280796 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:00.412492 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:00.410594 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:01.269910 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.269872 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9"] Oct 30 17:14:01.271238 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.271216 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c0bcf171-31dc-4ee9-bbab-95ea9843eab7" containerName="revoke-keys" Oct 30 17:14:01.271238 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.271239 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0bcf171-31dc-4ee9-bbab-95ea9843eab7" containerName="revoke-keys" Oct 30 17:14:01.271438 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.271424 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0bcf171-31dc-4ee9-bbab-95ea9843eab7" containerName="revoke-keys" Oct 30 17:14:01.277132 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.277108 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9" Oct 30 17:14:01.292660 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.292626 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9"] Oct 30 17:14:01.298607 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.298576 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:01.300385 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.300362 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:14:01.300643 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.300626 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:14:01.300990 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.300968 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:14:01.311090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.311062 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:14:01.424907 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.424873 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3082585b-6424-472e-8014-0eb3b5ad15a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9\" (UID: \"3082585b-6424-472e-8014-0eb3b5ad15a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9" Oct 30 17:14:01.425092 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.425014 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9cpx\" (UniqueName: \"kubernetes.io/projected/3082585b-6424-472e-8014-0eb3b5ad15a8-kube-api-access-x9cpx\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9\" (UID: \"3082585b-6424-472e-8014-0eb3b5ad15a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9" Oct 30 17:14:01.425092 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.425075 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3082585b-6424-472e-8014-0eb3b5ad15a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9\" (UID: \"3082585b-6424-472e-8014-0eb3b5ad15a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9" Oct 30 17:14:01.526736 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.526658 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3082585b-6424-472e-8014-0eb3b5ad15a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9\" (UID: \"3082585b-6424-472e-8014-0eb3b5ad15a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9" Oct 30 17:14:01.526946 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.526891 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3082585b-6424-472e-8014-0eb3b5ad15a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9\" (UID: \"3082585b-6424-472e-8014-0eb3b5ad15a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9" Oct 30 17:14:01.527031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.526968 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9cpx\" (UniqueName: \"kubernetes.io/projected/3082585b-6424-472e-8014-0eb3b5ad15a8-kube-api-access-x9cpx\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9\" (UID: \"3082585b-6424-472e-8014-0eb3b5ad15a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9" Oct 30 17:14:01.527668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.527646 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3082585b-6424-472e-8014-0eb3b5ad15a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9\" (UID: \"3082585b-6424-472e-8014-0eb3b5ad15a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9" Oct 30 17:14:01.529219 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.529202 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3082585b-6424-472e-8014-0eb3b5ad15a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9\" (UID: \"3082585b-6424-472e-8014-0eb3b5ad15a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9" Oct 30 17:14:01.551681 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.551644 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9cpx\" (UniqueName: \"kubernetes.io/projected/3082585b-6424-472e-8014-0eb3b5ad15a8-kube-api-access-x9cpx\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9\" (UID: \"3082585b-6424-472e-8014-0eb3b5ad15a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9" Oct 30 17:14:01.605913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.605879 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9" Oct 30 17:14:01.786691 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.786600 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd"] Oct 30 17:14:01.791718 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.791693 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd" Oct 30 17:14:01.812755 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.812724 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd"] Oct 30 17:14:01.830870 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.830836 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd\" (UID: \"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd" Oct 30 17:14:01.831119 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.831040 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m6mx\" (UniqueName: \"kubernetes.io/projected/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-kube-api-access-4m6mx\") pod \"maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd\" (UID: \"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd" Oct 30 17:14:01.831268 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.831241 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd\" (UID: \"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd" Oct 30 17:14:01.919069 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:14:01.919037 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3082585b_6424_472e_8014_0eb3b5ad15a8.slice/crio-7128a8a6dd866195ba2c6b977452f57cc18b2c17a83b00564f5b7a50807b118d WatchSource:0}: Error finding container 7128a8a6dd866195ba2c6b977452f57cc18b2c17a83b00564f5b7a50807b118d: Status 404 returned error can't find the container with id 7128a8a6dd866195ba2c6b977452f57cc18b2c17a83b00564f5b7a50807b118d Oct 30 17:14:01.922125 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.922020 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9"] Oct 30 17:14:01.932278 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.932252 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd\" (UID: \"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd" Oct 30 17:14:01.932394 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.932335 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd\" (UID: \"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd" Oct 30 17:14:01.932463 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.932432 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m6mx\" (UniqueName: \"kubernetes.io/projected/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-kube-api-access-4m6mx\") pod \"maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd\" (UID: \"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd" Oct 30 17:14:01.932963 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.932939 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd\" (UID: \"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd" Oct 30 17:14:01.934706 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.934689 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd\" (UID: \"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd" Oct 30 17:14:01.959470 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:01.959435 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m6mx\" (UniqueName: \"kubernetes.io/projected/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-kube-api-access-4m6mx\") pod \"maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd\" (UID: \"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd" Oct 30 17:14:02.106466 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:02.106433 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd" Oct 30 17:14:02.343992 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:02.340605 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9" event={"ID":"3082585b-6424-472e-8014-0eb3b5ad15a8","Type":"ContainerStarted","Data":"7128a8a6dd866195ba2c6b977452f57cc18b2c17a83b00564f5b7a50807b118d"} Oct 30 17:14:02.734313 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:02.734208 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:02.780067 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:14:02.773323 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a655f6d_9e7c_4a6f_8dd1_9666d9fb392a.slice/crio-686641eba9d5bea92a449c082f270d3c11f9d49a29ac70a4db1e4c7a13b9ad05 WatchSource:0}: Error finding container 686641eba9d5bea92a449c082f270d3c11f9d49a29ac70a4db1e4c7a13b9ad05: Status 404 returned error can't find the container with id 686641eba9d5bea92a449c082f270d3c11f9d49a29ac70a4db1e4c7a13b9ad05 Oct 30 17:14:02.780990 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:02.780959 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd"] Oct 30 17:14:03.354035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:03.354001 6202 generic.go:356] "Generic (PLEG): container finished" podID="3082585b-6424-472e-8014-0eb3b5ad15a8" containerID="aaaaab53f3a5e64e01df7036cc81e706050cd6d3e1f4d060b495d64a2772addd" exitCode=0 Oct 30 17:14:03.354525 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:03.354095 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9" event={"ID":"3082585b-6424-472e-8014-0eb3b5ad15a8","Type":"ContainerDied","Data":"aaaaab53f3a5e64e01df7036cc81e706050cd6d3e1f4d060b495d64a2772addd"} Oct 30 17:14:03.356176 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:03.356146 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd" event={"ID":"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a","Type":"ContainerStarted","Data":"f381bea4acb2b9b99d6c395b94315cc539ee5ad209bba1eee1addf424e4c86d8"} Oct 30 17:14:03.356318 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:03.356203 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd" event={"ID":"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a","Type":"ContainerStarted","Data":"686641eba9d5bea92a449c082f270d3c11f9d49a29ac70a4db1e4c7a13b9ad05"} Oct 30 17:14:03.441407 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:03.441342 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd" podStartSLOduration=2.441323861 podStartE2EDuration="2.441323861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:03.438294051 +0000 UTC m=+3157.148748795" watchObservedRunningTime="2026-10-30 17:14:03.441323861 +0000 UTC m=+3157.151778607" Oct 30 17:14:03.611443 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:03.611348 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:04.367935 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:04.367896 6202 generic.go:356] "Generic (PLEG): container finished" podID="5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a" containerID="f381bea4acb2b9b99d6c395b94315cc539ee5ad209bba1eee1addf424e4c86d8" exitCode=0 Oct 30 17:14:04.368445 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:04.367981 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd" event={"ID":"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a","Type":"ContainerDied","Data":"f381bea4acb2b9b99d6c395b94315cc539ee5ad209bba1eee1addf424e4c86d8"} Oct 30 17:14:04.559301 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:04.559271 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9" Oct 30 17:14:04.689442 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:04.689363 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3082585b-6424-472e-8014-0eb3b5ad15a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3082585b-6424-472e-8014-0eb3b5ad15a8-maas-api-xks-ca\") pod \"3082585b-6424-472e-8014-0eb3b5ad15a8\" (UID: \"3082585b-6424-472e-8014-0eb3b5ad15a8\") " Oct 30 17:14:04.689594 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:04.689483 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3082585b-6424-472e-8014-0eb3b5ad15a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3082585b-6424-472e-8014-0eb3b5ad15a8-maas-api-openshift-ca\") pod \"3082585b-6424-472e-8014-0eb3b5ad15a8\" (UID: \"3082585b-6424-472e-8014-0eb3b5ad15a8\") " Oct 30 17:14:04.689658 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:04.689642 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3082585b-6424-472e-8014-0eb3b5ad15a8-kube-api-access-x9cpx\" (UniqueName: \"kubernetes.io/projected/3082585b-6424-472e-8014-0eb3b5ad15a8-kube-api-access-x9cpx\") pod \"3082585b-6424-472e-8014-0eb3b5ad15a8\" (UID: \"3082585b-6424-472e-8014-0eb3b5ad15a8\") " Oct 30 17:14:04.689939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:04.689907 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3082585b-6424-472e-8014-0eb3b5ad15a8-maas-api-openshift-ca" pod "3082585b-6424-472e-8014-0eb3b5ad15a8" (UID: "3082585b-6424-472e-8014-0eb3b5ad15a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:04.690200 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:04.690171 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3082585b-6424-472e-8014-0eb3b5ad15a8-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:04.691622 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:04.691596 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3082585b-6424-472e-8014-0eb3b5ad15a8-maas-api-xks-ca" pod "3082585b-6424-472e-8014-0eb3b5ad15a8" (UID: "3082585b-6424-472e-8014-0eb3b5ad15a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:04.691828 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:04.691802 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3082585b-6424-472e-8014-0eb3b5ad15a8-kube-api-access-x9cpx" pod "3082585b-6424-472e-8014-0eb3b5ad15a8" (UID: "3082585b-6424-472e-8014-0eb3b5ad15a8"). InnerVolumeSpecName "kube-api-access-x9cpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:04.791521 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:04.791476 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9cpx\" (UniqueName: \"kubernetes.io/projected/3082585b-6424-472e-8014-0eb3b5ad15a8-kube-api-access-x9cpx\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:04.791521 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:04.791513 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3082585b-6424-472e-8014-0eb3b5ad15a8-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:05.324548 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:05.324508 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:05.434046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:05.433956 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9" event={"ID":"3082585b-6424-472e-8014-0eb3b5ad15a8","Type":"ContainerDied","Data":"7128a8a6dd866195ba2c6b977452f57cc18b2c17a83b00564f5b7a50807b118d"} Oct 30 17:14:05.434046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:05.433982 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9" Oct 30 17:14:05.434046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:05.434014 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7128a8a6dd866195ba2c6b977452f57cc18b2c17a83b00564f5b7a50807b118d" Oct 30 17:14:05.621882 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:05.621242 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd" Oct 30 17:14:05.718919 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:05.718880 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-kube-api-access-4m6mx\" (UniqueName: \"kubernetes.io/projected/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-kube-api-access-4m6mx\") pod \"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a\" (UID: \"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a\") " Oct 30 17:14:05.719094 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:05.718967 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-maas-api-openshift-ca\") pod \"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a\" (UID: \"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a\") " Oct 30 17:14:05.719094 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:05.719017 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-maas-api-xks-ca\") pod \"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a\" (UID: \"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a\") " Oct 30 17:14:05.719346 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:05.719319 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-maas-api-openshift-ca" pod "5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a" (UID: "5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:05.719554 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:05.719532 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:05.721298 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:05.721264 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-maas-api-xks-ca" pod "5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a" (UID: "5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:05.721406 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:05.721315 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-kube-api-access-4m6mx" pod "5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a" (UID: "5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a"). InnerVolumeSpecName "kube-api-access-4m6mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:05.821516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:05.821484 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m6mx\" (UniqueName: \"kubernetes.io/projected/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-kube-api-access-4m6mx\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:05.821516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:05.821515 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:06.443130 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:06.443098 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd" Oct 30 17:14:06.443613 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:06.443136 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd" event={"ID":"5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a","Type":"ContainerDied","Data":"686641eba9d5bea92a449c082f270d3c11f9d49a29ac70a4db1e4c7a13b9ad05"} Oct 30 17:14:06.443613 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:06.443186 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="686641eba9d5bea92a449c082f270d3c11f9d49a29ac70a4db1e4c7a13b9ad05" Oct 30 17:14:06.967088 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:06.967052 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:07.031878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:07.031843 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q"] Oct 30 17:14:07.046973 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:07.046938 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96da0afe-rgg4q"] Oct 30 17:14:07.154827 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:07.154789 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:08.412547 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:08.412512 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa9db899-8ed9-4944-8c2c-9ffea59595d1" path="/var/lib/kubelet/pods/fa9db899-8ed9-4944-8c2c-9ffea59595d1/volumes" Oct 30 17:14:08.934351 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:08.934303 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:09.044386 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:09.044351 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:10.826090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:10.826050 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:12.858095 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:12.858055 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:12.969931 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:12.969900 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:13.171613 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.165060 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm"] Oct 30 17:14:13.171613 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.165274 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" podUID="05ae3695-e79b-410d-86a3-06151a2c8038" containerName="istio-proxy" containerID="cri-o://957c7bcfc148b69b7f84a985154b13481ae133011345956d2adac35d821817bf" gracePeriod=30 Oct 30 17:14:13.287831 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.287727 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:13.616835 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.616756 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:13.745414 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.745377 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8"] Oct 30 17:14:13.746234 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.746215 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3082585b-6424-472e-8014-0eb3b5ad15a8" containerName="revoke-keys" Oct 30 17:14:13.746318 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.746235 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3082585b-6424-472e-8014-0eb3b5ad15a8" containerName="revoke-keys" Oct 30 17:14:13.746318 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.746278 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a" containerName="revoke-keys" Oct 30 17:14:13.746318 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.746283 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a" containerName="revoke-keys" Oct 30 17:14:13.746471 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.746385 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a" containerName="revoke-keys" Oct 30 17:14:13.746471 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.746398 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3082585b-6424-472e-8014-0eb3b5ad15a8" containerName="revoke-keys" Oct 30 17:14:13.752432 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.752405 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8" Oct 30 17:14:13.770914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.770886 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8"] Oct 30 17:14:13.771220 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.771205 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:14:13.771220 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.771204 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:14:13.774685 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.774662 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:14:13.778926 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.778906 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:14:13.895541 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.895456 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8\" (UID: \"f6bf0eb3-cf41-4fb0-9545-17307559ffe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8" Oct 30 17:14:13.895983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.895640 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8\" (UID: \"f6bf0eb3-cf41-4fb0-9545-17307559ffe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8" Oct 30 17:14:13.895983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.895737 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frwjw\" (UniqueName: \"kubernetes.io/projected/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-kube-api-access-frwjw\") pod \"maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8\" (UID: \"f6bf0eb3-cf41-4fb0-9545-17307559ffe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8" Oct 30 17:14:13.997336 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.997297 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8\" (UID: \"f6bf0eb3-cf41-4fb0-9545-17307559ffe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8" Oct 30 17:14:13.997520 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.997371 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwjw\" (UniqueName: \"kubernetes.io/projected/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-kube-api-access-frwjw\") pod \"maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8\" (UID: \"f6bf0eb3-cf41-4fb0-9545-17307559ffe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8" Oct 30 17:14:13.997570 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.997553 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8\" (UID: \"f6bf0eb3-cf41-4fb0-9545-17307559ffe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8" Oct 30 17:14:13.998143 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.998124 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8\" (UID: \"f6bf0eb3-cf41-4fb0-9545-17307559ffe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8" Oct 30 17:14:13.999880 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:13.999858 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8\" (UID: \"f6bf0eb3-cf41-4fb0-9545-17307559ffe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8" Oct 30 17:14:14.022582 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:14.022547 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frwjw\" (UniqueName: \"kubernetes.io/projected/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-kube-api-access-frwjw\") pod \"maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8\" (UID: \"f6bf0eb3-cf41-4fb0-9545-17307559ffe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8" Oct 30 17:14:14.038153 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:14.038128 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs"] Oct 30 17:14:14.042005 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:14.041979 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6640d8-kntqs"] Oct 30 17:14:14.070936 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:14:14.070904 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:14:14.072205 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:14.072190 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8" Oct 30 17:14:14.375818 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:14:14.375784 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6bf0eb3_cf41_4fb0_9545_17307559ffe0.slice/crio-eb6abc0e552ba3b4df1617c1478eba8435aa914a5a474793b67de972d948eadb WatchSource:0}: Error finding container eb6abc0e552ba3b4df1617c1478eba8435aa914a5a474793b67de972d948eadb: Status 404 returned error can't find the container with id eb6abc0e552ba3b4df1617c1478eba8435aa914a5a474793b67de972d948eadb Oct 30 17:14:14.377304 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:14.377277 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8"] Oct 30 17:14:14.413307 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:14.413275 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2175025-c963-4a01-9c1e-d8fdff9cc0ae" path="/var/lib/kubelet/pods/c2175025-c963-4a01-9c1e-d8fdff9cc0ae/volumes" Oct 30 17:14:14.530115 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:14.530072 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8" event={"ID":"f6bf0eb3-cf41-4fb0-9545-17307559ffe0","Type":"ContainerStarted","Data":"eb6abc0e552ba3b4df1617c1478eba8435aa914a5a474793b67de972d948eadb"} Oct 30 17:14:15.031947 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:15.031857 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g"] Oct 30 17:14:15.046247 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:15.046217 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbed3744-d5g5g"] Oct 30 17:14:15.543167 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:15.543126 6202 generic.go:356] "Generic (PLEG): container finished" podID="f6bf0eb3-cf41-4fb0-9545-17307559ffe0" containerID="855e06add1bf915a460a51ea58d0b9ff41f972ca5081e9041280c86d607d223b" exitCode=0 Oct 30 17:14:15.543368 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:15.543208 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8" event={"ID":"f6bf0eb3-cf41-4fb0-9545-17307559ffe0","Type":"ContainerDied","Data":"855e06add1bf915a460a51ea58d0b9ff41f972ca5081e9041280c86d607d223b"} Oct 30 17:14:16.184177 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:16.184141 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:16.412938 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:16.412908 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a339742e-1b29-4044-9908-0304b50ba5e2" path="/var/lib/kubelet/pods/a339742e-1b29-4044-9908-0304b50ba5e2/volumes" Oct 30 17:14:16.739618 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:16.739590 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8" Oct 30 17:14:16.844064 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:16.844029 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-kube-api-access-frwjw\" (UniqueName: \"kubernetes.io/projected/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-kube-api-access-frwjw\") pod \"f6bf0eb3-cf41-4fb0-9545-17307559ffe0\" (UID: \"f6bf0eb3-cf41-4fb0-9545-17307559ffe0\") " Oct 30 17:14:16.844273 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:16.844239 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-maas-api-openshift-ca\") pod \"f6bf0eb3-cf41-4fb0-9545-17307559ffe0\" (UID: \"f6bf0eb3-cf41-4fb0-9545-17307559ffe0\") " Oct 30 17:14:16.844348 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:16.844283 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-maas-api-xks-ca\") pod \"f6bf0eb3-cf41-4fb0-9545-17307559ffe0\" (UID: \"f6bf0eb3-cf41-4fb0-9545-17307559ffe0\") " Oct 30 17:14:16.844677 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:16.844643 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-maas-api-openshift-ca" pod "f6bf0eb3-cf41-4fb0-9545-17307559ffe0" (UID: "f6bf0eb3-cf41-4fb0-9545-17307559ffe0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:16.844952 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:16.844934 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:16.846597 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:16.846561 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-kube-api-access-frwjw" pod "f6bf0eb3-cf41-4fb0-9545-17307559ffe0" (UID: "f6bf0eb3-cf41-4fb0-9545-17307559ffe0"). InnerVolumeSpecName "kube-api-access-frwjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:16.847022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:16.846991 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-maas-api-xks-ca" pod "f6bf0eb3-cf41-4fb0-9545-17307559ffe0" (UID: "f6bf0eb3-cf41-4fb0-9545-17307559ffe0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:16.946289 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:16.946247 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frwjw\" (UniqueName: \"kubernetes.io/projected/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-kube-api-access-frwjw\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:16.946289 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:16.946288 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6bf0eb3-cf41-4fb0-9545-17307559ffe0-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:17.585735 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:17.585689 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8" event={"ID":"f6bf0eb3-cf41-4fb0-9545-17307559ffe0","Type":"ContainerDied","Data":"eb6abc0e552ba3b4df1617c1478eba8435aa914a5a474793b67de972d948eadb"} Oct 30 17:14:17.586657 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:17.586626 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb6abc0e552ba3b4df1617c1478eba8435aa914a5a474793b67de972d948eadb" Oct 30 17:14:17.586922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:17.586223 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8" Oct 30 17:14:17.860137 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:17.860043 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:18.031496 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.031462 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp"] Oct 30 17:14:18.036235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.036206 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c2d9bc-cpdjp"] Oct 30 17:14:18.295549 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.295511 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body= Oct 30 17:14:18.295690 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.295572 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" podUID="05ae3695-e79b-410d-86a3-06151a2c8038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" Oct 30 17:14:18.417487 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.417447 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="326044d6-f5fb-4f8d-b348-f2c4f41b5cc7" path="/var/lib/kubelet/pods/326044d6-f5fb-4f8d-b348-f2c4f41b5cc7/volumes" Oct 30 17:14:18.508672 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.508648 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:14:18.599418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.599381 6202 generic.go:356] "Generic (PLEG): container finished" podID="05ae3695-e79b-410d-86a3-06151a2c8038" containerID="957c7bcfc148b69b7f84a985154b13481ae133011345956d2adac35d821817bf" exitCode=0 Oct 30 17:14:18.599959 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.599443 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" Oct 30 17:14:18.599959 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.599504 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" event={"ID":"05ae3695-e79b-410d-86a3-06151a2c8038","Type":"ContainerDied","Data":"957c7bcfc148b69b7f84a985154b13481ae133011345956d2adac35d821817bf"} Oct 30 17:14:18.599959 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.599555 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm" event={"ID":"05ae3695-e79b-410d-86a3-06151a2c8038","Type":"ContainerDied","Data":"02bc56bf6460affc36bcf36ada31ea83d394f886690cb7ed542f2179ebbbcdb1"} Oct 30 17:14:18.599959 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.599587 6202 scope.go:122] "RemoveContainer" containerID="957c7bcfc148b69b7f84a985154b13481ae133011345956d2adac35d821817bf" Oct 30 17:14:18.612976 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.612951 6202 scope.go:122] "RemoveContainer" containerID="957c7bcfc148b69b7f84a985154b13481ae133011345956d2adac35d821817bf" Oct 30 17:14:18.613332 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:14:18.613305 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"957c7bcfc148b69b7f84a985154b13481ae133011345956d2adac35d821817bf\": container with ID starting with 957c7bcfc148b69b7f84a985154b13481ae133011345956d2adac35d821817bf not found: ID does not exist" containerID="957c7bcfc148b69b7f84a985154b13481ae133011345956d2adac35d821817bf" Oct 30 17:14:18.613451 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.613339 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"957c7bcfc148b69b7f84a985154b13481ae133011345956d2adac35d821817bf"} err="failed to get container status \"957c7bcfc148b69b7f84a985154b13481ae133011345956d2adac35d821817bf\": rpc error: code = NotFound desc = could not find container \"957c7bcfc148b69b7f84a985154b13481ae133011345956d2adac35d821817bf\": container with ID starting with 957c7bcfc148b69b7f84a985154b13481ae133011345956d2adac35d821817bf not found: ID does not exist" Oct 30 17:14:18.675009 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.674966 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ae3695-e79b-410d-86a3-06151a2c8038-kube-api-access-rr9g4\" (UniqueName: \"kubernetes.io/projected/05ae3695-e79b-410d-86a3-06151a2c8038-kube-api-access-rr9g4\") pod \"05ae3695-e79b-410d-86a3-06151a2c8038\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " Oct 30 17:14:18.675224 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.675039 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05ae3695-e79b-410d-86a3-06151a2c8038-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ae3695-e79b-410d-86a3-06151a2c8038-istiod-ca-cert\") pod \"05ae3695-e79b-410d-86a3-06151a2c8038\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " Oct 30 17:14:18.675224 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.675189 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-workload-certs\") pod \"05ae3695-e79b-410d-86a3-06151a2c8038\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " Oct 30 17:14:18.675373 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.675222 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ae3695-e79b-410d-86a3-06151a2c8038-istio-token\" (UniqueName: \"kubernetes.io/projected/05ae3695-e79b-410d-86a3-06151a2c8038-istio-token\") pod \"05ae3695-e79b-410d-86a3-06151a2c8038\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " Oct 30 17:14:18.675373 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.675354 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05ae3695-e79b-410d-86a3-06151a2c8038-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ae3695-e79b-410d-86a3-06151a2c8038-istio-podinfo\") pod \"05ae3695-e79b-410d-86a3-06151a2c8038\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " Oct 30 17:14:18.675494 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.675429 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-credential-socket\") pod \"05ae3695-e79b-410d-86a3-06151a2c8038\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " Oct 30 17:14:18.675494 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.675465 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-istio-envoy\") pod \"05ae3695-e79b-410d-86a3-06151a2c8038\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " Oct 30 17:14:18.675633 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.675493 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-istio-data\") pod \"05ae3695-e79b-410d-86a3-06151a2c8038\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " Oct 30 17:14:18.675633 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.675546 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-workload-socket\") pod \"05ae3695-e79b-410d-86a3-06151a2c8038\" (UID: \"05ae3695-e79b-410d-86a3-06151a2c8038\") " Oct 30 17:14:18.676609 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.676494 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-workload-socket" pod "05ae3695-e79b-410d-86a3-06151a2c8038" (UID: "05ae3695-e79b-410d-86a3-06151a2c8038"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.676836 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.676812 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-workload-certs" pod "05ae3695-e79b-410d-86a3-06151a2c8038" (UID: "05ae3695-e79b-410d-86a3-06151a2c8038"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.677025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.676998 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-istio-data" pod "05ae3695-e79b-410d-86a3-06151a2c8038" (UID: "05ae3695-e79b-410d-86a3-06151a2c8038"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.677814 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.677746 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05ae3695-e79b-410d-86a3-06151a2c8038-istiod-ca-cert" pod "05ae3695-e79b-410d-86a3-06151a2c8038" (UID: "05ae3695-e79b-410d-86a3-06151a2c8038"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:18.678078 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.678050 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-credential-socket" pod "05ae3695-e79b-410d-86a3-06151a2c8038" (UID: "05ae3695-e79b-410d-86a3-06151a2c8038"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.679845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.679821 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05ae3695-e79b-410d-86a3-06151a2c8038-istio-podinfo" pod "05ae3695-e79b-410d-86a3-06151a2c8038" (UID: "05ae3695-e79b-410d-86a3-06151a2c8038"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:14:18.680438 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.680412 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ae3695-e79b-410d-86a3-06151a2c8038-kube-api-access-rr9g4" pod "05ae3695-e79b-410d-86a3-06151a2c8038" (UID: "05ae3695-e79b-410d-86a3-06151a2c8038"). InnerVolumeSpecName "kube-api-access-rr9g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:18.681618 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.681591 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ae3695-e79b-410d-86a3-06151a2c8038-istio-token" pod "05ae3695-e79b-410d-86a3-06151a2c8038" (UID: "05ae3695-e79b-410d-86a3-06151a2c8038"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:18.681720 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.681630 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-istio-envoy" pod "05ae3695-e79b-410d-86a3-06151a2c8038" (UID: "05ae3695-e79b-410d-86a3-06151a2c8038"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.777539 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.777453 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr9g4\" (UniqueName: \"kubernetes.io/projected/05ae3695-e79b-410d-86a3-06151a2c8038-kube-api-access-rr9g4\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.777539 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.777488 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ae3695-e79b-410d-86a3-06151a2c8038-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.777539 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.777498 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.777539 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.777506 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ae3695-e79b-410d-86a3-06151a2c8038-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.777539 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.777515 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ae3695-e79b-410d-86a3-06151a2c8038-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.777539 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.777523 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.777539 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.777532 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.777539 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.777540 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.777539 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.777547 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ae3695-e79b-410d-86a3-06151a2c8038-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.950199 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.950156 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm"] Oct 30 17:14:18.963099 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:18.963066 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-nqjqm"] Oct 30 17:14:20.043561 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:20.043517 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:20.412042 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:20.412011 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05ae3695-e79b-410d-86a3-06151a2c8038" path="/var/lib/kubelet/pods/05ae3695-e79b-410d-86a3-06151a2c8038/volumes" Oct 30 17:14:21.980329 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:21.980286 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:23.675378 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:23.675336 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:25.229337 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.229304 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:25.315618 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.315588 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg"] Oct 30 17:14:25.316178 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.316141 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg" podUID="2dad0e9c-8af9-45d7-b040-f144b1aac81d" containerName="payload-pre-processing" containerID="cri-o://c9b202937c906a5408346b9ac913cb5f3cee367036ed4b01f624565fcb3c349b" gracePeriod=30 Oct 30 17:14:25.404692 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.404647 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:25.548521 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.548469 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:25.633454 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.633424 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg" Oct 30 17:14:25.803775 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.802317 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:25.811087 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.809919 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dad0e9c-8af9-45d7-b040-f144b1aac81d-kube-api-access-4zpbb\" (UniqueName: \"kubernetes.io/projected/2dad0e9c-8af9-45d7-b040-f144b1aac81d-kube-api-access-4zpbb\") pod \"2dad0e9c-8af9-45d7-b040-f144b1aac81d\" (UID: \"2dad0e9c-8af9-45d7-b040-f144b1aac81d\") " Oct 30 17:14:25.811087 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.810144 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dad0e9c-8af9-45d7-b040-f144b1aac81d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dad0e9c-8af9-45d7-b040-f144b1aac81d-plugins-config-volume\") pod \"2dad0e9c-8af9-45d7-b040-f144b1aac81d\" (UID: \"2dad0e9c-8af9-45d7-b040-f144b1aac81d\") " Oct 30 17:14:25.811803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.811486 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dad0e9c-8af9-45d7-b040-f144b1aac81d-plugins-config-volume" pod "2dad0e9c-8af9-45d7-b040-f144b1aac81d" (UID: "2dad0e9c-8af9-45d7-b040-f144b1aac81d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:25.814625 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.814589 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dad0e9c-8af9-45d7-b040-f144b1aac81d-kube-api-access-4zpbb" pod "2dad0e9c-8af9-45d7-b040-f144b1aac81d" (UID: "2dad0e9c-8af9-45d7-b040-f144b1aac81d"). InnerVolumeSpecName "kube-api-access-4zpbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:25.860828 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.858907 6202 generic.go:356] "Generic (PLEG): container finished" podID="2dad0e9c-8af9-45d7-b040-f144b1aac81d" containerID="c9b202937c906a5408346b9ac913cb5f3cee367036ed4b01f624565fcb3c349b" exitCode=0 Oct 30 17:14:25.860828 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.858979 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg" event={"ID":"2dad0e9c-8af9-45d7-b040-f144b1aac81d","Type":"ContainerDied","Data":"c9b202937c906a5408346b9ac913cb5f3cee367036ed4b01f624565fcb3c349b"} Oct 30 17:14:25.860828 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.859060 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg" event={"ID":"2dad0e9c-8af9-45d7-b040-f144b1aac81d","Type":"ContainerDied","Data":"71744aab5a7ceeb926b7171f380d5153284d41cf886974d417ffc7ffab08ae9b"} Oct 30 17:14:25.860828 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.859095 6202 scope.go:122] "RemoveContainer" containerID="c9b202937c906a5408346b9ac913cb5f3cee367036ed4b01f624565fcb3c349b" Oct 30 17:14:25.860828 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.859291 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg" Oct 30 17:14:25.912964 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.912872 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dad0e9c-8af9-45d7-b040-f144b1aac81d-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:25.912964 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.912904 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zpbb\" (UniqueName: \"kubernetes.io/projected/2dad0e9c-8af9-45d7-b040-f144b1aac81d-kube-api-access-4zpbb\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:25.914612 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.914580 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:25.916838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.915592 6202 scope.go:122] "RemoveContainer" containerID="c9b202937c906a5408346b9ac913cb5f3cee367036ed4b01f624565fcb3c349b" Oct 30 17:14:25.916838 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:14:25.916374 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b202937c906a5408346b9ac913cb5f3cee367036ed4b01f624565fcb3c349b\": container with ID starting with c9b202937c906a5408346b9ac913cb5f3cee367036ed4b01f624565fcb3c349b not found: ID does not exist" containerID="c9b202937c906a5408346b9ac913cb5f3cee367036ed4b01f624565fcb3c349b" Oct 30 17:14:25.916838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.916413 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b202937c906a5408346b9ac913cb5f3cee367036ed4b01f624565fcb3c349b"} err="failed to get container status \"c9b202937c906a5408346b9ac913cb5f3cee367036ed4b01f624565fcb3c349b\": rpc error: code = NotFound desc = could not find container \"c9b202937c906a5408346b9ac913cb5f3cee367036ed4b01f624565fcb3c349b\": container with ID starting with c9b202937c906a5408346b9ac913cb5f3cee367036ed4b01f624565fcb3c349b not found: ID does not exist" Oct 30 17:14:25.923726 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.923420 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv"] Oct 30 17:14:25.927291 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.927266 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-55e83e-c2qlv"] Oct 30 17:14:25.952678 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.952643 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg"] Oct 30 17:14:25.956367 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:25.956343 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55e83e-b6f8b6695-5wjvg"] Oct 30 17:14:26.005878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.000843 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm"] Oct 30 17:14:26.007549 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.007491 6202 cpu_manager.go:414] "containerMap: removing container" podUID="2dad0e9c-8af9-45d7-b040-f144b1aac81d" containerName="payload-pre-processing" Oct 30 17:14:26.007760 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.007744 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dad0e9c-8af9-45d7-b040-f144b1aac81d" containerName="payload-pre-processing" Oct 30 17:14:26.007954 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.007935 6202 cpu_manager.go:414] "containerMap: removing container" podUID="05ae3695-e79b-410d-86a3-06151a2c8038" containerName="istio-proxy" Oct 30 17:14:26.008076 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.008061 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ae3695-e79b-410d-86a3-06151a2c8038" containerName="istio-proxy" Oct 30 17:14:26.008192 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.008179 6202 cpu_manager.go:414] "containerMap: removing container" podUID="f6bf0eb3-cf41-4fb0-9545-17307559ffe0" containerName="revoke-keys" Oct 30 17:14:26.008285 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.008270 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6bf0eb3-cf41-4fb0-9545-17307559ffe0" containerName="revoke-keys" Oct 30 17:14:26.008633 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.008614 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dad0e9c-8af9-45d7-b040-f144b1aac81d" containerName="payload-pre-processing" Oct 30 17:14:26.008789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.008757 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="05ae3695-e79b-410d-86a3-06151a2c8038" containerName="istio-proxy" Oct 30 17:14:26.008903 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.008891 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6bf0eb3-cf41-4fb0-9545-17307559ffe0" containerName="revoke-keys" Oct 30 17:14:26.015943 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.015920 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.023725 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.023698 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm"] Oct 30 17:14:26.032112 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.032077 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-967fac-openshift-default-dockercfg-lfjgc\"" Oct 30 17:14:26.115254 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.115222 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.115444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.115265 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.115444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.115393 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.115526 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.115445 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.115526 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.115503 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.115601 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.115587 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4n5q\" (UniqueName: \"kubernetes.io/projected/e0a6beae-02be-487a-980a-c69c3ce2bc9e-kube-api-access-h4n5q\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.115654 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.115641 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.115693 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.115671 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.115740 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.115704 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.217225 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.217190 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.217449 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.217264 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.217449 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.217328 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.217449 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.217395 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4n5q\" (UniqueName: \"kubernetes.io/projected/e0a6beae-02be-487a-980a-c69c3ce2bc9e-kube-api-access-h4n5q\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.217638 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.217459 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.217638 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.217486 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.217638 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.217532 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.217638 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.217565 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.217638 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.217614 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.217973 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.217682 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.217973 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.217753 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.217973 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.217924 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.218092 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.218040 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.218255 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.218228 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.220113 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.220089 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.220306 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.220282 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.242086 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.242059 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.242547 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.242111 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4n5q\" (UniqueName: \"kubernetes.io/projected/e0a6beae-02be-487a-980a-c69c3ce2bc9e-kube-api-access-h4n5q\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.332785 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.332744 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:26.417259 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.417221 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:26.422059 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.422035 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dad0e9c-8af9-45d7-b040-f144b1aac81d" path="/var/lib/kubelet/pods/2dad0e9c-8af9-45d7-b040-f144b1aac81d/volumes" Oct 30 17:14:26.422635 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.422620 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3537b1e-c21c-47d2-88de-821acfb3f34a" path="/var/lib/kubelet/pods/c3537b1e-c21c-47d2-88de-821acfb3f34a/volumes" Oct 30 17:14:26.661146 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:14:26.661075 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a6beae_02be_487a_980a_c69c3ce2bc9e.slice/crio-faee1f25bbe2ae5dfe62f23bdd8d3f31ca3bb96b9c7334484d38d0bd7b7313a0 WatchSource:0}: Error finding container faee1f25bbe2ae5dfe62f23bdd8d3f31ca3bb96b9c7334484d38d0bd7b7313a0: Status 404 returned error can't find the container with id faee1f25bbe2ae5dfe62f23bdd8d3f31ca3bb96b9c7334484d38d0bd7b7313a0 Oct 30 17:14:26.662928 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.662889 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm"] Oct 30 17:14:26.665049 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.665008 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:14:26.665165 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.665097 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:14:26.665274 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.665166 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:14:26.877361 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.877320 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" event={"ID":"e0a6beae-02be-487a-980a-c69c3ce2bc9e","Type":"ContainerStarted","Data":"adcdaccf7f75090eb495a281b449684e9cf3a5c3ca138c8b5edb9d0ef86f1990"} Oct 30 17:14:26.877574 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.877406 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" event={"ID":"e0a6beae-02be-487a-980a-c69c3ce2bc9e","Type":"ContainerStarted","Data":"faee1f25bbe2ae5dfe62f23bdd8d3f31ca3bb96b9c7334484d38d0bd7b7313a0"} Oct 30 17:14:26.927923 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:26.927816 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" podStartSLOduration=1.9278017520000001 podStartE2EDuration="1.927801752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:26.925423635 +0000 UTC m=+3180.635878379" watchObservedRunningTime="2026-10-30 17:14:26.927801752 +0000 UTC m=+3180.638256496" Oct 30 17:14:27.333951 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:27.333915 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:27.335456 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:27.335428 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Oct 30 17:14:27.335615 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:27.335485 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" podUID="e0a6beae-02be-487a-980a-c69c3ce2bc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Oct 30 17:14:27.669344 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:27.669212 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:28.343559 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:28.343505 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Oct 30 17:14:28.344647 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:28.344612 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" podUID="e0a6beae-02be-487a-980a-c69c3ce2bc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Oct 30 17:14:28.625354 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:28.625251 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:28.775807 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:28.775759 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:28.824271 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:28.824233 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv"] Oct 30 17:14:28.837299 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:28.837273 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-967fac-m85tv"] Oct 30 17:14:28.884898 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:28.884813 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:29.333910 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:29.333872 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Oct 30 17:14:29.334117 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:29.333943 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" podUID="e0a6beae-02be-487a-980a-c69c3ce2bc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Oct 30 17:14:29.368093 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:29.368058 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:30.334164 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:30.334127 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Oct 30 17:14:30.334369 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:30.334190 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" podUID="e0a6beae-02be-487a-980a-c69c3ce2bc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Oct 30 17:14:30.414679 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:30.414642 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e6af0aa-cdab-4e6b-bbc3-526caee0b65d" path="/var/lib/kubelet/pods/5e6af0aa-cdab-4e6b-bbc3-526caee0b65d/volumes" Oct 30 17:14:30.711178 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:30.711093 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:31.333659 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:31.333623 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Oct 30 17:14:31.333881 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:31.333679 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" podUID="e0a6beae-02be-487a-980a-c69c3ce2bc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Oct 30 17:14:31.467269 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:31.467231 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:32.333761 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:32.333722 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Oct 30 17:14:32.334044 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:32.333808 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" podUID="e0a6beae-02be-487a-980a-c69c3ce2bc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Oct 30 17:14:33.000714 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:33.000675 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:33.333678 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:33.333638 6202 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body= Oct 30 17:14:33.333879 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:33.333709 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" podUID="e0a6beae-02be-487a-980a-c69c3ce2bc9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" Oct 30 17:14:34.159756 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:34.159723 6202 scope.go:122] "RemoveContainer" containerID="dba2e9da5c056d578af128e805dc441dbc1c37ace9f46aa9a713b93d465e3396" Oct 30 17:14:34.174012 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:34.173984 6202 scope.go:122] "RemoveContainer" containerID="811a9c382f91a857c074ecbba4c74f404110c48fb509a9bd0c69398932926663" Oct 30 17:14:34.687295 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:34.687257 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:34.780195 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:34.780145 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:34.781690 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:34.781659 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:34.782790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:34.782723 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:35.420286 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:35.420246 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:35.537053 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:35.537011 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:35.738464 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:35.738367 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm"] Oct 30 17:14:35.774644 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:35.774595 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:36.043264 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:36.043179 6202 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" secret="" err="secret \"e2e-worker-w5-967fac-openshift-default-dockercfg-lfjgc\" not found" Oct 30 17:14:36.533804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:36.533744 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:37.052369 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:37.052328 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" podUID="e0a6beae-02be-487a-980a-c69c3ce2bc9e" containerName="istio-proxy" containerID="cri-o://adcdaccf7f75090eb495a281b449684e9cf3a5c3ca138c8b5edb9d0ef86f1990" gracePeriod=30 Oct 30 17:14:38.370107 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:38.370065 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:40.781378 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:40.781335 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:41.681476 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:41.681441 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:41.804327 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:41.804287 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:41.914960 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:41.914924 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:42.443031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.443005 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:42.587569 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.587456 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:42.594619 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.594592 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istiod-ca-cert\") pod \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " Oct 30 17:14:42.594800 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.594633 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-workload-socket\") pod \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " Oct 30 17:14:42.594800 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.594673 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-podinfo\") pod \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " Oct 30 17:14:42.594800 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.594759 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-envoy\") pod \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " Oct 30 17:14:42.594990 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.594826 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-token\" (UniqueName: \"kubernetes.io/projected/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-token\") pod \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " Oct 30 17:14:42.594990 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.594905 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a6beae-02be-487a-980a-c69c3ce2bc9e-kube-api-access-h4n5q\" (UniqueName: \"kubernetes.io/projected/e0a6beae-02be-487a-980a-c69c3ce2bc9e-kube-api-access-h4n5q\") pod \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " Oct 30 17:14:42.594990 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.594949 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-credential-socket\") pod \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " Oct 30 17:14:42.594990 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.594977 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-workload-certs\") pod \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " Oct 30 17:14:42.595476 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.595011 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-data\") pod \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\" (UID: \"e0a6beae-02be-487a-980a-c69c3ce2bc9e\") " Oct 30 17:14:42.595476 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.595080 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istiod-ca-cert" pod "e0a6beae-02be-487a-980a-c69c3ce2bc9e" (UID: "e0a6beae-02be-487a-980a-c69c3ce2bc9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:42.595476 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.595082 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-workload-socket" pod "e0a6beae-02be-487a-980a-c69c3ce2bc9e" (UID: "e0a6beae-02be-487a-980a-c69c3ce2bc9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:42.595476 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.595316 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-workload-certs" pod "e0a6beae-02be-487a-980a-c69c3ce2bc9e" (UID: "e0a6beae-02be-487a-980a-c69c3ce2bc9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:42.595476 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.595327 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-credential-socket" pod "e0a6beae-02be-487a-980a-c69c3ce2bc9e" (UID: "e0a6beae-02be-487a-980a-c69c3ce2bc9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:42.595719 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.595517 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-data" pod "e0a6beae-02be-487a-980a-c69c3ce2bc9e" (UID: "e0a6beae-02be-487a-980a-c69c3ce2bc9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:42.595892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.595865 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:42.595956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.595895 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:42.595956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.595905 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:42.595956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.595914 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:42.595956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.595922 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:42.597391 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.597367 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-token" pod "e0a6beae-02be-487a-980a-c69c3ce2bc9e" (UID: "e0a6beae-02be-487a-980a-c69c3ce2bc9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:42.597927 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.597898 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-envoy" pod "e0a6beae-02be-487a-980a-c69c3ce2bc9e" (UID: "e0a6beae-02be-487a-980a-c69c3ce2bc9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:42.597927 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.597908 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-podinfo" pod "e0a6beae-02be-487a-980a-c69c3ce2bc9e" (UID: "e0a6beae-02be-487a-980a-c69c3ce2bc9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:14:42.598061 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.597939 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a6beae-02be-487a-980a-c69c3ce2bc9e-kube-api-access-h4n5q" pod "e0a6beae-02be-487a-980a-c69c3ce2bc9e" (UID: "e0a6beae-02be-487a-980a-c69c3ce2bc9e"). InnerVolumeSpecName "kube-api-access-h4n5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:42.697502 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.697462 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:42.697502 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.697503 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:42.697740 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.697518 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a6beae-02be-487a-980a-c69c3ce2bc9e-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:42.697740 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.697534 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4n5q\" (UniqueName: \"kubernetes.io/projected/e0a6beae-02be-487a-980a-c69c3ce2bc9e-kube-api-access-h4n5q\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:42.699278 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:42.699237 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:43.143053 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:43.143020 6202 generic.go:356] "Generic (PLEG): container finished" podID="e0a6beae-02be-487a-980a-c69c3ce2bc9e" containerID="adcdaccf7f75090eb495a281b449684e9cf3a5c3ca138c8b5edb9d0ef86f1990" exitCode=0 Oct 30 17:14:43.143579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:43.143067 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" Oct 30 17:14:43.143579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:43.143108 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" event={"ID":"e0a6beae-02be-487a-980a-c69c3ce2bc9e","Type":"ContainerDied","Data":"adcdaccf7f75090eb495a281b449684e9cf3a5c3ca138c8b5edb9d0ef86f1990"} Oct 30 17:14:43.143579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:43.143179 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm" event={"ID":"e0a6beae-02be-487a-980a-c69c3ce2bc9e","Type":"ContainerDied","Data":"faee1f25bbe2ae5dfe62f23bdd8d3f31ca3bb96b9c7334484d38d0bd7b7313a0"} Oct 30 17:14:43.143579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:43.143208 6202 scope.go:122] "RemoveContainer" containerID="adcdaccf7f75090eb495a281b449684e9cf3a5c3ca138c8b5edb9d0ef86f1990" Oct 30 17:14:43.157019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:43.157001 6202 scope.go:122] "RemoveContainer" containerID="adcdaccf7f75090eb495a281b449684e9cf3a5c3ca138c8b5edb9d0ef86f1990" Oct 30 17:14:43.157336 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:14:43.157308 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adcdaccf7f75090eb495a281b449684e9cf3a5c3ca138c8b5edb9d0ef86f1990\": container with ID starting with adcdaccf7f75090eb495a281b449684e9cf3a5c3ca138c8b5edb9d0ef86f1990 not found: ID does not exist" containerID="adcdaccf7f75090eb495a281b449684e9cf3a5c3ca138c8b5edb9d0ef86f1990" Oct 30 17:14:43.157430 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:43.157350 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adcdaccf7f75090eb495a281b449684e9cf3a5c3ca138c8b5edb9d0ef86f1990"} err="failed to get container status \"adcdaccf7f75090eb495a281b449684e9cf3a5c3ca138c8b5edb9d0ef86f1990\": rpc error: code = NotFound desc = could not find container \"adcdaccf7f75090eb495a281b449684e9cf3a5c3ca138c8b5edb9d0ef86f1990\": container with ID starting with adcdaccf7f75090eb495a281b449684e9cf3a5c3ca138c8b5edb9d0ef86f1990 not found: ID does not exist" Oct 30 17:14:43.213691 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:43.213655 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm"] Oct 30 17:14:43.220110 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:43.220083 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-4nnwm"] Oct 30 17:14:44.023804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.023740 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:44.072887 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:14:44.072850 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:14:44.122409 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.122366 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv"] Oct 30 17:14:44.122652 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.122627 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv" podUID="55546d54-81a4-402e-8ad5-7110cf03a17c" containerName="payload-processing" containerID="cri-o://b2ee111001cdebadb31555053db5e95ed6a1c04510ac7e068bea3711d64ceb87" gracePeriod=30 Oct 30 17:14:44.148613 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.142686 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4"] Oct 30 17:14:44.148613 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.142924 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4" podUID="66ed3178-5d8b-4cb5-9ec6-737aa118b73a" containerName="payload-pre-processing" containerID="cri-o://7338bbcc995e02b946317d450b09f450cf68b44ddbb457c05934fad8d40d5c22" gracePeriod=30 Oct 30 17:14:44.208966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.208931 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:44.319349 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.319318 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:44.421082 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.421040 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0a6beae-02be-487a-980a-c69c3ce2bc9e" path="/var/lib/kubelet/pods/e0a6beae-02be-487a-980a-c69c3ce2bc9e/volumes" Oct 30 17:14:44.605275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.602605 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:44.659746 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.659703 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv" Oct 30 17:14:44.713683 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.713651 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:44.749517 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.749490 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd"] Oct 30 17:14:44.750939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.750912 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4" Oct 30 17:14:44.754817 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.754794 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-698c71-rkscd"] Oct 30 17:14:44.788852 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.787306 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55546d54-81a4-402e-8ad5-7110cf03a17c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55546d54-81a4-402e-8ad5-7110cf03a17c-plugins-config-volume\") pod \"55546d54-81a4-402e-8ad5-7110cf03a17c\" (UID: \"55546d54-81a4-402e-8ad5-7110cf03a17c\") " Oct 30 17:14:44.802347 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.802305 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55546d54-81a4-402e-8ad5-7110cf03a17c-plugins-config-volume" pod "55546d54-81a4-402e-8ad5-7110cf03a17c" (UID: "55546d54-81a4-402e-8ad5-7110cf03a17c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:44.806183 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.806132 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55546d54-81a4-402e-8ad5-7110cf03a17c-kube-api-access-kt7xj\" (UniqueName: \"kubernetes.io/projected/55546d54-81a4-402e-8ad5-7110cf03a17c-kube-api-access-kt7xj\") pod \"55546d54-81a4-402e-8ad5-7110cf03a17c\" (UID: \"55546d54-81a4-402e-8ad5-7110cf03a17c\") " Oct 30 17:14:44.822404 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.822372 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:44.822599 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.822302 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55546d54-81a4-402e-8ad5-7110cf03a17c-kube-api-access-kt7xj" pod "55546d54-81a4-402e-8ad5-7110cf03a17c" (UID: "55546d54-81a4-402e-8ad5-7110cf03a17c"). InnerVolumeSpecName "kube-api-access-kt7xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:44.908574 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.908484 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ed3178-5d8b-4cb5-9ec6-737aa118b73a-kube-api-access-lmbq6\" (UniqueName: \"kubernetes.io/projected/66ed3178-5d8b-4cb5-9ec6-737aa118b73a-kube-api-access-lmbq6\") pod \"66ed3178-5d8b-4cb5-9ec6-737aa118b73a\" (UID: \"66ed3178-5d8b-4cb5-9ec6-737aa118b73a\") " Oct 30 17:14:44.908820 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.908583 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ed3178-5d8b-4cb5-9ec6-737aa118b73a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ed3178-5d8b-4cb5-9ec6-737aa118b73a-plugins-config-volume\") pod \"66ed3178-5d8b-4cb5-9ec6-737aa118b73a\" (UID: \"66ed3178-5d8b-4cb5-9ec6-737aa118b73a\") " Oct 30 17:14:44.909009 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.908987 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ed3178-5d8b-4cb5-9ec6-737aa118b73a-plugins-config-volume" pod "66ed3178-5d8b-4cb5-9ec6-737aa118b73a" (UID: "66ed3178-5d8b-4cb5-9ec6-737aa118b73a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:44.909076 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.909061 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt7xj\" (UniqueName: \"kubernetes.io/projected/55546d54-81a4-402e-8ad5-7110cf03a17c-kube-api-access-kt7xj\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:44.909117 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.909077 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55546d54-81a4-402e-8ad5-7110cf03a17c-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:44.910883 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:44.910858 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ed3178-5d8b-4cb5-9ec6-737aa118b73a-kube-api-access-lmbq6" pod "66ed3178-5d8b-4cb5-9ec6-737aa118b73a" (UID: "66ed3178-5d8b-4cb5-9ec6-737aa118b73a"). InnerVolumeSpecName "kube-api-access-lmbq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:45.011791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.010934 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmbq6\" (UniqueName: \"kubernetes.io/projected/66ed3178-5d8b-4cb5-9ec6-737aa118b73a-kube-api-access-lmbq6\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:45.011791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.010973 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ed3178-5d8b-4cb5-9ec6-737aa118b73a-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:45.176001 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.175898 6202 generic.go:356] "Generic (PLEG): container finished" podID="66ed3178-5d8b-4cb5-9ec6-737aa118b73a" containerID="7338bbcc995e02b946317d450b09f450cf68b44ddbb457c05934fad8d40d5c22" exitCode=0 Oct 30 17:14:45.176001 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.175950 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4" Oct 30 17:14:45.176548 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.175999 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4" event={"ID":"66ed3178-5d8b-4cb5-9ec6-737aa118b73a","Type":"ContainerDied","Data":"7338bbcc995e02b946317d450b09f450cf68b44ddbb457c05934fad8d40d5c22"} Oct 30 17:14:45.176548 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.176061 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4" event={"ID":"66ed3178-5d8b-4cb5-9ec6-737aa118b73a","Type":"ContainerDied","Data":"da26dad81747a804d1ed620467e41ae98ecf821680b5503cf336a86a62aa7523"} Oct 30 17:14:45.176548 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.176091 6202 scope.go:122] "RemoveContainer" containerID="7338bbcc995e02b946317d450b09f450cf68b44ddbb457c05934fad8d40d5c22" Oct 30 17:14:45.177890 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.177864 6202 generic.go:356] "Generic (PLEG): container finished" podID="55546d54-81a4-402e-8ad5-7110cf03a17c" containerID="b2ee111001cdebadb31555053db5e95ed6a1c04510ac7e068bea3711d64ceb87" exitCode=0 Oct 30 17:14:45.178001 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.177918 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv" Oct 30 17:14:45.178001 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.177972 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv" event={"ID":"55546d54-81a4-402e-8ad5-7110cf03a17c","Type":"ContainerDied","Data":"b2ee111001cdebadb31555053db5e95ed6a1c04510ac7e068bea3711d64ceb87"} Oct 30 17:14:45.178093 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.178009 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv" event={"ID":"55546d54-81a4-402e-8ad5-7110cf03a17c","Type":"ContainerDied","Data":"6fcf90dae61c6752013272d6bc9899dbe4869e8424d3227d9bd7836bfa1c86ab"} Oct 30 17:14:45.189703 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.189647 6202 scope.go:122] "RemoveContainer" containerID="7338bbcc995e02b946317d450b09f450cf68b44ddbb457c05934fad8d40d5c22" Oct 30 17:14:45.190024 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:14:45.189997 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7338bbcc995e02b946317d450b09f450cf68b44ddbb457c05934fad8d40d5c22\": container with ID starting with 7338bbcc995e02b946317d450b09f450cf68b44ddbb457c05934fad8d40d5c22 not found: ID does not exist" containerID="7338bbcc995e02b946317d450b09f450cf68b44ddbb457c05934fad8d40d5c22" Oct 30 17:14:45.190128 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.190030 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7338bbcc995e02b946317d450b09f450cf68b44ddbb457c05934fad8d40d5c22"} err="failed to get container status \"7338bbcc995e02b946317d450b09f450cf68b44ddbb457c05934fad8d40d5c22\": rpc error: code = NotFound desc = could not find container \"7338bbcc995e02b946317d450b09f450cf68b44ddbb457c05934fad8d40d5c22\": container with ID starting with 7338bbcc995e02b946317d450b09f450cf68b44ddbb457c05934fad8d40d5c22 not found: ID does not exist" Oct 30 17:14:45.190128 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.190048 6202 scope.go:122] "RemoveContainer" containerID="b2ee111001cdebadb31555053db5e95ed6a1c04510ac7e068bea3711d64ceb87" Oct 30 17:14:45.200564 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.200543 6202 scope.go:122] "RemoveContainer" containerID="b2ee111001cdebadb31555053db5e95ed6a1c04510ac7e068bea3711d64ceb87" Oct 30 17:14:45.200864 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:14:45.200837 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ee111001cdebadb31555053db5e95ed6a1c04510ac7e068bea3711d64ceb87\": container with ID starting with b2ee111001cdebadb31555053db5e95ed6a1c04510ac7e068bea3711d64ceb87 not found: ID does not exist" containerID="b2ee111001cdebadb31555053db5e95ed6a1c04510ac7e068bea3711d64ceb87" Oct 30 17:14:45.200956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.200874 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ee111001cdebadb31555053db5e95ed6a1c04510ac7e068bea3711d64ceb87"} err="failed to get container status \"b2ee111001cdebadb31555053db5e95ed6a1c04510ac7e068bea3711d64ceb87\": rpc error: code = NotFound desc = could not find container \"b2ee111001cdebadb31555053db5e95ed6a1c04510ac7e068bea3711d64ceb87\": container with ID starting with b2ee111001cdebadb31555053db5e95ed6a1c04510ac7e068bea3711d64ceb87 not found: ID does not exist" Oct 30 17:14:45.233880 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.233851 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv"] Oct 30 17:14:45.238442 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.238416 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-698c71-78f7cc5fd8-dxkrv"] Oct 30 17:14:45.276741 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.276697 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4"] Oct 30 17:14:45.282316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:45.282289 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-698c71-754c6f77dc-kjmx4"] Oct 30 17:14:46.412721 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:46.412680 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55546d54-81a4-402e-8ad5-7110cf03a17c" path="/var/lib/kubelet/pods/55546d54-81a4-402e-8ad5-7110cf03a17c/volumes" Oct 30 17:14:46.413302 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:46.413279 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a" path="/var/lib/kubelet/pods/5a655f6d-9e7c-4a6f-8dd1-9666d9fb392a/volumes" Oct 30 17:14:46.413844 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:46.413821 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ed3178-5d8b-4cb5-9ec6-737aa118b73a" path="/var/lib/kubelet/pods/66ed3178-5d8b-4cb5-9ec6-737aa118b73a/volumes" Oct 30 17:14:46.802012 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:46.801928 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:48.552071 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:48.552013 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:50.176178 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:50.176142 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:51.985131 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:51.985091 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:53.650422 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:53.650374 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:55.057697 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.057663 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:55.173868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.173826 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4"] Oct 30 17:14:55.174107 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.174065 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4" podUID="16c3ab98-a5d7-4895-bdf0-5f584c1f510c" containerName="payload-pre-processing" containerID="cri-o://af61721dbb7cbb0643728dc5c046bc8f857ef1a71b054cc7c139f5308b143001" gracePeriod=30 Oct 30 17:14:55.243182 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.243148 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:55.308379 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.308315 6202 generic.go:356] "Generic (PLEG): container finished" podID="16c3ab98-a5d7-4895-bdf0-5f584c1f510c" containerID="af61721dbb7cbb0643728dc5c046bc8f857ef1a71b054cc7c139f5308b143001" exitCode=0 Oct 30 17:14:55.308516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.308383 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4" event={"ID":"16c3ab98-a5d7-4895-bdf0-5f584c1f510c","Type":"ContainerDied","Data":"af61721dbb7cbb0643728dc5c046bc8f857ef1a71b054cc7c139f5308b143001"} Oct 30 17:14:55.354849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.354807 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:55.504084 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.504056 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:55.504674 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.504658 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4" Oct 30 17:14:55.614204 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.614170 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16c3ab98-a5d7-4895-bdf0-5f584c1f510c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16c3ab98-a5d7-4895-bdf0-5f584c1f510c-plugins-config-volume\") pod \"16c3ab98-a5d7-4895-bdf0-5f584c1f510c\" (UID: \"16c3ab98-a5d7-4895-bdf0-5f584c1f510c\") " Oct 30 17:14:55.614416 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.614359 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c3ab98-a5d7-4895-bdf0-5f584c1f510c-kube-api-access-6pfqz\" (UniqueName: \"kubernetes.io/projected/16c3ab98-a5d7-4895-bdf0-5f584c1f510c-kube-api-access-6pfqz\") pod \"16c3ab98-a5d7-4895-bdf0-5f584c1f510c\" (UID: \"16c3ab98-a5d7-4895-bdf0-5f584c1f510c\") " Oct 30 17:14:55.614821 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.614590 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c3ab98-a5d7-4895-bdf0-5f584c1f510c-plugins-config-volume" pod "16c3ab98-a5d7-4895-bdf0-5f584c1f510c" (UID: "16c3ab98-a5d7-4895-bdf0-5f584c1f510c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:55.615018 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.614995 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16c3ab98-a5d7-4895-bdf0-5f584c1f510c-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:55.617423 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.617392 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c3ab98-a5d7-4895-bdf0-5f584c1f510c-kube-api-access-6pfqz" pod "16c3ab98-a5d7-4895-bdf0-5f584c1f510c" (UID: "16c3ab98-a5d7-4895-bdf0-5f584c1f510c"). InnerVolumeSpecName "kube-api-access-6pfqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:55.629235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.629174 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:55.632264 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.632235 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks"] Oct 30 17:14:55.632509 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.632465 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks" podUID="df7a2ad9-d72a-428a-b638-eac0d32e7751" containerName="payload-pre-processing" containerID="cri-o://b3ec86806cd8459ec395bff43395dc7d147f8dd8574ffe5bcf7593a922d13fa9" gracePeriod=30 Oct 30 17:14:55.723247 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.723208 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pfqz\" (UniqueName: \"kubernetes.io/projected/16c3ab98-a5d7-4895-bdf0-5f584c1f510c-kube-api-access-6pfqz\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:55.742356 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.742301 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:55.757583 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.757541 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9"] Oct 30 17:14:55.765064 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.763758 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac125-4v4d9"] Oct 30 17:14:56.001792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:55.983980 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks" Oct 30 17:14:56.133801 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.132564 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df7a2ad9-d72a-428a-b638-eac0d32e7751-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df7a2ad9-d72a-428a-b638-eac0d32e7751-plugins-config-volume\") pod \"df7a2ad9-d72a-428a-b638-eac0d32e7751\" (UID: \"df7a2ad9-d72a-428a-b638-eac0d32e7751\") " Oct 30 17:14:56.133801 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.132629 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df7a2ad9-d72a-428a-b638-eac0d32e7751-kube-api-access-klhl8\" (UniqueName: \"kubernetes.io/projected/df7a2ad9-d72a-428a-b638-eac0d32e7751-kube-api-access-klhl8\") pod \"df7a2ad9-d72a-428a-b638-eac0d32e7751\" (UID: \"df7a2ad9-d72a-428a-b638-eac0d32e7751\") " Oct 30 17:14:56.152386 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.152347 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df7a2ad9-d72a-428a-b638-eac0d32e7751-plugins-config-volume" pod "df7a2ad9-d72a-428a-b638-eac0d32e7751" (UID: "df7a2ad9-d72a-428a-b638-eac0d32e7751"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:56.184439 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.184375 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7a2ad9-d72a-428a-b638-eac0d32e7751-kube-api-access-klhl8" pod "df7a2ad9-d72a-428a-b638-eac0d32e7751" (UID: "df7a2ad9-d72a-428a-b638-eac0d32e7751"). InnerVolumeSpecName "kube-api-access-klhl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:56.201526 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.201448 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:56.234265 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.234233 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df7a2ad9-d72a-428a-b638-eac0d32e7751-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:56.234265 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.234266 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klhl8\" (UniqueName: \"kubernetes.io/projected/df7a2ad9-d72a-428a-b638-eac0d32e7751-kube-api-access-klhl8\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:14:56.317655 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.317621 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:56.318256 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.318240 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4" Oct 30 17:14:56.318319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.318278 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4" event={"ID":"16c3ab98-a5d7-4895-bdf0-5f584c1f510c","Type":"ContainerDied","Data":"b82e9eb7da590565d403973f8e3f1b2118d1fdaadab8cc027a26a2ab5b644f89"} Oct 30 17:14:56.318366 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.318340 6202 scope.go:122] "RemoveContainer" containerID="af61721dbb7cbb0643728dc5c046bc8f857ef1a71b054cc7c139f5308b143001" Oct 30 17:14:56.319965 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.319934 6202 generic.go:356] "Generic (PLEG): container finished" podID="df7a2ad9-d72a-428a-b638-eac0d32e7751" containerID="b3ec86806cd8459ec395bff43395dc7d147f8dd8574ffe5bcf7593a922d13fa9" exitCode=0 Oct 30 17:14:56.320245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.320021 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks" event={"ID":"df7a2ad9-d72a-428a-b638-eac0d32e7751","Type":"ContainerDied","Data":"b3ec86806cd8459ec395bff43395dc7d147f8dd8574ffe5bcf7593a922d13fa9"} Oct 30 17:14:56.320245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.320057 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks" Oct 30 17:14:56.320245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.320066 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks" event={"ID":"df7a2ad9-d72a-428a-b638-eac0d32e7751","Type":"ContainerDied","Data":"15ac9a8162ca08fb8a775112999dc1f0f4a36b2263129ca07ede103e273f6986"} Oct 30 17:14:56.331359 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.331341 6202 scope.go:122] "RemoveContainer" containerID="b3ec86806cd8459ec395bff43395dc7d147f8dd8574ffe5bcf7593a922d13fa9" Oct 30 17:14:56.343075 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.343044 6202 scope.go:122] "RemoveContainer" containerID="b3ec86806cd8459ec395bff43395dc7d147f8dd8574ffe5bcf7593a922d13fa9" Oct 30 17:14:56.343420 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:14:56.343394 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ec86806cd8459ec395bff43395dc7d147f8dd8574ffe5bcf7593a922d13fa9\": container with ID starting with b3ec86806cd8459ec395bff43395dc7d147f8dd8574ffe5bcf7593a922d13fa9 not found: ID does not exist" containerID="b3ec86806cd8459ec395bff43395dc7d147f8dd8574ffe5bcf7593a922d13fa9" Oct 30 17:14:56.343501 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.343433 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ec86806cd8459ec395bff43395dc7d147f8dd8574ffe5bcf7593a922d13fa9"} err="failed to get container status \"b3ec86806cd8459ec395bff43395dc7d147f8dd8574ffe5bcf7593a922d13fa9\": rpc error: code = NotFound desc = could not find container \"b3ec86806cd8459ec395bff43395dc7d147f8dd8574ffe5bcf7593a922d13fa9\": container with ID starting with b3ec86806cd8459ec395bff43395dc7d147f8dd8574ffe5bcf7593a922d13fa9 not found: ID does not exist" Oct 30 17:14:56.366246 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.366214 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8"] Oct 30 17:14:56.369501 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.369475 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4def7e-ll7w8"] Oct 30 17:14:56.395803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.382082 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4"] Oct 30 17:14:56.395803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.391386 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac125-d7544b67f-vwpr4"] Oct 30 17:14:56.429831 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.426920 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16c3ab98-a5d7-4895-bdf0-5f584c1f510c" path="/var/lib/kubelet/pods/16c3ab98-a5d7-4895-bdf0-5f584c1f510c/volumes" Oct 30 17:14:56.441905 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.441857 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3082585b-6424-472e-8014-0eb3b5ad15a8" path="/var/lib/kubelet/pods/3082585b-6424-472e-8014-0eb3b5ad15a8/volumes" Oct 30 17:14:56.443747 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.443717 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6bf0eb3-cf41-4fb0-9545-17307559ffe0" path="/var/lib/kubelet/pods/f6bf0eb3-cf41-4fb0-9545-17307559ffe0/volumes" Oct 30 17:14:56.462758 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.462674 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks"] Oct 30 17:14:56.468353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.468327 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4def7e-56478584f5-ph9ks"] Oct 30 17:14:56.522093 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.522055 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:56.866555 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:56.866513 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:58.130079 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:58.130043 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:58.411605 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:14:58.411531 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df7a2ad9-d72a-428a-b638-eac0d32e7751" path="/var/lib/kubelet/pods/df7a2ad9-d72a-428a-b638-eac0d32e7751/volumes" Oct 30 17:15:00.145627 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.145591 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg"] Oct 30 17:15:00.146739 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.146714 6202 cpu_manager.go:414] "containerMap: removing container" podUID="df7a2ad9-d72a-428a-b638-eac0d32e7751" containerName="payload-pre-processing" Oct 30 17:15:00.146914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.146741 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df7a2ad9-d72a-428a-b638-eac0d32e7751" containerName="payload-pre-processing" Oct 30 17:15:00.146914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.146779 6202 cpu_manager.go:414] "containerMap: removing container" podUID="16c3ab98-a5d7-4895-bdf0-5f584c1f510c" containerName="payload-pre-processing" Oct 30 17:15:00.146914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.146788 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16c3ab98-a5d7-4895-bdf0-5f584c1f510c" containerName="payload-pre-processing" Oct 30 17:15:00.146914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.146839 6202 cpu_manager.go:414] "containerMap: removing container" podUID="55546d54-81a4-402e-8ad5-7110cf03a17c" containerName="payload-processing" Oct 30 17:15:00.146914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.146847 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55546d54-81a4-402e-8ad5-7110cf03a17c" containerName="payload-processing" Oct 30 17:15:00.146914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.146866 6202 cpu_manager.go:414] "containerMap: removing container" podUID="66ed3178-5d8b-4cb5-9ec6-737aa118b73a" containerName="payload-pre-processing" Oct 30 17:15:00.146914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.146873 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ed3178-5d8b-4cb5-9ec6-737aa118b73a" containerName="payload-pre-processing" Oct 30 17:15:00.146914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.146904 6202 cpu_manager.go:414] "containerMap: removing container" podUID="e0a6beae-02be-487a-980a-c69c3ce2bc9e" containerName="istio-proxy" Oct 30 17:15:00.146914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.146911 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a6beae-02be-487a-980a-c69c3ce2bc9e" containerName="istio-proxy" Oct 30 17:15:00.147422 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.147102 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="df7a2ad9-d72a-428a-b638-eac0d32e7751" containerName="payload-pre-processing" Oct 30 17:15:00.147422 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.147129 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="16c3ab98-a5d7-4895-bdf0-5f584c1f510c" containerName="payload-pre-processing" Oct 30 17:15:00.147422 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.147146 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="55546d54-81a4-402e-8ad5-7110cf03a17c" containerName="payload-processing" Oct 30 17:15:00.147422 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.147157 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ed3178-5d8b-4cb5-9ec6-737aa118b73a" containerName="payload-pre-processing" Oct 30 17:15:00.147422 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.147168 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a6beae-02be-487a-980a-c69c3ce2bc9e" containerName="istio-proxy" Oct 30 17:15:00.153274 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.153247 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7"] Oct 30 17:15:00.153409 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.153369 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" Oct 30 17:15:00.157639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.157618 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57"] Oct 30 17:15:00.157828 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.157755 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" Oct 30 17:15:00.163171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.163128 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2"] Oct 30 17:15:00.164751 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.164364 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" Oct 30 17:15:00.168378 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.168357 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h"] Oct 30 17:15:00.168591 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.168566 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" Oct 30 17:15:00.171338 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.171319 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:15:00.174721 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.174701 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94"] Oct 30 17:15:00.174863 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.174843 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" Oct 30 17:15:00.175079 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.175061 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:15:00.179999 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.179979 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d"] Oct 30 17:15:00.184147 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.184126 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" Oct 30 17:15:00.184989 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.184969 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk"] Oct 30 17:15:00.185623 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.185036 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" Oct 30 17:15:00.200377 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.200351 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5"] Oct 30 17:15:00.202528 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.202320 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" Oct 30 17:15:00.223842 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.223816 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57"] Oct 30 17:15:00.224101 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.224062 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg"] Oct 30 17:15:00.224101 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.224088 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" Oct 30 17:15:00.224568 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.224118 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2"] Oct 30 17:15:00.224568 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.224144 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h"] Oct 30 17:15:00.224568 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.224181 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn"] Oct 30 17:15:00.237191 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.237132 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" Oct 30 17:15:00.237847 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.237829 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d"] Oct 30 17:15:00.238076 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.238062 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk"] Oct 30 17:15:00.255082 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.254700 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p"] Oct 30 17:15:00.273039 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.273006 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:15:00.275740 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.275716 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" Oct 30 17:15:00.297354 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.297331 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5"] Oct 30 17:15:00.299169 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.299147 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad83a2b-ebfd-4902-9b86-bfdc8552f763-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h\" (UID: \"7ad83a2b-ebfd-4902-9b86-bfdc8552f763\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" Oct 30 17:15:00.299349 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.299332 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdeaeb1-059f-49ac-864d-0b5a4a504a50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk\" (UID: \"3fdeaeb1-059f-49ac-864d-0b5a4a504a50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" Oct 30 17:15:00.299600 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.299569 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95rs7\" (UniqueName: \"kubernetes.io/projected/67afc1c4-1c1f-4c7b-9de3-df2899b44392-kube-api-access-95rs7\") pod \"maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg\" (UID: \"67afc1c4-1c1f-4c7b-9de3-df2899b44392\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" Oct 30 17:15:00.299661 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.299631 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d7018a-d448-4e25-afc4-2a002931f75f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d\" (UID: \"73d7018a-d448-4e25-afc4-2a002931f75f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" Oct 30 17:15:00.299724 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.299708 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5n2r\" (UniqueName: \"kubernetes.io/projected/3fdeaeb1-059f-49ac-864d-0b5a4a504a50-kube-api-access-x5n2r\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk\" (UID: \"3fdeaeb1-059f-49ac-864d-0b5a4a504a50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" Oct 30 17:15:00.299834 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.299817 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa542e0-72f9-4e5b-ab62-25fab055df11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2\" (UID: \"3fa542e0-72f9-4e5b-ab62-25fab055df11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" Oct 30 17:15:00.299877 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.299860 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8hcp\" (UniqueName: \"kubernetes.io/projected/4c9ed38f-5c3a-4ea8-8969-0d2b506cc260-kube-api-access-t8hcp\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57\" (UID: \"4c9ed38f-5c3a-4ea8-8969-0d2b506cc260\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" Oct 30 17:15:00.299933 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.299898 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9ed38f-5c3a-4ea8-8969-0d2b506cc260-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57\" (UID: \"4c9ed38f-5c3a-4ea8-8969-0d2b506cc260\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" Oct 30 17:15:00.299979 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.299929 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4193469-f0ea-48f5-af10-3479e0980736-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94\" (UID: \"c4193469-f0ea-48f5-af10-3479e0980736\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" Oct 30 17:15:00.300018 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.299979 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6r72\" (UniqueName: \"kubernetes.io/projected/9a22ef35-5fa6-4126-9677-94883fe7fe72-kube-api-access-p6r72\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7\" (UID: \"9a22ef35-5fa6-4126-9677-94883fe7fe72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" Oct 30 17:15:00.300060 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.300028 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a22ef35-5fa6-4126-9677-94883fe7fe72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7\" (UID: \"9a22ef35-5fa6-4126-9677-94883fe7fe72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" Oct 30 17:15:00.300100 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.300081 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm78r\" (UniqueName: \"kubernetes.io/projected/73d7018a-d448-4e25-afc4-2a002931f75f-kube-api-access-qm78r\") pod \"maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d\" (UID: \"73d7018a-d448-4e25-afc4-2a002931f75f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" Oct 30 17:15:00.300136 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.300116 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hjtt\" (UniqueName: \"kubernetes.io/projected/3fa542e0-72f9-4e5b-ab62-25fab055df11-kube-api-access-7hjtt\") pod \"maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2\" (UID: \"3fa542e0-72f9-4e5b-ab62-25fab055df11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" Oct 30 17:15:00.300312 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.300295 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67afc1c4-1c1f-4c7b-9de3-df2899b44392-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg\" (UID: \"67afc1c4-1c1f-4c7b-9de3-df2899b44392\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" Oct 30 17:15:00.300351 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.300330 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhqlr\" (UniqueName: \"kubernetes.io/projected/7ad83a2b-ebfd-4902-9b86-bfdc8552f763-kube-api-access-fhqlr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h\" (UID: \"7ad83a2b-ebfd-4902-9b86-bfdc8552f763\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" Oct 30 17:15:00.300388 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.300355 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z6r8\" (UniqueName: \"kubernetes.io/projected/c4193469-f0ea-48f5-af10-3479e0980736-kube-api-access-6z6r8\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94\" (UID: \"c4193469-f0ea-48f5-af10-3479e0980736\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" Oct 30 17:15:00.301933 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.301907 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp"] Oct 30 17:15:00.307853 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.307742 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p"] Oct 30 17:15:00.307853 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.307847 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" Oct 30 17:15:00.311593 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.311567 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94"] Oct 30 17:15:00.314262 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.314242 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7"] Oct 30 17:15:00.318184 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.318165 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn"] Oct 30 17:15:00.321040 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.321021 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp"] Oct 30 17:15:00.401465 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.401370 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5n2r\" (UniqueName: \"kubernetes.io/projected/3fdeaeb1-059f-49ac-864d-0b5a4a504a50-kube-api-access-x5n2r\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk\" (UID: \"3fdeaeb1-059f-49ac-864d-0b5a4a504a50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" Oct 30 17:15:00.401636 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.401480 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa542e0-72f9-4e5b-ab62-25fab055df11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2\" (UID: \"3fa542e0-72f9-4e5b-ab62-25fab055df11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" Oct 30 17:15:00.401636 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.401517 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8hcp\" (UniqueName: \"kubernetes.io/projected/4c9ed38f-5c3a-4ea8-8969-0d2b506cc260-kube-api-access-t8hcp\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57\" (UID: \"4c9ed38f-5c3a-4ea8-8969-0d2b506cc260\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" Oct 30 17:15:00.401636 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.401547 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9ed38f-5c3a-4ea8-8969-0d2b506cc260-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57\" (UID: \"4c9ed38f-5c3a-4ea8-8969-0d2b506cc260\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" Oct 30 17:15:00.401636 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.401576 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4193469-f0ea-48f5-af10-3479e0980736-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94\" (UID: \"c4193469-f0ea-48f5-af10-3479e0980736\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" Oct 30 17:15:00.401636 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.401624 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6r72\" (UniqueName: \"kubernetes.io/projected/9a22ef35-5fa6-4126-9677-94883fe7fe72-kube-api-access-p6r72\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7\" (UID: \"9a22ef35-5fa6-4126-9677-94883fe7fe72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" Oct 30 17:15:00.402043 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.401654 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a22ef35-5fa6-4126-9677-94883fe7fe72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7\" (UID: \"9a22ef35-5fa6-4126-9677-94883fe7fe72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" Oct 30 17:15:00.402043 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.401710 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm78r\" (UniqueName: \"kubernetes.io/projected/73d7018a-d448-4e25-afc4-2a002931f75f-kube-api-access-qm78r\") pod \"maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d\" (UID: \"73d7018a-d448-4e25-afc4-2a002931f75f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" Oct 30 17:15:00.402043 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.401744 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hjtt\" (UniqueName: \"kubernetes.io/projected/3fa542e0-72f9-4e5b-ab62-25fab055df11-kube-api-access-7hjtt\") pod \"maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2\" (UID: \"3fa542e0-72f9-4e5b-ab62-25fab055df11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" Oct 30 17:15:00.402043 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.401829 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb9rv\" (UniqueName: \"kubernetes.io/projected/3784ac5c-2448-4e4a-b17d-80717638a281-kube-api-access-gb9rv\") pod \"maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5\" (UID: \"3784ac5c-2448-4e4a-b17d-80717638a281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" Oct 30 17:15:00.402043 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.401946 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g554k\" (UniqueName: \"kubernetes.io/projected/fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf-kube-api-access-g554k\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn\" (UID: \"fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" Oct 30 17:15:00.402386 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.402176 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn\" (UID: \"fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" Oct 30 17:15:00.402386 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.402251 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67afc1c4-1c1f-4c7b-9de3-df2899b44392-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg\" (UID: \"67afc1c4-1c1f-4c7b-9de3-df2899b44392\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" Oct 30 17:15:00.402386 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.402303 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhqlr\" (UniqueName: \"kubernetes.io/projected/7ad83a2b-ebfd-4902-9b86-bfdc8552f763-kube-api-access-fhqlr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h\" (UID: \"7ad83a2b-ebfd-4902-9b86-bfdc8552f763\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" Oct 30 17:15:00.402386 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.402331 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z6r8\" (UniqueName: \"kubernetes.io/projected/c4193469-f0ea-48f5-af10-3479e0980736-kube-api-access-6z6r8\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94\" (UID: \"c4193469-f0ea-48f5-af10-3479e0980736\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" Oct 30 17:15:00.402386 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.402365 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3784ac5c-2448-4e4a-b17d-80717638a281-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5\" (UID: \"3784ac5c-2448-4e4a-b17d-80717638a281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" Oct 30 17:15:00.402680 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.402393 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad83a2b-ebfd-4902-9b86-bfdc8552f763-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h\" (UID: \"7ad83a2b-ebfd-4902-9b86-bfdc8552f763\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" Oct 30 17:15:00.402680 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.402424 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdeaeb1-059f-49ac-864d-0b5a4a504a50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk\" (UID: \"3fdeaeb1-059f-49ac-864d-0b5a4a504a50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" Oct 30 17:15:00.402680 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.402417 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4193469-f0ea-48f5-af10-3479e0980736-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94\" (UID: \"c4193469-f0ea-48f5-af10-3479e0980736\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" Oct 30 17:15:00.402680 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.402433 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa542e0-72f9-4e5b-ab62-25fab055df11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2\" (UID: \"3fa542e0-72f9-4e5b-ab62-25fab055df11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" Oct 30 17:15:00.402680 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.402511 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11be79af-8b98-43ef-8880-3dc9ac2c3a6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p\" (UID: \"11be79af-8b98-43ef-8880-3dc9ac2c3a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" Oct 30 17:15:00.402680 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.402559 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95rs7\" (UniqueName: \"kubernetes.io/projected/67afc1c4-1c1f-4c7b-9de3-df2899b44392-kube-api-access-95rs7\") pod \"maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg\" (UID: \"67afc1c4-1c1f-4c7b-9de3-df2899b44392\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" Oct 30 17:15:00.402680 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.402592 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vzm7\" (UniqueName: \"kubernetes.io/projected/11be79af-8b98-43ef-8880-3dc9ac2c3a6f-kube-api-access-7vzm7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p\" (UID: \"11be79af-8b98-43ef-8880-3dc9ac2c3a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" Oct 30 17:15:00.402680 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.402628 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a22ef35-5fa6-4126-9677-94883fe7fe72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7\" (UID: \"9a22ef35-5fa6-4126-9677-94883fe7fe72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" Oct 30 17:15:00.402680 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.402392 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9ed38f-5c3a-4ea8-8969-0d2b506cc260-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57\" (UID: \"4c9ed38f-5c3a-4ea8-8969-0d2b506cc260\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" Oct 30 17:15:00.403457 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.403090 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d7018a-d448-4e25-afc4-2a002931f75f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d\" (UID: \"73d7018a-d448-4e25-afc4-2a002931f75f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" Oct 30 17:15:00.403457 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.403134 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad83a2b-ebfd-4902-9b86-bfdc8552f763-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h\" (UID: \"7ad83a2b-ebfd-4902-9b86-bfdc8552f763\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" Oct 30 17:15:00.403457 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.403250 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67afc1c4-1c1f-4c7b-9de3-df2899b44392-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg\" (UID: \"67afc1c4-1c1f-4c7b-9de3-df2899b44392\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" Oct 30 17:15:00.403457 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.403322 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdeaeb1-059f-49ac-864d-0b5a4a504a50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk\" (UID: \"3fdeaeb1-059f-49ac-864d-0b5a4a504a50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" Oct 30 17:15:00.403735 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.403717 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d7018a-d448-4e25-afc4-2a002931f75f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d\" (UID: \"73d7018a-d448-4e25-afc4-2a002931f75f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" Oct 30 17:15:00.427432 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.427381 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5n2r\" (UniqueName: \"kubernetes.io/projected/3fdeaeb1-059f-49ac-864d-0b5a4a504a50-kube-api-access-x5n2r\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk\" (UID: \"3fdeaeb1-059f-49ac-864d-0b5a4a504a50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" Oct 30 17:15:00.427432 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.427399 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6r72\" (UniqueName: \"kubernetes.io/projected/9a22ef35-5fa6-4126-9677-94883fe7fe72-kube-api-access-p6r72\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7\" (UID: \"9a22ef35-5fa6-4126-9677-94883fe7fe72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" Oct 30 17:15:00.427702 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.427683 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hjtt\" (UniqueName: \"kubernetes.io/projected/3fa542e0-72f9-4e5b-ab62-25fab055df11-kube-api-access-7hjtt\") pod \"maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2\" (UID: \"3fa542e0-72f9-4e5b-ab62-25fab055df11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" Oct 30 17:15:00.427842 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.427717 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm78r\" (UniqueName: \"kubernetes.io/projected/73d7018a-d448-4e25-afc4-2a002931f75f-kube-api-access-qm78r\") pod \"maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d\" (UID: \"73d7018a-d448-4e25-afc4-2a002931f75f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" Oct 30 17:15:00.428043 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.428020 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95rs7\" (UniqueName: \"kubernetes.io/projected/67afc1c4-1c1f-4c7b-9de3-df2899b44392-kube-api-access-95rs7\") pod \"maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg\" (UID: \"67afc1c4-1c1f-4c7b-9de3-df2899b44392\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" Oct 30 17:15:00.428135 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.428024 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8hcp\" (UniqueName: \"kubernetes.io/projected/4c9ed38f-5c3a-4ea8-8969-0d2b506cc260-kube-api-access-t8hcp\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57\" (UID: \"4c9ed38f-5c3a-4ea8-8969-0d2b506cc260\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" Oct 30 17:15:00.428135 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.428102 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z6r8\" (UniqueName: \"kubernetes.io/projected/c4193469-f0ea-48f5-af10-3479e0980736-kube-api-access-6z6r8\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94\" (UID: \"c4193469-f0ea-48f5-af10-3479e0980736\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" Oct 30 17:15:00.428606 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.428590 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhqlr\" (UniqueName: \"kubernetes.io/projected/7ad83a2b-ebfd-4902-9b86-bfdc8552f763-kube-api-access-fhqlr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h\" (UID: \"7ad83a2b-ebfd-4902-9b86-bfdc8552f763\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" Oct 30 17:15:00.472111 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.472078 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" Oct 30 17:15:00.486921 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.486884 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" Oct 30 17:15:00.503283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.503260 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" Oct 30 17:15:00.505033 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.505003 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5tgw\" (UniqueName: \"kubernetes.io/projected/e9e9f54b-cee4-43c3-96b5-e75eaf17adad-kube-api-access-s5tgw\") pod \"maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp\" (UID: \"e9e9f54b-cee4-43c3-96b5-e75eaf17adad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" Oct 30 17:15:00.505172 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.505153 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb9rv\" (UniqueName: \"kubernetes.io/projected/3784ac5c-2448-4e4a-b17d-80717638a281-kube-api-access-gb9rv\") pod \"maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5\" (UID: \"3784ac5c-2448-4e4a-b17d-80717638a281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" Oct 30 17:15:00.505249 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.505233 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g554k\" (UniqueName: \"kubernetes.io/projected/fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf-kube-api-access-g554k\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn\" (UID: \"fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" Oct 30 17:15:00.505326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.505303 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn\" (UID: \"fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" Oct 30 17:15:00.505441 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.505378 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3784ac5c-2448-4e4a-b17d-80717638a281-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5\" (UID: \"3784ac5c-2448-4e4a-b17d-80717638a281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" Oct 30 17:15:00.505544 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.505484 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11be79af-8b98-43ef-8880-3dc9ac2c3a6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p\" (UID: \"11be79af-8b98-43ef-8880-3dc9ac2c3a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" Oct 30 17:15:00.505544 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.505527 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vzm7\" (UniqueName: \"kubernetes.io/projected/11be79af-8b98-43ef-8880-3dc9ac2c3a6f-kube-api-access-7vzm7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p\" (UID: \"11be79af-8b98-43ef-8880-3dc9ac2c3a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" Oct 30 17:15:00.505703 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.505683 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e9f54b-cee4-43c3-96b5-e75eaf17adad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp\" (UID: \"e9e9f54b-cee4-43c3-96b5-e75eaf17adad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" Oct 30 17:15:00.506070 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.506039 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn\" (UID: \"fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" Oct 30 17:15:00.506307 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.506283 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3784ac5c-2448-4e4a-b17d-80717638a281-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5\" (UID: \"3784ac5c-2448-4e4a-b17d-80717638a281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" Oct 30 17:15:00.506662 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.506642 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11be79af-8b98-43ef-8880-3dc9ac2c3a6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p\" (UID: \"11be79af-8b98-43ef-8880-3dc9ac2c3a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" Oct 30 17:15:00.517333 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.517302 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:00.530393 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.529570 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" Oct 30 17:15:00.534003 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.533538 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vzm7\" (UniqueName: \"kubernetes.io/projected/11be79af-8b98-43ef-8880-3dc9ac2c3a6f-kube-api-access-7vzm7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p\" (UID: \"11be79af-8b98-43ef-8880-3dc9ac2c3a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" Oct 30 17:15:00.537147 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.537088 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g554k\" (UniqueName: \"kubernetes.io/projected/fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf-kube-api-access-g554k\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn\" (UID: \"fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" Oct 30 17:15:00.542721 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.542636 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb9rv\" (UniqueName: \"kubernetes.io/projected/3784ac5c-2448-4e4a-b17d-80717638a281-kube-api-access-gb9rv\") pod \"maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5\" (UID: \"3784ac5c-2448-4e4a-b17d-80717638a281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" Oct 30 17:15:00.565568 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.563638 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" Oct 30 17:15:00.648939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.601062 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" Oct 30 17:15:00.648939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.601246 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" Oct 30 17:15:00.648939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.610759 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e9f54b-cee4-43c3-96b5-e75eaf17adad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp\" (UID: \"e9e9f54b-cee4-43c3-96b5-e75eaf17adad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" Oct 30 17:15:00.648939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.610872 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5tgw\" (UniqueName: \"kubernetes.io/projected/e9e9f54b-cee4-43c3-96b5-e75eaf17adad-kube-api-access-s5tgw\") pod \"maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp\" (UID: \"e9e9f54b-cee4-43c3-96b5-e75eaf17adad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" Oct 30 17:15:00.648939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.612145 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" Oct 30 17:15:00.648939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.616091 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" Oct 30 17:15:00.648939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.628165 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" Oct 30 17:15:00.648939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.639283 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" Oct 30 17:15:00.648939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.641827 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5tgw\" (UniqueName: \"kubernetes.io/projected/e9e9f54b-cee4-43c3-96b5-e75eaf17adad-kube-api-access-s5tgw\") pod \"maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp\" (UID: \"e9e9f54b-cee4-43c3-96b5-e75eaf17adad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" Oct 30 17:15:00.667048 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.657455 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e9f54b-cee4-43c3-96b5-e75eaf17adad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp\" (UID: \"e9e9f54b-cee4-43c3-96b5-e75eaf17adad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" Oct 30 17:15:00.948904 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:00.948013 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" Oct 30 17:15:01.003179 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:01.003145 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg"] Oct 30 17:15:01.006054 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:15:01.006021 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67afc1c4_1c1f_4c7b_9de3_df2899b44392.slice/crio-939ed0173eb3b7d0a780a054f83fc361b0821d80eb28e70c161a1cc83ae5b390 WatchSource:0}: Error finding container 939ed0173eb3b7d0a780a054f83fc361b0821d80eb28e70c161a1cc83ae5b390: Status 404 returned error can't find the container with id 939ed0173eb3b7d0a780a054f83fc361b0821d80eb28e70c161a1cc83ae5b390 Oct 30 17:15:01.107376 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:15:01.107332 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9ed38f_5c3a_4ea8_8969_0d2b506cc260.slice/crio-2ca3b0fce1c834086e2f63abbdbbb8751dd2db75df616f5ebc68bd22d2979e7f WatchSource:0}: Error finding container 2ca3b0fce1c834086e2f63abbdbbb8751dd2db75df616f5ebc68bd22d2979e7f: Status 404 returned error can't find the container with id 2ca3b0fce1c834086e2f63abbdbbb8751dd2db75df616f5ebc68bd22d2979e7f Oct 30 17:15:01.107961 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:01.107937 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57"] Oct 30 17:15:01.119944 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:01.119912 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2"] Oct 30 17:15:01.122810 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:15:01.122744 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa542e0_72f9_4e5b_ab62_25fab055df11.slice/crio-18a5389450a13abc6107a2e92120349f6d3b5ba3eec8f4ada28355ddf9397f41 WatchSource:0}: Error finding container 18a5389450a13abc6107a2e92120349f6d3b5ba3eec8f4ada28355ddf9397f41: Status 404 returned error can't find the container with id 18a5389450a13abc6107a2e92120349f6d3b5ba3eec8f4ada28355ddf9397f41 Oct 30 17:15:01.266029 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:01.265487 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7"] Oct 30 17:15:01.281542 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:01.281507 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p"] Oct 30 17:15:01.303631 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:01.303601 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp"] Oct 30 17:15:01.331310 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:01.331279 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk"] Oct 30 17:15:01.347126 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:15:01.346903 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a22ef35_5fa6_4126_9677_94883fe7fe72.slice/crio-73d7095bd6a5d5ed00cec4c71580c2da52c972ec594d9fc4df5707ce0dcbfd0d WatchSource:0}: Error finding container 73d7095bd6a5d5ed00cec4c71580c2da52c972ec594d9fc4df5707ce0dcbfd0d: Status 404 returned error can't find the container with id 73d7095bd6a5d5ed00cec4c71580c2da52c972ec594d9fc4df5707ce0dcbfd0d Oct 30 17:15:01.348702 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:01.348664 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn"] Oct 30 17:15:01.358727 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:01.358702 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d"] Oct 30 17:15:01.386123 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:01.385988 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" event={"ID":"11be79af-8b98-43ef-8880-3dc9ac2c3a6f","Type":"ContainerStarted","Data":"e029b9b9700f433febfa2f4f4ce5145b634d31e52c65598353924b5e270d445a"} Oct 30 17:15:01.392072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:01.391219 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" event={"ID":"4c9ed38f-5c3a-4ea8-8969-0d2b506cc260","Type":"ContainerStarted","Data":"2ca3b0fce1c834086e2f63abbdbbb8751dd2db75df616f5ebc68bd22d2979e7f"} Oct 30 17:15:01.393586 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:01.393558 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94"] Oct 30 17:15:01.394028 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:01.393997 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" event={"ID":"67afc1c4-1c1f-4c7b-9de3-df2899b44392","Type":"ContainerStarted","Data":"939ed0173eb3b7d0a780a054f83fc361b0821d80eb28e70c161a1cc83ae5b390"} Oct 30 17:15:01.395808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:01.395780 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" event={"ID":"3fa542e0-72f9-4e5b-ab62-25fab055df11","Type":"ContainerStarted","Data":"18a5389450a13abc6107a2e92120349f6d3b5ba3eec8f4ada28355ddf9397f41"} Oct 30 17:15:01.397322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:01.397295 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" event={"ID":"9a22ef35-5fa6-4126-9677-94883fe7fe72","Type":"ContainerStarted","Data":"73d7095bd6a5d5ed00cec4c71580c2da52c972ec594d9fc4df5707ce0dcbfd0d"} Oct 30 17:15:01.397415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:01.397375 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5"] Oct 30 17:15:01.510282 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:01.509275 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h"] Oct 30 17:15:02.163335 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.163293 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:02.278002 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.277977 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:02.412379 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.412289 6202 generic.go:356] "Generic (PLEG): container finished" podID="4c9ed38f-5c3a-4ea8-8969-0d2b506cc260" containerID="69bf53a913300aba8efd5e3598d7f7e6f58871ecf8dcb7ce3f0e18879d4208df" exitCode=60 Oct 30 17:15:02.412658 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.412635 6202 scope.go:122] "RemoveContainer" containerID="69bf53a913300aba8efd5e3598d7f7e6f58871ecf8dcb7ce3f0e18879d4208df" Oct 30 17:15:02.413334 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.413309 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" event={"ID":"11be79af-8b98-43ef-8880-3dc9ac2c3a6f","Type":"ContainerStarted","Data":"f78b3c3c01da430e198e7e61513648b46311c43a29b43cb1451d343682f00595"} Oct 30 17:15:02.413435 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.413421 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" event={"ID":"3784ac5c-2448-4e4a-b17d-80717638a281","Type":"ContainerStarted","Data":"fb6ba931d24629fdc46f5e96fe9cfeea601379ae555233c3906a8a8c3b885972"} Oct 30 17:15:02.413495 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.413449 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" event={"ID":"3784ac5c-2448-4e4a-b17d-80717638a281","Type":"ContainerStarted","Data":"d10621e22540be35a4c59203a76f074eec39cff3f3a14314bfa78b584ad5b54d"} Oct 30 17:15:02.413495 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.413471 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" event={"ID":"4c9ed38f-5c3a-4ea8-8969-0d2b506cc260","Type":"ContainerDied","Data":"69bf53a913300aba8efd5e3598d7f7e6f58871ecf8dcb7ce3f0e18879d4208df"} Oct 30 17:15:02.414369 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.414345 6202 generic.go:356] "Generic (PLEG): container finished" podID="67afc1c4-1c1f-4c7b-9de3-df2899b44392" containerID="8908d6d6ec14111f77ba73efc17304c62aae2490e67fe65d2326eedadfb9924d" exitCode=60 Oct 30 17:15:02.414508 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.414418 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" event={"ID":"67afc1c4-1c1f-4c7b-9de3-df2899b44392","Type":"ContainerDied","Data":"8908d6d6ec14111f77ba73efc17304c62aae2490e67fe65d2326eedadfb9924d"} Oct 30 17:15:02.414730 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.414706 6202 scope.go:122] "RemoveContainer" containerID="8908d6d6ec14111f77ba73efc17304c62aae2490e67fe65d2326eedadfb9924d" Oct 30 17:15:02.416435 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.416413 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" event={"ID":"73d7018a-d448-4e25-afc4-2a002931f75f","Type":"ContainerStarted","Data":"161c94d138dad5cc79daf29287a6559c3a8b4c1daa6461a29183cb6e54b87213"} Oct 30 17:15:02.416515 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.416461 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" event={"ID":"73d7018a-d448-4e25-afc4-2a002931f75f","Type":"ContainerStarted","Data":"6c157a29718e2625e20156d43b340f34a8447a0fd61d1ada9b94a08ef35bc7e5"} Oct 30 17:15:02.417898 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.417872 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" event={"ID":"3fa542e0-72f9-4e5b-ab62-25fab055df11","Type":"ContainerStarted","Data":"e84f89eaac353df8e223f88900f4556cd5fd76ce9dbca184aade074620fb0fb3"} Oct 30 17:15:02.419445 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.419417 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" event={"ID":"fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf","Type":"ContainerStarted","Data":"0b9e885f85d2de9a0621493324846b9574eb8579a0102e7af7e25911b7ea8125"} Oct 30 17:15:02.419524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.419464 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" event={"ID":"fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf","Type":"ContainerStarted","Data":"8b8c0f552f73462679b062a1c90fd0bf186ea5348ff6402cc8aa95c2fce91197"} Oct 30 17:15:02.421115 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.421088 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" event={"ID":"7ad83a2b-ebfd-4902-9b86-bfdc8552f763","Type":"ContainerStarted","Data":"4a91d1c791051b31d3f0b23d516361504e0d9667576b8de7c810d5227df61b89"} Oct 30 17:15:02.421228 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.421135 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" event={"ID":"7ad83a2b-ebfd-4902-9b86-bfdc8552f763","Type":"ContainerStarted","Data":"332a3aadf67ba39102f5b45a5746223405c41aeb4960e8c5c10b90ecb0b97c19"} Oct 30 17:15:02.422803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.422780 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" event={"ID":"3fdeaeb1-059f-49ac-864d-0b5a4a504a50","Type":"ContainerStarted","Data":"2172d0ccf40e844e1c99865f1c8cbc97537fa6f74955bb5951c5f12e2f2c510c"} Oct 30 17:15:02.422895 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.422825 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" event={"ID":"3fdeaeb1-059f-49ac-864d-0b5a4a504a50","Type":"ContainerStarted","Data":"838b2437bd1d8f25eaeafd44a1517124298697ef2dade9130555cc358c905679"} Oct 30 17:15:02.424398 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.424373 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" event={"ID":"9a22ef35-5fa6-4126-9677-94883fe7fe72","Type":"ContainerStarted","Data":"d337ea314ea35f3ce2bc20c1a153babb5e72bddf9b1abbee924a473b99bb5da6"} Oct 30 17:15:02.426169 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.426145 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" event={"ID":"e9e9f54b-cee4-43c3-96b5-e75eaf17adad","Type":"ContainerStarted","Data":"58efcf2d343c36606c0556b206fdfe26b128251c791727a2eb35a8e3e0c676fe"} Oct 30 17:15:02.426290 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.426185 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" event={"ID":"e9e9f54b-cee4-43c3-96b5-e75eaf17adad","Type":"ContainerStarted","Data":"e0e61989e1a3e0fcbcd9f3fbe74adbafb2330ef510b5c843f7c8fc3f8ecca74b"} Oct 30 17:15:02.428031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.428001 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" event={"ID":"c4193469-f0ea-48f5-af10-3479e0980736","Type":"ContainerStarted","Data":"f1da6266a582dbe7adeea274affa26d31489e4a48e7dba3396c08629d149d676"} Oct 30 17:15:02.428129 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.428050 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" event={"ID":"c4193469-f0ea-48f5-af10-3479e0980736","Type":"ContainerStarted","Data":"5930a4ec63227c5c6daeb5de428504403448664446a1e74713a877b7949f9a02"} Oct 30 17:15:02.465081 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.465025 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" podStartSLOduration=2.465010367 podStartE2EDuration="2.465010367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.458340863 +0000 UTC m=+3216.168795609" watchObservedRunningTime="2026-10-30 17:15:02.465010367 +0000 UTC m=+3216.175465112" Oct 30 17:15:02.497591 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.497559 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6"] Oct 30 17:15:02.497833 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.497812 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" podUID="bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" containerName="istio-proxy" containerID="cri-o://0574e77ea7356cbbaf388ff42559d5378a52ea15581a759e3145645ce020b607" gracePeriod=30 Oct 30 17:15:02.512823 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.510803 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" podStartSLOduration=2.51078437 podStartE2EDuration="2.51078437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.506054538 +0000 UTC m=+3216.216509284" watchObservedRunningTime="2026-10-30 17:15:02.51078437 +0000 UTC m=+3216.221239107" Oct 30 17:15:02.553019 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.552971 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" podStartSLOduration=2.55295445 podStartE2EDuration="2.55295445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.551662169 +0000 UTC m=+3216.262116913" watchObservedRunningTime="2026-10-30 17:15:02.55295445 +0000 UTC m=+3216.263409195" Oct 30 17:15:02.553583 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.553543 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:02.795896 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.785826 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" podStartSLOduration=2.785806593 podStartE2EDuration="2.785806593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.732451803 +0000 UTC m=+3216.442906624" watchObservedRunningTime="2026-10-30 17:15:02.785806593 +0000 UTC m=+3216.496261339" Oct 30 17:15:02.915996 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.906952 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" podStartSLOduration=2.906930584 podStartE2EDuration="2.906930584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.862259299 +0000 UTC m=+3216.572714044" watchObservedRunningTime="2026-10-30 17:15:02.906930584 +0000 UTC m=+3216.617385329" Oct 30 17:15:02.992830 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:02.992736 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" podStartSLOduration=2.99270996 podStartE2EDuration="2.99270996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.905194334 +0000 UTC m=+3216.615649079" watchObservedRunningTime="2026-10-30 17:15:02.99270996 +0000 UTC m=+3216.703164707" Oct 30 17:15:03.050433 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.050350 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" podStartSLOduration=3.050331457 podStartE2EDuration="3.050331457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.045609574 +0000 UTC m=+3216.756064318" watchObservedRunningTime="2026-10-30 17:15:03.050331457 +0000 UTC m=+3216.760786202" Oct 30 17:15:03.076429 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.076380 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f"] Oct 30 17:15:03.081403 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.081384 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f" Oct 30 17:15:03.099832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.098643 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" podStartSLOduration=3.09862632 podStartE2EDuration="3.09862632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.097020316 +0000 UTC m=+3216.807475054" watchObservedRunningTime="2026-10-30 17:15:03.09862632 +0000 UTC m=+3216.809081066" Oct 30 17:15:03.112264 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.112067 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:15:03.138178 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.138141 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f"] Oct 30 17:15:03.161403 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.161363 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68156cc-8c00-43ec-a4eb-b0613b136629-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f\" (UID: \"d68156cc-8c00-43ec-a4eb-b0613b136629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f" Oct 30 17:15:03.161617 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.161502 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz6hl\" (UniqueName: \"kubernetes.io/projected/d68156cc-8c00-43ec-a4eb-b0613b136629-kube-api-access-nz6hl\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f\" (UID: \"d68156cc-8c00-43ec-a4eb-b0613b136629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f" Oct 30 17:15:03.161689 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.161653 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68156cc-8c00-43ec-a4eb-b0613b136629-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f\" (UID: \"d68156cc-8c00-43ec-a4eb-b0613b136629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f" Oct 30 17:15:03.194863 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.194783 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" podStartSLOduration=3.194750105 podStartE2EDuration="3.194750105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.193895907 +0000 UTC m=+3216.904350651" watchObservedRunningTime="2026-10-30 17:15:03.194750105 +0000 UTC m=+3216.905204848" Oct 30 17:15:03.197975 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.197924 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" podStartSLOduration=3.197910688 podStartE2EDuration="3.197910688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.151227673 +0000 UTC m=+3216.861682415" watchObservedRunningTime="2026-10-30 17:15:03.197910688 +0000 UTC m=+3216.908365469" Oct 30 17:15:03.262779 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.262738 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68156cc-8c00-43ec-a4eb-b0613b136629-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f\" (UID: \"d68156cc-8c00-43ec-a4eb-b0613b136629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f" Oct 30 17:15:03.263094 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.262880 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz6hl\" (UniqueName: \"kubernetes.io/projected/d68156cc-8c00-43ec-a4eb-b0613b136629-kube-api-access-nz6hl\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f\" (UID: \"d68156cc-8c00-43ec-a4eb-b0613b136629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f" Oct 30 17:15:03.263094 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.262974 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68156cc-8c00-43ec-a4eb-b0613b136629-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f\" (UID: \"d68156cc-8c00-43ec-a4eb-b0613b136629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f" Oct 30 17:15:03.263922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.263893 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68156cc-8c00-43ec-a4eb-b0613b136629-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f\" (UID: \"d68156cc-8c00-43ec-a4eb-b0613b136629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f" Oct 30 17:15:03.265454 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.265433 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68156cc-8c00-43ec-a4eb-b0613b136629-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f\" (UID: \"d68156cc-8c00-43ec-a4eb-b0613b136629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f" Oct 30 17:15:03.289307 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.289273 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz6hl\" (UniqueName: \"kubernetes.io/projected/d68156cc-8c00-43ec-a4eb-b0613b136629-kube-api-access-nz6hl\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f\" (UID: \"d68156cc-8c00-43ec-a4eb-b0613b136629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f" Oct 30 17:15:03.296743 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.296716 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f" Oct 30 17:15:03.448750 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.448711 6202 generic.go:356] "Generic (PLEG): container finished" podID="3fa542e0-72f9-4e5b-ab62-25fab055df11" containerID="e84f89eaac353df8e223f88900f4556cd5fd76ce9dbca184aade074620fb0fb3" exitCode=60 Oct 30 17:15:03.448976 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.448901 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" event={"ID":"3fa542e0-72f9-4e5b-ab62-25fab055df11","Type":"ContainerDied","Data":"e84f89eaac353df8e223f88900f4556cd5fd76ce9dbca184aade074620fb0fb3"} Oct 30 17:15:03.449325 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.449305 6202 scope.go:122] "RemoveContainer" containerID="e84f89eaac353df8e223f88900f4556cd5fd76ce9dbca184aade074620fb0fb3" Oct 30 17:15:03.451725 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.451641 6202 generic.go:356] "Generic (PLEG): container finished" podID="fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf" containerID="0b9e885f85d2de9a0621493324846b9574eb8579a0102e7af7e25911b7ea8125" exitCode=60 Oct 30 17:15:03.451725 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.451707 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" event={"ID":"fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf","Type":"ContainerDied","Data":"0b9e885f85d2de9a0621493324846b9574eb8579a0102e7af7e25911b7ea8125"} Oct 30 17:15:03.452153 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.452130 6202 scope.go:122] "RemoveContainer" containerID="0b9e885f85d2de9a0621493324846b9574eb8579a0102e7af7e25911b7ea8125" Oct 30 17:15:03.454336 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.454315 6202 generic.go:356] "Generic (PLEG): container finished" podID="7ad83a2b-ebfd-4902-9b86-bfdc8552f763" containerID="4a91d1c791051b31d3f0b23d516361504e0d9667576b8de7c810d5227df61b89" exitCode=60 Oct 30 17:15:03.454451 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.454396 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" event={"ID":"7ad83a2b-ebfd-4902-9b86-bfdc8552f763","Type":"ContainerDied","Data":"4a91d1c791051b31d3f0b23d516361504e0d9667576b8de7c810d5227df61b89"} Oct 30 17:15:03.455225 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.455203 6202 scope.go:122] "RemoveContainer" containerID="4a91d1c791051b31d3f0b23d516361504e0d9667576b8de7c810d5227df61b89" Oct 30 17:15:03.456555 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.456527 6202 generic.go:356] "Generic (PLEG): container finished" podID="3fdeaeb1-059f-49ac-864d-0b5a4a504a50" containerID="2172d0ccf40e844e1c99865f1c8cbc97537fa6f74955bb5951c5f12e2f2c510c" exitCode=60 Oct 30 17:15:03.456653 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.456607 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" event={"ID":"3fdeaeb1-059f-49ac-864d-0b5a4a504a50","Type":"ContainerDied","Data":"2172d0ccf40e844e1c99865f1c8cbc97537fa6f74955bb5951c5f12e2f2c510c"} Oct 30 17:15:03.456837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.456817 6202 scope.go:122] "RemoveContainer" containerID="2172d0ccf40e844e1c99865f1c8cbc97537fa6f74955bb5951c5f12e2f2c510c" Oct 30 17:15:03.458537 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.458516 6202 generic.go:356] "Generic (PLEG): container finished" podID="9a22ef35-5fa6-4126-9677-94883fe7fe72" containerID="d337ea314ea35f3ce2bc20c1a153babb5e72bddf9b1abbee924a473b99bb5da6" exitCode=60 Oct 30 17:15:03.458701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.458584 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" event={"ID":"9a22ef35-5fa6-4126-9677-94883fe7fe72","Type":"ContainerDied","Data":"d337ea314ea35f3ce2bc20c1a153babb5e72bddf9b1abbee924a473b99bb5da6"} Oct 30 17:15:03.458882 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.458844 6202 scope.go:122] "RemoveContainer" containerID="d337ea314ea35f3ce2bc20c1a153babb5e72bddf9b1abbee924a473b99bb5da6" Oct 30 17:15:03.460553 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.460530 6202 generic.go:356] "Generic (PLEG): container finished" podID="e9e9f54b-cee4-43c3-96b5-e75eaf17adad" containerID="58efcf2d343c36606c0556b206fdfe26b128251c791727a2eb35a8e3e0c676fe" exitCode=60 Oct 30 17:15:03.460661 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.460605 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" event={"ID":"e9e9f54b-cee4-43c3-96b5-e75eaf17adad","Type":"ContainerDied","Data":"58efcf2d343c36606c0556b206fdfe26b128251c791727a2eb35a8e3e0c676fe"} Oct 30 17:15:03.460910 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.460865 6202 scope.go:122] "RemoveContainer" containerID="58efcf2d343c36606c0556b206fdfe26b128251c791727a2eb35a8e3e0c676fe" Oct 30 17:15:03.463395 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.463367 6202 generic.go:356] "Generic (PLEG): container finished" podID="c4193469-f0ea-48f5-af10-3479e0980736" containerID="f1da6266a582dbe7adeea274affa26d31489e4a48e7dba3396c08629d149d676" exitCode=60 Oct 30 17:15:03.463533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.463505 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" event={"ID":"c4193469-f0ea-48f5-af10-3479e0980736","Type":"ContainerDied","Data":"f1da6266a582dbe7adeea274affa26d31489e4a48e7dba3396c08629d149d676"} Oct 30 17:15:03.463962 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.463840 6202 scope.go:122] "RemoveContainer" containerID="f1da6266a582dbe7adeea274affa26d31489e4a48e7dba3396c08629d149d676" Oct 30 17:15:03.469034 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.469012 6202 generic.go:356] "Generic (PLEG): container finished" podID="11be79af-8b98-43ef-8880-3dc9ac2c3a6f" containerID="f78b3c3c01da430e198e7e61513648b46311c43a29b43cb1451d343682f00595" exitCode=60 Oct 30 17:15:03.469146 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.469089 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" event={"ID":"11be79af-8b98-43ef-8880-3dc9ac2c3a6f","Type":"ContainerDied","Data":"f78b3c3c01da430e198e7e61513648b46311c43a29b43cb1451d343682f00595"} Oct 30 17:15:03.469294 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.469279 6202 scope.go:122] "RemoveContainer" containerID="f78b3c3c01da430e198e7e61513648b46311c43a29b43cb1451d343682f00595" Oct 30 17:15:03.470918 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.470900 6202 generic.go:356] "Generic (PLEG): container finished" podID="3784ac5c-2448-4e4a-b17d-80717638a281" containerID="fb6ba931d24629fdc46f5e96fe9cfeea601379ae555233c3906a8a8c3b885972" exitCode=60 Oct 30 17:15:03.471015 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.470989 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" event={"ID":"3784ac5c-2448-4e4a-b17d-80717638a281","Type":"ContainerDied","Data":"fb6ba931d24629fdc46f5e96fe9cfeea601379ae555233c3906a8a8c3b885972"} Oct 30 17:15:03.471130 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.471117 6202 scope.go:122] "RemoveContainer" containerID="fb6ba931d24629fdc46f5e96fe9cfeea601379ae555233c3906a8a8c3b885972" Oct 30 17:15:03.475818 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.473259 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" event={"ID":"4c9ed38f-5c3a-4ea8-8969-0d2b506cc260","Type":"ContainerStarted","Data":"b1ed244c488a8a779985aebc87dfb03a349d5f52c86f0ae9fe8df8ebea069721"} Oct 30 17:15:03.481364 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.480876 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" event={"ID":"67afc1c4-1c1f-4c7b-9de3-df2899b44392","Type":"ContainerStarted","Data":"4f1d6fa3285db8df65dc2f59795f620d496286e8137b74f2c21162ddd73d7bba"} Oct 30 17:15:03.483870 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.483812 6202 generic.go:356] "Generic (PLEG): container finished" podID="73d7018a-d448-4e25-afc4-2a002931f75f" containerID="161c94d138dad5cc79daf29287a6559c3a8b4c1daa6461a29183cb6e54b87213" exitCode=60 Oct 30 17:15:03.484039 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.483988 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" event={"ID":"73d7018a-d448-4e25-afc4-2a002931f75f","Type":"ContainerDied","Data":"161c94d138dad5cc79daf29287a6559c3a8b4c1daa6461a29183cb6e54b87213"} Oct 30 17:15:03.484247 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.484230 6202 scope.go:122] "RemoveContainer" containerID="161c94d138dad5cc79daf29287a6559c3a8b4c1daa6461a29183cb6e54b87213" Oct 30 17:15:03.600134 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.600083 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" podStartSLOduration=3.600067447 podStartE2EDuration="3.600067447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.598444373 +0000 UTC m=+3217.308899108" watchObservedRunningTime="2026-10-30 17:15:03.600067447 +0000 UTC m=+3217.310522231" Oct 30 17:15:03.608022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.607996 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f"] Oct 30 17:15:03.744266 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:15:03.744235 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68156cc_8c00_43ec_a4eb_b0613b136629.slice/crio-15e5545725c45102dd8d5646e672949d320ae31d49fde9600a69226f6a9dade5 WatchSource:0}: Error finding container 15e5545725c45102dd8d5646e672949d320ae31d49fde9600a69226f6a9dade5: Status 404 returned error can't find the container with id 15e5545725c45102dd8d5646e672949d320ae31d49fde9600a69226f6a9dade5 Oct 30 17:15:03.779942 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:03.779891 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" podStartSLOduration=3.7798653079999998 podStartE2EDuration="3.779865308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.777941956 +0000 UTC m=+3217.488396700" watchObservedRunningTime="2026-10-30 17:15:03.779865308 +0000 UTC m=+3217.490320056" Oct 30 17:15:04.321614 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.321582 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:04.575044 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.574089 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" event={"ID":"3784ac5c-2448-4e4a-b17d-80717638a281","Type":"ContainerStarted","Data":"c94c333e9ca1242a28d9cc4c2e9bfc0c759245769ab27d45e5ed31551641a7a8"} Oct 30 17:15:04.583459 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.583416 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f" event={"ID":"d68156cc-8c00-43ec-a4eb-b0613b136629","Type":"ContainerStarted","Data":"15e5545725c45102dd8d5646e672949d320ae31d49fde9600a69226f6a9dade5"} Oct 30 17:15:04.590254 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.590220 6202 generic.go:356] "Generic (PLEG): container finished" podID="4c9ed38f-5c3a-4ea8-8969-0d2b506cc260" containerID="b1ed244c488a8a779985aebc87dfb03a349d5f52c86f0ae9fe8df8ebea069721" exitCode=60 Oct 30 17:15:04.590522 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.590499 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" event={"ID":"4c9ed38f-5c3a-4ea8-8969-0d2b506cc260","Type":"ContainerDied","Data":"b1ed244c488a8a779985aebc87dfb03a349d5f52c86f0ae9fe8df8ebea069721"} Oct 30 17:15:04.590717 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.590700 6202 scope.go:122] "RemoveContainer" containerID="69bf53a913300aba8efd5e3598d7f7e6f58871ecf8dcb7ce3f0e18879d4208df" Oct 30 17:15:04.591453 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.591432 6202 scope.go:122] "RemoveContainer" containerID="b1ed244c488a8a779985aebc87dfb03a349d5f52c86f0ae9fe8df8ebea069721" Oct 30 17:15:04.592197 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:04.592169 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57_odh-ai-gateway-infra(4c9ed38f-5c3a-4ea8-8969-0d2b506cc260)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" podUID="4c9ed38f-5c3a-4ea8-8969-0d2b506cc260" Oct 30 17:15:04.598031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.598008 6202 generic.go:356] "Generic (PLEG): container finished" podID="67afc1c4-1c1f-4c7b-9de3-df2899b44392" containerID="4f1d6fa3285db8df65dc2f59795f620d496286e8137b74f2c21162ddd73d7bba" exitCode=60 Oct 30 17:15:04.598893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.598463 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" event={"ID":"67afc1c4-1c1f-4c7b-9de3-df2899b44392","Type":"ContainerDied","Data":"4f1d6fa3285db8df65dc2f59795f620d496286e8137b74f2c21162ddd73d7bba"} Oct 30 17:15:04.600270 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.600251 6202 scope.go:122] "RemoveContainer" containerID="4f1d6fa3285db8df65dc2f59795f620d496286e8137b74f2c21162ddd73d7bba" Oct 30 17:15:04.600949 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:04.600925 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg_odh-ai-gateway-infra(67afc1c4-1c1f-4c7b-9de3-df2899b44392)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" podUID="67afc1c4-1c1f-4c7b-9de3-df2899b44392" Oct 30 17:15:04.607581 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.607413 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" event={"ID":"3fa542e0-72f9-4e5b-ab62-25fab055df11","Type":"ContainerStarted","Data":"ef43905129b1ca42dc0a7fd4ef64242d46006c84f0d04716561a7d7121c68b57"} Oct 30 17:15:04.613444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.613239 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" event={"ID":"fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf","Type":"ContainerStarted","Data":"6837bb970432bcc79fefd716b8d2b2d1d312270b5ead10fd1b9f64699512103c"} Oct 30 17:15:04.616506 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.616377 6202 scope.go:122] "RemoveContainer" containerID="8908d6d6ec14111f77ba73efc17304c62aae2490e67fe65d2326eedadfb9924d" Oct 30 17:15:04.619384 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.619361 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" event={"ID":"7ad83a2b-ebfd-4902-9b86-bfdc8552f763","Type":"ContainerStarted","Data":"03b56cb62b5da0a976344ef42446c2cff73ae4cb8db4a5809e1891f8f687e41d"} Oct 30 17:15:04.622739 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.622711 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" event={"ID":"3fdeaeb1-059f-49ac-864d-0b5a4a504a50","Type":"ContainerStarted","Data":"c7d66d14e5144be8eb149f90d4f1b959dcd10f7c8d628a19f4b4ded0e1828909"} Oct 30 17:15:04.625652 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.625612 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" event={"ID":"9a22ef35-5fa6-4126-9677-94883fe7fe72","Type":"ContainerStarted","Data":"9caae8bc289737557f5ec4b62c13f34799ece22ae8a407363f2c493b339de7b8"} Oct 30 17:15:04.627820 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.627795 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" event={"ID":"e9e9f54b-cee4-43c3-96b5-e75eaf17adad","Type":"ContainerStarted","Data":"5e80fb6fc6c153cf79cd3d168086d4514af908a310d47b9f3fd670685f69e83e"} Oct 30 17:15:04.819508 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.819473 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:04.953435 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:04.953381 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:05.181275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.181244 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:05.308965 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.308886 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:05.649439 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.649394 6202 generic.go:356] "Generic (PLEG): container finished" podID="3fa542e0-72f9-4e5b-ab62-25fab055df11" containerID="ef43905129b1ca42dc0a7fd4ef64242d46006c84f0d04716561a7d7121c68b57" exitCode=60 Oct 30 17:15:05.649944 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.649484 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" event={"ID":"3fa542e0-72f9-4e5b-ab62-25fab055df11","Type":"ContainerDied","Data":"ef43905129b1ca42dc0a7fd4ef64242d46006c84f0d04716561a7d7121c68b57"} Oct 30 17:15:05.649944 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.649544 6202 scope.go:122] "RemoveContainer" containerID="e84f89eaac353df8e223f88900f4556cd5fd76ce9dbca184aade074620fb0fb3" Oct 30 17:15:05.650101 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.650082 6202 scope.go:122] "RemoveContainer" containerID="ef43905129b1ca42dc0a7fd4ef64242d46006c84f0d04716561a7d7121c68b57" Oct 30 17:15:05.650598 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:05.650560 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2_odh-ai-gateway-infra(3fa542e0-72f9-4e5b-ab62-25fab055df11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" podUID="3fa542e0-72f9-4e5b-ab62-25fab055df11" Oct 30 17:15:05.656570 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.655545 6202 generic.go:356] "Generic (PLEG): container finished" podID="fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf" containerID="6837bb970432bcc79fefd716b8d2b2d1d312270b5ead10fd1b9f64699512103c" exitCode=60 Oct 30 17:15:05.656570 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.655626 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" event={"ID":"fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf","Type":"ContainerDied","Data":"6837bb970432bcc79fefd716b8d2b2d1d312270b5ead10fd1b9f64699512103c"} Oct 30 17:15:05.656570 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.656099 6202 scope.go:122] "RemoveContainer" containerID="6837bb970432bcc79fefd716b8d2b2d1d312270b5ead10fd1b9f64699512103c" Oct 30 17:15:05.656570 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:05.656493 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn_odh-ai-gateway-infra(fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" podUID="fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf" Oct 30 17:15:05.660319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.659732 6202 generic.go:356] "Generic (PLEG): container finished" podID="7ad83a2b-ebfd-4902-9b86-bfdc8552f763" containerID="03b56cb62b5da0a976344ef42446c2cff73ae4cb8db4a5809e1891f8f687e41d" exitCode=60 Oct 30 17:15:05.660319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.659837 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" event={"ID":"7ad83a2b-ebfd-4902-9b86-bfdc8552f763","Type":"ContainerDied","Data":"03b56cb62b5da0a976344ef42446c2cff73ae4cb8db4a5809e1891f8f687e41d"} Oct 30 17:15:05.660319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.659881 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7"] Oct 30 17:15:05.660319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.660223 6202 scope.go:122] "RemoveContainer" containerID="03b56cb62b5da0a976344ef42446c2cff73ae4cb8db4a5809e1891f8f687e41d" Oct 30 17:15:05.660966 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:05.660809 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h_odh-ai-gateway-infra(7ad83a2b-ebfd-4902-9b86-bfdc8552f763)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" podUID="7ad83a2b-ebfd-4902-9b86-bfdc8552f763" Oct 30 17:15:05.661063 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.660976 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" podUID="bbcf3ac0-0197-4960-8672-2fd5d9053729" containerName="istio-proxy" containerID="cri-o://829c0276f97159669765ababa07fcd992cc8472b9a4ffb144489de9795365a89" gracePeriod=30 Oct 30 17:15:05.663183 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.663164 6202 generic.go:356] "Generic (PLEG): container finished" podID="3fdeaeb1-059f-49ac-864d-0b5a4a504a50" containerID="c7d66d14e5144be8eb149f90d4f1b959dcd10f7c8d628a19f4b4ded0e1828909" exitCode=60 Oct 30 17:15:05.663329 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.663311 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" event={"ID":"3fdeaeb1-059f-49ac-864d-0b5a4a504a50","Type":"ContainerDied","Data":"c7d66d14e5144be8eb149f90d4f1b959dcd10f7c8d628a19f4b4ded0e1828909"} Oct 30 17:15:05.663644 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.663628 6202 scope.go:122] "RemoveContainer" containerID="c7d66d14e5144be8eb149f90d4f1b959dcd10f7c8d628a19f4b4ded0e1828909" Oct 30 17:15:05.664083 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:05.664055 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk_odh-ai-gateway-infra(3fdeaeb1-059f-49ac-864d-0b5a4a504a50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" podUID="3fdeaeb1-059f-49ac-864d-0b5a4a504a50" Oct 30 17:15:05.666403 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.666350 6202 generic.go:356] "Generic (PLEG): container finished" podID="9a22ef35-5fa6-4126-9677-94883fe7fe72" containerID="9caae8bc289737557f5ec4b62c13f34799ece22ae8a407363f2c493b339de7b8" exitCode=60 Oct 30 17:15:05.666510 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.666427 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" event={"ID":"9a22ef35-5fa6-4126-9677-94883fe7fe72","Type":"ContainerDied","Data":"9caae8bc289737557f5ec4b62c13f34799ece22ae8a407363f2c493b339de7b8"} Oct 30 17:15:05.667392 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.667000 6202 scope.go:122] "RemoveContainer" containerID="9caae8bc289737557f5ec4b62c13f34799ece22ae8a407363f2c493b339de7b8" Oct 30 17:15:05.667392 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:05.667339 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7_odh-ai-gateway-infra(9a22ef35-5fa6-4126-9677-94883fe7fe72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" podUID="9a22ef35-5fa6-4126-9677-94883fe7fe72" Oct 30 17:15:05.669137 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.669115 6202 generic.go:356] "Generic (PLEG): container finished" podID="e9e9f54b-cee4-43c3-96b5-e75eaf17adad" containerID="5e80fb6fc6c153cf79cd3d168086d4514af908a310d47b9f3fd670685f69e83e" exitCode=60 Oct 30 17:15:05.669326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.669154 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" event={"ID":"e9e9f54b-cee4-43c3-96b5-e75eaf17adad","Type":"ContainerDied","Data":"5e80fb6fc6c153cf79cd3d168086d4514af908a310d47b9f3fd670685f69e83e"} Oct 30 17:15:05.669577 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.669550 6202 scope.go:122] "RemoveContainer" containerID="5e80fb6fc6c153cf79cd3d168086d4514af908a310d47b9f3fd670685f69e83e" Oct 30 17:15:05.670117 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:05.670012 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp_odh-ai-gateway-infra(e9e9f54b-cee4-43c3-96b5-e75eaf17adad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" podUID="e9e9f54b-cee4-43c3-96b5-e75eaf17adad" Oct 30 17:15:05.671878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.671857 6202 generic.go:356] "Generic (PLEG): container finished" podID="c4193469-f0ea-48f5-af10-3479e0980736" containerID="0f5cb560a320851bd3d2879c92836cb73e18696874c7005e55bed908fceaac23" exitCode=60 Oct 30 17:15:05.671977 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.671955 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" event={"ID":"c4193469-f0ea-48f5-af10-3479e0980736","Type":"ContainerDied","Data":"0f5cb560a320851bd3d2879c92836cb73e18696874c7005e55bed908fceaac23"} Oct 30 17:15:05.672273 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.672257 6202 scope.go:122] "RemoveContainer" containerID="0f5cb560a320851bd3d2879c92836cb73e18696874c7005e55bed908fceaac23" Oct 30 17:15:05.672560 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:05.672539 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94_odh-ai-gateway-infra(c4193469-f0ea-48f5-af10-3479e0980736)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" podUID="c4193469-f0ea-48f5-af10-3479e0980736" Oct 30 17:15:05.675380 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.675357 6202 generic.go:356] "Generic (PLEG): container finished" podID="11be79af-8b98-43ef-8880-3dc9ac2c3a6f" containerID="f5cc5fda39e4431cd635df047e84d6c1e0dc6986d2a1a53d4bae0f4a654bafb0" exitCode=60 Oct 30 17:15:05.675524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.675498 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" event={"ID":"11be79af-8b98-43ef-8880-3dc9ac2c3a6f","Type":"ContainerDied","Data":"f5cc5fda39e4431cd635df047e84d6c1e0dc6986d2a1a53d4bae0f4a654bafb0"} Oct 30 17:15:05.676412 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.676391 6202 scope.go:122] "RemoveContainer" containerID="f5cc5fda39e4431cd635df047e84d6c1e0dc6986d2a1a53d4bae0f4a654bafb0" Oct 30 17:15:05.676955 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:05.676901 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p_odh-ai-gateway-infra(11be79af-8b98-43ef-8880-3dc9ac2c3a6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" podUID="11be79af-8b98-43ef-8880-3dc9ac2c3a6f" Oct 30 17:15:05.677796 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.677753 6202 generic.go:356] "Generic (PLEG): container finished" podID="3784ac5c-2448-4e4a-b17d-80717638a281" containerID="c94c333e9ca1242a28d9cc4c2e9bfc0c759245769ab27d45e5ed31551641a7a8" exitCode=60 Oct 30 17:15:05.677904 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.677794 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" event={"ID":"3784ac5c-2448-4e4a-b17d-80717638a281","Type":"ContainerDied","Data":"c94c333e9ca1242a28d9cc4c2e9bfc0c759245769ab27d45e5ed31551641a7a8"} Oct 30 17:15:05.678134 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.678111 6202 scope.go:122] "RemoveContainer" containerID="c94c333e9ca1242a28d9cc4c2e9bfc0c759245769ab27d45e5ed31551641a7a8" Oct 30 17:15:05.678401 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:05.678376 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5_odh-ai-gateway-infra(3784ac5c-2448-4e4a-b17d-80717638a281)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" podUID="3784ac5c-2448-4e4a-b17d-80717638a281" Oct 30 17:15:05.679688 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.679666 6202 generic.go:356] "Generic (PLEG): container finished" podID="d68156cc-8c00-43ec-a4eb-b0613b136629" containerID="72bdcb876f0b9a6e43aa1f26c616ec3be405fcb29857145a00ed91a5cf09eb5b" exitCode=0 Oct 30 17:15:05.679900 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.679799 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f" event={"ID":"d68156cc-8c00-43ec-a4eb-b0613b136629","Type":"ContainerDied","Data":"72bdcb876f0b9a6e43aa1f26c616ec3be405fcb29857145a00ed91a5cf09eb5b"} Oct 30 17:15:05.682068 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.682045 6202 scope.go:122] "RemoveContainer" containerID="b1ed244c488a8a779985aebc87dfb03a349d5f52c86f0ae9fe8df8ebea069721" Oct 30 17:15:05.682348 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:05.682322 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57_odh-ai-gateway-infra(4c9ed38f-5c3a-4ea8-8969-0d2b506cc260)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" podUID="4c9ed38f-5c3a-4ea8-8969-0d2b506cc260" Oct 30 17:15:05.684480 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.684455 6202 scope.go:122] "RemoveContainer" containerID="4f1d6fa3285db8df65dc2f59795f620d496286e8137b74f2c21162ddd73d7bba" Oct 30 17:15:05.684738 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:05.684712 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg_odh-ai-gateway-infra(67afc1c4-1c1f-4c7b-9de3-df2899b44392)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" podUID="67afc1c4-1c1f-4c7b-9de3-df2899b44392" Oct 30 17:15:05.686376 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.686353 6202 generic.go:356] "Generic (PLEG): container finished" podID="73d7018a-d448-4e25-afc4-2a002931f75f" containerID="dba6f2bfecc138ef8fa968da7b3a6d6b832b7a48eb9898361827e2a46a79f896" exitCode=60 Oct 30 17:15:05.686486 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.686435 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" event={"ID":"73d7018a-d448-4e25-afc4-2a002931f75f","Type":"ContainerDied","Data":"dba6f2bfecc138ef8fa968da7b3a6d6b832b7a48eb9898361827e2a46a79f896"} Oct 30 17:15:05.686610 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.686596 6202 scope.go:122] "RemoveContainer" containerID="dba6f2bfecc138ef8fa968da7b3a6d6b832b7a48eb9898361827e2a46a79f896" Oct 30 17:15:05.686832 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:05.686795 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d_odh-ai-gateway-infra(73d7018a-d448-4e25-afc4-2a002931f75f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" podUID="73d7018a-d448-4e25-afc4-2a002931f75f" Oct 30 17:15:05.697542 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.697516 6202 scope.go:122] "RemoveContainer" containerID="0b9e885f85d2de9a0621493324846b9574eb8579a0102e7af7e25911b7ea8125" Oct 30 17:15:05.717544 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.717527 6202 scope.go:122] "RemoveContainer" containerID="4a91d1c791051b31d3f0b23d516361504e0d9667576b8de7c810d5227df61b89" Oct 30 17:15:05.723282 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.723261 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:05.732261 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.732244 6202 scope.go:122] "RemoveContainer" containerID="2172d0ccf40e844e1c99865f1c8cbc97537fa6f74955bb5951c5f12e2f2c510c" Oct 30 17:15:05.749394 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.749310 6202 scope.go:122] "RemoveContainer" containerID="d337ea314ea35f3ce2bc20c1a153babb5e72bddf9b1abbee924a473b99bb5da6" Oct 30 17:15:05.906042 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.902198 6202 scope.go:122] "RemoveContainer" containerID="58efcf2d343c36606c0556b206fdfe26b128251c791727a2eb35a8e3e0c676fe" Oct 30 17:15:05.943363 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.943335 6202 scope.go:122] "RemoveContainer" containerID="f1da6266a582dbe7adeea274affa26d31489e4a48e7dba3396c08629d149d676" Oct 30 17:15:05.961679 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.961641 6202 scope.go:122] "RemoveContainer" containerID="f78b3c3c01da430e198e7e61513648b46311c43a29b43cb1451d343682f00595" Oct 30 17:15:05.977681 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.977657 6202 scope.go:122] "RemoveContainer" containerID="fb6ba931d24629fdc46f5e96fe9cfeea601379ae555233c3906a8a8c3b885972" Oct 30 17:15:05.992504 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:05.992479 6202 scope.go:122] "RemoveContainer" containerID="161c94d138dad5cc79daf29287a6559c3a8b4c1daa6461a29183cb6e54b87213" Oct 30 17:15:06.319393 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:06.319298 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:06.697958 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:06.697915 6202 scope.go:122] "RemoveContainer" containerID="ef43905129b1ca42dc0a7fd4ef64242d46006c84f0d04716561a7d7121c68b57" Oct 30 17:15:06.698394 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:06.698353 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2_odh-ai-gateway-infra(3fa542e0-72f9-4e5b-ab62-25fab055df11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" podUID="3fa542e0-72f9-4e5b-ab62-25fab055df11" Oct 30 17:15:06.700031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:06.699998 6202 scope.go:122] "RemoveContainer" containerID="6837bb970432bcc79fefd716b8d2b2d1d312270b5ead10fd1b9f64699512103c" Oct 30 17:15:06.700407 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:06.700378 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn_odh-ai-gateway-infra(fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" podUID="fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf" Oct 30 17:15:06.701978 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:06.701957 6202 scope.go:122] "RemoveContainer" containerID="03b56cb62b5da0a976344ef42446c2cff73ae4cb8db4a5809e1891f8f687e41d" Oct 30 17:15:06.702323 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:06.702301 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h_odh-ai-gateway-infra(7ad83a2b-ebfd-4902-9b86-bfdc8552f763)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" podUID="7ad83a2b-ebfd-4902-9b86-bfdc8552f763" Oct 30 17:15:06.703706 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:06.703685 6202 scope.go:122] "RemoveContainer" containerID="c7d66d14e5144be8eb149f90d4f1b959dcd10f7c8d628a19f4b4ded0e1828909" Oct 30 17:15:06.704088 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:06.704051 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk_odh-ai-gateway-infra(3fdeaeb1-059f-49ac-864d-0b5a4a504a50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" podUID="3fdeaeb1-059f-49ac-864d-0b5a4a504a50" Oct 30 17:15:06.705448 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:06.705431 6202 scope.go:122] "RemoveContainer" containerID="9caae8bc289737557f5ec4b62c13f34799ece22ae8a407363f2c493b339de7b8" Oct 30 17:15:06.705700 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:06.705680 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7_odh-ai-gateway-infra(9a22ef35-5fa6-4126-9677-94883fe7fe72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" podUID="9a22ef35-5fa6-4126-9677-94883fe7fe72" Oct 30 17:15:06.711340 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:06.711311 6202 scope.go:122] "RemoveContainer" containerID="5e80fb6fc6c153cf79cd3d168086d4514af908a310d47b9f3fd670685f69e83e" Oct 30 17:15:06.712021 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:06.711990 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp_odh-ai-gateway-infra(e9e9f54b-cee4-43c3-96b5-e75eaf17adad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" podUID="e9e9f54b-cee4-43c3-96b5-e75eaf17adad" Oct 30 17:15:06.725617 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:06.725594 6202 scope.go:122] "RemoveContainer" containerID="f5cc5fda39e4431cd635df047e84d6c1e0dc6986d2a1a53d4bae0f4a654bafb0" Oct 30 17:15:06.725872 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:06.725845 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p_odh-ai-gateway-infra(11be79af-8b98-43ef-8880-3dc9ac2c3a6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" podUID="11be79af-8b98-43ef-8880-3dc9ac2c3a6f" Oct 30 17:15:06.727318 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:06.727296 6202 scope.go:122] "RemoveContainer" containerID="c94c333e9ca1242a28d9cc4c2e9bfc0c759245769ab27d45e5ed31551641a7a8" Oct 30 17:15:06.727549 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:06.727531 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5_odh-ai-gateway-infra(3784ac5c-2448-4e4a-b17d-80717638a281)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" podUID="3784ac5c-2448-4e4a-b17d-80717638a281" Oct 30 17:15:06.729221 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:06.728914 6202 scope.go:122] "RemoveContainer" containerID="dba6f2bfecc138ef8fa968da7b3a6d6b832b7a48eb9898361827e2a46a79f896" Oct 30 17:15:06.729221 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:06.729094 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d_odh-ai-gateway-infra(73d7018a-d448-4e25-afc4-2a002931f75f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" podUID="73d7018a-d448-4e25-afc4-2a002931f75f" Oct 30 17:15:06.897028 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:06.897001 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f" Oct 30 17:15:06.927933 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:06.927905 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d68156cc-8c00-43ec-a4eb-b0613b136629-kube-api-access-nz6hl\" (UniqueName: \"kubernetes.io/projected/d68156cc-8c00-43ec-a4eb-b0613b136629-kube-api-access-nz6hl\") pod \"d68156cc-8c00-43ec-a4eb-b0613b136629\" (UID: \"d68156cc-8c00-43ec-a4eb-b0613b136629\") " Oct 30 17:15:06.928145 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:06.928053 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d68156cc-8c00-43ec-a4eb-b0613b136629-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68156cc-8c00-43ec-a4eb-b0613b136629-maas-api-xks-ca\") pod \"d68156cc-8c00-43ec-a4eb-b0613b136629\" (UID: \"d68156cc-8c00-43ec-a4eb-b0613b136629\") " Oct 30 17:15:06.928238 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:06.928219 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d68156cc-8c00-43ec-a4eb-b0613b136629-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68156cc-8c00-43ec-a4eb-b0613b136629-maas-api-openshift-ca\") pod \"d68156cc-8c00-43ec-a4eb-b0613b136629\" (UID: \"d68156cc-8c00-43ec-a4eb-b0613b136629\") " Oct 30 17:15:06.928571 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:06.928541 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68156cc-8c00-43ec-a4eb-b0613b136629-maas-api-openshift-ca" pod "d68156cc-8c00-43ec-a4eb-b0613b136629" (UID: "d68156cc-8c00-43ec-a4eb-b0613b136629"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:06.928832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:06.928809 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68156cc-8c00-43ec-a4eb-b0613b136629-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:06.930702 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:06.930674 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d68156cc-8c00-43ec-a4eb-b0613b136629-maas-api-xks-ca" pod "d68156cc-8c00-43ec-a4eb-b0613b136629" (UID: "d68156cc-8c00-43ec-a4eb-b0613b136629"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:06.930826 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:06.930742 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68156cc-8c00-43ec-a4eb-b0613b136629-kube-api-access-nz6hl" pod "d68156cc-8c00-43ec-a4eb-b0613b136629" (UID: "d68156cc-8c00-43ec-a4eb-b0613b136629"). InnerVolumeSpecName "kube-api-access-nz6hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:07.029894 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.029807 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68156cc-8c00-43ec-a4eb-b0613b136629-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:07.029894 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.029842 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz6hl\" (UniqueName: \"kubernetes.io/projected/d68156cc-8c00-43ec-a4eb-b0613b136629-kube-api-access-nz6hl\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:07.741046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.741002 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f" event={"ID":"d68156cc-8c00-43ec-a4eb-b0613b136629","Type":"ContainerDied","Data":"15e5545725c45102dd8d5646e672949d320ae31d49fde9600a69226f6a9dade5"} Oct 30 17:15:07.741046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.741040 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f" Oct 30 17:15:07.741658 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.741075 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15e5545725c45102dd8d5646e672949d320ae31d49fde9600a69226f6a9dade5" Oct 30 17:15:07.748428 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.748398 6202 generic.go:356] "Generic (PLEG): container finished" podID="bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" containerID="0574e77ea7356cbbaf388ff42559d5378a52ea15581a759e3145645ce020b607" exitCode=0 Oct 30 17:15:07.748681 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.748660 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" event={"ID":"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9","Type":"ContainerDied","Data":"0574e77ea7356cbbaf388ff42559d5378a52ea15581a759e3145645ce020b607"} Oct 30 17:15:07.748792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.748722 6202 scope.go:122] "RemoveContainer" containerID="1ae008250328678a3eafc9e9c153fde7b99dba55cb6f0b19cbba29a784e1521b" Oct 30 17:15:07.789217 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.789176 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:15:07.841276 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.841238 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-workload-socket\") pod \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " Oct 30 17:15:07.841495 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.841291 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istiod-ca-cert\") pod \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " Oct 30 17:15:07.841495 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.841316 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-token\" (UniqueName: \"kubernetes.io/projected/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-token\") pod \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " Oct 30 17:15:07.841495 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.841375 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-workload-certs\") pod \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " Oct 30 17:15:07.841495 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.841395 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-envoy\") pod \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " Oct 30 17:15:07.841495 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.841440 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-credential-socket\") pod \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " Oct 30 17:15:07.841495 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.841473 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-kube-api-access-sjp4x\" (UniqueName: \"kubernetes.io/projected/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-kube-api-access-sjp4x\") pod \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " Oct 30 17:15:07.841864 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.841543 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-data\") pod \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " Oct 30 17:15:07.841864 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.841633 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-podinfo\") pod \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\" (UID: \"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9\") " Oct 30 17:15:07.842226 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.842196 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-workload-certs" pod "bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" (UID: "bcdb58e9-32c9-4901-8878-b2e85c2a1ee9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:07.842499 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.842479 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:07.842787 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.842736 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-workload-socket" pod "bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" (UID: "bcdb58e9-32c9-4901-8878-b2e85c2a1ee9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:07.843605 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.843573 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-data" pod "bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" (UID: "bcdb58e9-32c9-4901-8878-b2e85c2a1ee9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:07.843715 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.843653 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-credential-socket" pod "bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" (UID: "bcdb58e9-32c9-4901-8878-b2e85c2a1ee9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:07.843967 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.843947 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istiod-ca-cert" pod "bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" (UID: "bcdb58e9-32c9-4901-8878-b2e85c2a1ee9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:07.845837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.845810 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-podinfo" pod "bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" (UID: "bcdb58e9-32c9-4901-8878-b2e85c2a1ee9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:15:07.845954 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.845838 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-token" pod "bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" (UID: "bcdb58e9-32c9-4901-8878-b2e85c2a1ee9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:07.846270 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.846245 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-envoy" pod "bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" (UID: "bcdb58e9-32c9-4901-8878-b2e85c2a1ee9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:07.849205 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.849184 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-kube-api-access-sjp4x" pod "bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" (UID: "bcdb58e9-32c9-4901-8878-b2e85c2a1ee9"). InnerVolumeSpecName "kube-api-access-sjp4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:07.943576 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.943537 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:07.943576 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.943567 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:07.943576 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.943576 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:07.943851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.943589 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:07.943851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.943602 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:07.943851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.943611 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjp4x\" (UniqueName: \"kubernetes.io/projected/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-kube-api-access-sjp4x\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:07.943851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.943619 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:07.943851 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:07.943628 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:08.349795 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:08.349564 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:08.561663 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:08.561626 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:08.761058 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:08.760979 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" Oct 30 17:15:08.761500 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:08.760974 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6" event={"ID":"bcdb58e9-32c9-4901-8878-b2e85c2a1ee9","Type":"ContainerDied","Data":"643283fd2f68d8cda0892275545e23275fd21b8fde1a33f783be4dcf5fd07137"} Oct 30 17:15:08.761500 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:08.761178 6202 scope.go:122] "RemoveContainer" containerID="0574e77ea7356cbbaf388ff42559d5378a52ea15581a759e3145645ce020b607" Oct 30 17:15:08.818827 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:08.818788 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6"] Oct 30 17:15:08.831711 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:08.831680 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-698c71-openshift-default-67b7987db5-vhnk6"] Oct 30 17:15:09.543523 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:09.543485 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:10.135682 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:10.135642 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:10.412710 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:10.412626 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" path="/var/lib/kubelet/pods/bcdb58e9-32c9-4901-8878-b2e85c2a1ee9/volumes" Oct 30 17:15:10.986465 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:10.986440 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:15:11.103021 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.102094 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-credential-socket\") pod \"bbcf3ac0-0197-4960-8672-2fd5d9053729\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " Oct 30 17:15:11.103021 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.102188 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-podinfo\") pod \"bbcf3ac0-0197-4960-8672-2fd5d9053729\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " Oct 30 17:15:11.103021 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.102237 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-workload-certs\") pod \"bbcf3ac0-0197-4960-8672-2fd5d9053729\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " Oct 30 17:15:11.103021 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.102263 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-token\" (UniqueName: \"kubernetes.io/projected/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-token\") pod \"bbcf3ac0-0197-4960-8672-2fd5d9053729\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " Oct 30 17:15:11.103021 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.102317 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-workload-socket\") pod \"bbcf3ac0-0197-4960-8672-2fd5d9053729\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " Oct 30 17:15:11.103021 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.102370 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbcf3ac0-0197-4960-8672-2fd5d9053729-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbcf3ac0-0197-4960-8672-2fd5d9053729-istiod-ca-cert\") pod \"bbcf3ac0-0197-4960-8672-2fd5d9053729\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " Oct 30 17:15:11.103021 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.102427 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-credential-socket" pod "bbcf3ac0-0197-4960-8672-2fd5d9053729" (UID: "bbcf3ac0-0197-4960-8672-2fd5d9053729"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:11.103021 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.102476 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbcf3ac0-0197-4960-8672-2fd5d9053729-kube-api-access-djvvw\" (UniqueName: \"kubernetes.io/projected/bbcf3ac0-0197-4960-8672-2fd5d9053729-kube-api-access-djvvw\") pod \"bbcf3ac0-0197-4960-8672-2fd5d9053729\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " Oct 30 17:15:11.103021 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.102506 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-envoy\") pod \"bbcf3ac0-0197-4960-8672-2fd5d9053729\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " Oct 30 17:15:11.103021 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.102624 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-data\") pod \"bbcf3ac0-0197-4960-8672-2fd5d9053729\" (UID: \"bbcf3ac0-0197-4960-8672-2fd5d9053729\") " Oct 30 17:15:11.103021 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.103023 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbcf3ac0-0197-4960-8672-2fd5d9053729-istiod-ca-cert" pod "bbcf3ac0-0197-4960-8672-2fd5d9053729" (UID: "bbcf3ac0-0197-4960-8672-2fd5d9053729"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:11.103984 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.103082 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-workload-socket" pod "bbcf3ac0-0197-4960-8672-2fd5d9053729" (UID: "bbcf3ac0-0197-4960-8672-2fd5d9053729"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:11.103984 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.103361 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-data" pod "bbcf3ac0-0197-4960-8672-2fd5d9053729" (UID: "bbcf3ac0-0197-4960-8672-2fd5d9053729"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:11.103984 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.103739 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-workload-certs" pod "bbcf3ac0-0197-4960-8672-2fd5d9053729" (UID: "bbcf3ac0-0197-4960-8672-2fd5d9053729"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:11.104876 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.104751 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:11.104876 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.104819 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:11.104876 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.104835 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:11.104876 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.104849 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbcf3ac0-0197-4960-8672-2fd5d9053729-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:11.107364 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.107331 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-token" pod "bbcf3ac0-0197-4960-8672-2fd5d9053729" (UID: "bbcf3ac0-0197-4960-8672-2fd5d9053729"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:11.109387 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.109350 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-envoy" pod "bbcf3ac0-0197-4960-8672-2fd5d9053729" (UID: "bbcf3ac0-0197-4960-8672-2fd5d9053729"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:11.109669 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.109640 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbcf3ac0-0197-4960-8672-2fd5d9053729-kube-api-access-djvvw" pod "bbcf3ac0-0197-4960-8672-2fd5d9053729" (UID: "bbcf3ac0-0197-4960-8672-2fd5d9053729"). InnerVolumeSpecName "kube-api-access-djvvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:11.110186 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.110157 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-podinfo" pod "bbcf3ac0-0197-4960-8672-2fd5d9053729" (UID: "bbcf3ac0-0197-4960-8672-2fd5d9053729"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:15:11.206798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.206680 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:11.206798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.206712 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:11.206798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.206722 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:11.206798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.206730 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djvvw\" (UniqueName: \"kubernetes.io/projected/bbcf3ac0-0197-4960-8672-2fd5d9053729-kube-api-access-djvvw\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:11.206798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.206739 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbcf3ac0-0197-4960-8672-2fd5d9053729-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:11.796423 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.796387 6202 generic.go:356] "Generic (PLEG): container finished" podID="bbcf3ac0-0197-4960-8672-2fd5d9053729" containerID="829c0276f97159669765ababa07fcd992cc8472b9a4ffb144489de9795365a89" exitCode=0 Oct 30 17:15:11.796655 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.796444 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" Oct 30 17:15:11.796655 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.796466 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" event={"ID":"bbcf3ac0-0197-4960-8672-2fd5d9053729","Type":"ContainerDied","Data":"829c0276f97159669765ababa07fcd992cc8472b9a4ffb144489de9795365a89"} Oct 30 17:15:11.796655 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.796527 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7" event={"ID":"bbcf3ac0-0197-4960-8672-2fd5d9053729","Type":"ContainerDied","Data":"6774cd66fac4e0c3903c2592811757f1133df4956a8212acad20694ba74542cd"} Oct 30 17:15:11.796655 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.796555 6202 scope.go:122] "RemoveContainer" containerID="829c0276f97159669765ababa07fcd992cc8472b9a4ffb144489de9795365a89" Oct 30 17:15:11.809594 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.809571 6202 scope.go:122] "RemoveContainer" containerID="44f4a301993e7495cd2e75baae830b425b878d7daabe9e253fe16b8483cd50df" Oct 30 17:15:11.822317 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.822297 6202 scope.go:122] "RemoveContainer" containerID="829c0276f97159669765ababa07fcd992cc8472b9a4ffb144489de9795365a89" Oct 30 17:15:11.822618 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:11.822597 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"829c0276f97159669765ababa07fcd992cc8472b9a4ffb144489de9795365a89\": container with ID starting with 829c0276f97159669765ababa07fcd992cc8472b9a4ffb144489de9795365a89 not found: ID does not exist" containerID="829c0276f97159669765ababa07fcd992cc8472b9a4ffb144489de9795365a89" Oct 30 17:15:11.822704 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.822627 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"829c0276f97159669765ababa07fcd992cc8472b9a4ffb144489de9795365a89"} err="failed to get container status \"829c0276f97159669765ababa07fcd992cc8472b9a4ffb144489de9795365a89\": rpc error: code = NotFound desc = could not find container \"829c0276f97159669765ababa07fcd992cc8472b9a4ffb144489de9795365a89\": container with ID starting with 829c0276f97159669765ababa07fcd992cc8472b9a4ffb144489de9795365a89 not found: ID does not exist" Oct 30 17:15:11.822704 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.822646 6202 scope.go:122] "RemoveContainer" containerID="44f4a301993e7495cd2e75baae830b425b878d7daabe9e253fe16b8483cd50df" Oct 30 17:15:11.822946 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:11.822924 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f4a301993e7495cd2e75baae830b425b878d7daabe9e253fe16b8483cd50df\": container with ID starting with 44f4a301993e7495cd2e75baae830b425b878d7daabe9e253fe16b8483cd50df not found: ID does not exist" containerID="44f4a301993e7495cd2e75baae830b425b878d7daabe9e253fe16b8483cd50df" Oct 30 17:15:11.823014 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.822956 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f4a301993e7495cd2e75baae830b425b878d7daabe9e253fe16b8483cd50df"} err="failed to get container status \"44f4a301993e7495cd2e75baae830b425b878d7daabe9e253fe16b8483cd50df\": rpc error: code = NotFound desc = could not find container \"44f4a301993e7495cd2e75baae830b425b878d7daabe9e253fe16b8483cd50df\": container with ID starting with 44f4a301993e7495cd2e75baae830b425b878d7daabe9e253fe16b8483cd50df not found: ID does not exist" Oct 30 17:15:11.879891 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.877880 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7"] Oct 30 17:15:11.887864 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:11.887832 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4def7e-openshift-default-58794569d7-55ts7"] Oct 30 17:15:12.413753 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:12.413721 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbcf3ac0-0197-4960-8672-2fd5d9053729" path="/var/lib/kubelet/pods/bbcf3ac0-0197-4960-8672-2fd5d9053729/volumes" Oct 30 17:15:12.474619 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:12.474579 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:13.548182 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:13.548146 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7"] Oct 30 17:15:13.566558 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:13.566523 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5"] Oct 30 17:15:13.566951 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:13.566913 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" podUID="84ccd5fe-cc47-482f-b2cb-e315080fae2a" containerName="istio-proxy" containerID="cri-o://c587cbb17d17c9cc9d611a6300d579abe2a9b4994f187f7d0c92f9620b070a0d" gracePeriod=30 Oct 30 17:15:13.689873 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:13.681349 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:13.804849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:13.801647 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:14.076461 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:14.076414 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:15:14.280893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:14.280829 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:15.403186 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:15.403153 6202 scope.go:122] "RemoveContainer" containerID="4f1d6fa3285db8df65dc2f59795f620d496286e8137b74f2c21162ddd73d7bba" Oct 30 17:15:15.403577 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:15.403265 6202 scope.go:122] "RemoveContainer" containerID="03b56cb62b5da0a976344ef42446c2cff73ae4cb8db4a5809e1891f8f687e41d" Oct 30 17:15:15.569077 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:15.569041 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:15.707791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:15.707355 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:16.408683 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:16.408650 6202 scope.go:122] "RemoveContainer" containerID="f5cc5fda39e4431cd635df047e84d6c1e0dc6986d2a1a53d4bae0f4a654bafb0" Oct 30 17:15:16.484445 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:16.484405 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:16.556049 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:16.554981 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb"] Oct 30 17:15:16.600025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:16.599995 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz"] Oct 30 17:15:16.621914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:16.621880 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:16.745340 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:16.745227 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:16.827440 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:16.827403 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj"] Oct 30 17:15:16.827668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:16.827640 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" podUID="6d43f09c-faf8-49d7-8c86-0cfd033cff9d" containerName="istio-proxy" containerID="cri-o://ace4c0667da616ff9ae2e597cad10f3c3bc37c6f062ebb8796fc8b930bff50db" gracePeriod=30 Oct 30 17:15:16.861307 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:16.861272 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:16.865304 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:16.865280 6202 generic.go:356] "Generic (PLEG): container finished" podID="67afc1c4-1c1f-4c7b-9de3-df2899b44392" containerID="667a2120048c09842b1ab9d39d586ee5094faf381d149f0a417919593042bc3a" exitCode=60 Oct 30 17:15:16.865456 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:16.865358 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" event={"ID":"67afc1c4-1c1f-4c7b-9de3-df2899b44392","Type":"ContainerDied","Data":"667a2120048c09842b1ab9d39d586ee5094faf381d149f0a417919593042bc3a"} Oct 30 17:15:16.865456 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:16.865427 6202 scope.go:122] "RemoveContainer" containerID="4f1d6fa3285db8df65dc2f59795f620d496286e8137b74f2c21162ddd73d7bba" Oct 30 17:15:16.866413 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:16.866198 6202 scope.go:122] "RemoveContainer" containerID="667a2120048c09842b1ab9d39d586ee5094faf381d149f0a417919593042bc3a" Oct 30 17:15:16.866750 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:16.866641 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg_odh-ai-gateway-infra(67afc1c4-1c1f-4c7b-9de3-df2899b44392)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" podUID="67afc1c4-1c1f-4c7b-9de3-df2899b44392" Oct 30 17:15:16.868152 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:16.868117 6202 generic.go:356] "Generic (PLEG): container finished" podID="7ad83a2b-ebfd-4902-9b86-bfdc8552f763" containerID="c720f991df0fbbcaeaeee082837d78551b8cccf19277e78031121734c63d03db" exitCode=60 Oct 30 17:15:16.868250 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:16.868151 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" event={"ID":"7ad83a2b-ebfd-4902-9b86-bfdc8552f763","Type":"ContainerDied","Data":"c720f991df0fbbcaeaeee082837d78551b8cccf19277e78031121734c63d03db"} Oct 30 17:15:16.868524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:16.868510 6202 scope.go:122] "RemoveContainer" containerID="c720f991df0fbbcaeaeee082837d78551b8cccf19277e78031121734c63d03db" Oct 30 17:15:16.868941 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:16.868922 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h_odh-ai-gateway-infra(7ad83a2b-ebfd-4902-9b86-bfdc8552f763)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" podUID="7ad83a2b-ebfd-4902-9b86-bfdc8552f763" Oct 30 17:15:16.886590 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:16.886567 6202 scope.go:122] "RemoveContainer" containerID="03b56cb62b5da0a976344ef42446c2cff73ae4cb8db4a5809e1891f8f687e41d" Oct 30 17:15:17.018878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:17.013807 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:17.403928 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:17.403894 6202 scope.go:122] "RemoveContainer" containerID="ef43905129b1ca42dc0a7fd4ef64242d46006c84f0d04716561a7d7121c68b57" Oct 30 17:15:17.404095 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:17.404015 6202 scope.go:122] "RemoveContainer" containerID="5e80fb6fc6c153cf79cd3d168086d4514af908a310d47b9f3fd670685f69e83e" Oct 30 17:15:17.404095 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:17.404078 6202 scope.go:122] "RemoveContainer" containerID="6837bb970432bcc79fefd716b8d2b2d1d312270b5ead10fd1b9f64699512103c" Oct 30 17:15:17.404446 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:17.404427 6202 scope.go:122] "RemoveContainer" containerID="c94c333e9ca1242a28d9cc4c2e9bfc0c759245769ab27d45e5ed31551641a7a8" Oct 30 17:15:17.882308 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:17.882269 6202 generic.go:356] "Generic (PLEG): container finished" podID="11be79af-8b98-43ef-8880-3dc9ac2c3a6f" containerID="10585cd1bf423066e75a9710711dc493522b443d80ad81bd61a4ac0cafc60cb8" exitCode=60 Oct 30 17:15:17.882799 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:17.882354 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" event={"ID":"11be79af-8b98-43ef-8880-3dc9ac2c3a6f","Type":"ContainerDied","Data":"10585cd1bf423066e75a9710711dc493522b443d80ad81bd61a4ac0cafc60cb8"} Oct 30 17:15:17.882799 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:17.882438 6202 scope.go:122] "RemoveContainer" containerID="f5cc5fda39e4431cd635df047e84d6c1e0dc6986d2a1a53d4bae0f4a654bafb0" Oct 30 17:15:17.882966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:17.882925 6202 scope.go:122] "RemoveContainer" containerID="10585cd1bf423066e75a9710711dc493522b443d80ad81bd61a4ac0cafc60cb8" Oct 30 17:15:17.883364 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:17.883330 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p_odh-ai-gateway-infra(11be79af-8b98-43ef-8880-3dc9ac2c3a6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" podUID="11be79af-8b98-43ef-8880-3dc9ac2c3a6f" Oct 30 17:15:17.934394 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:17.934309 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h"] Oct 30 17:15:18.027347 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.027295 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg"] Oct 30 17:15:18.156506 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.156187 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" Oct 30 17:15:18.219646 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.219620 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" Oct 30 17:15:18.255617 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.255565 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ad83a2b-ebfd-4902-9b86-bfdc8552f763-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad83a2b-ebfd-4902-9b86-bfdc8552f763-maas-api-service-ca\") pod \"7ad83a2b-ebfd-4902-9b86-bfdc8552f763\" (UID: \"7ad83a2b-ebfd-4902-9b86-bfdc8552f763\") " Oct 30 17:15:18.255916 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.255865 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad83a2b-ebfd-4902-9b86-bfdc8552f763-kube-api-access-fhqlr\" (UniqueName: \"kubernetes.io/projected/7ad83a2b-ebfd-4902-9b86-bfdc8552f763-kube-api-access-fhqlr\") pod \"7ad83a2b-ebfd-4902-9b86-bfdc8552f763\" (UID: \"7ad83a2b-ebfd-4902-9b86-bfdc8552f763\") " Oct 30 17:15:18.256648 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.256620 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad83a2b-ebfd-4902-9b86-bfdc8552f763-maas-api-service-ca" pod "7ad83a2b-ebfd-4902-9b86-bfdc8552f763" (UID: "7ad83a2b-ebfd-4902-9b86-bfdc8552f763"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:18.257061 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.257040 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad83a2b-ebfd-4902-9b86-bfdc8552f763-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:18.261948 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.261919 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad83a2b-ebfd-4902-9b86-bfdc8552f763-kube-api-access-fhqlr" pod "7ad83a2b-ebfd-4902-9b86-bfdc8552f763" (UID: "7ad83a2b-ebfd-4902-9b86-bfdc8552f763"). InnerVolumeSpecName "kube-api-access-fhqlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:18.365508 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.364645 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67afc1c4-1c1f-4c7b-9de3-df2899b44392-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67afc1c4-1c1f-4c7b-9de3-df2899b44392-maas-api-service-ca\") pod \"67afc1c4-1c1f-4c7b-9de3-df2899b44392\" (UID: \"67afc1c4-1c1f-4c7b-9de3-df2899b44392\") " Oct 30 17:15:18.365508 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.364909 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67afc1c4-1c1f-4c7b-9de3-df2899b44392-kube-api-access-95rs7\" (UniqueName: \"kubernetes.io/projected/67afc1c4-1c1f-4c7b-9de3-df2899b44392-kube-api-access-95rs7\") pod \"67afc1c4-1c1f-4c7b-9de3-df2899b44392\" (UID: \"67afc1c4-1c1f-4c7b-9de3-df2899b44392\") " Oct 30 17:15:18.366143 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.366112 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhqlr\" (UniqueName: \"kubernetes.io/projected/7ad83a2b-ebfd-4902-9b86-bfdc8552f763-kube-api-access-fhqlr\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:18.367062 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.367033 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67afc1c4-1c1f-4c7b-9de3-df2899b44392-maas-api-service-ca" pod "67afc1c4-1c1f-4c7b-9de3-df2899b44392" (UID: "67afc1c4-1c1f-4c7b-9de3-df2899b44392"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:18.374168 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.374144 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67afc1c4-1c1f-4c7b-9de3-df2899b44392-kube-api-access-95rs7" pod "67afc1c4-1c1f-4c7b-9de3-df2899b44392" (UID: "67afc1c4-1c1f-4c7b-9de3-df2899b44392"). InnerVolumeSpecName "kube-api-access-95rs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:18.402974 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.402941 6202 scope.go:122] "RemoveContainer" containerID="dba6f2bfecc138ef8fa968da7b3a6d6b832b7a48eb9898361827e2a46a79f896" Oct 30 17:15:18.403136 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.403117 6202 scope.go:122] "RemoveContainer" containerID="b1ed244c488a8a779985aebc87dfb03a349d5f52c86f0ae9fe8df8ebea069721" Oct 30 17:15:18.403255 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.403238 6202 scope.go:122] "RemoveContainer" containerID="c7d66d14e5144be8eb149f90d4f1b959dcd10f7c8d628a19f4b4ded0e1828909" Oct 30 17:15:18.467688 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.467615 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67afc1c4-1c1f-4c7b-9de3-df2899b44392-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:18.467688 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.467645 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95rs7\" (UniqueName: \"kubernetes.io/projected/67afc1c4-1c1f-4c7b-9de3-df2899b44392-kube-api-access-95rs7\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:18.551319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.549928 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" podUID="2d98a198-b4de-422f-a2f4-df5e059d105d" containerName="payload-processing" containerID="cri-o://e1c17a9be77260afe59bef057caab8c8060c9f1e911ee1b0533718c9f7559cfd" gracePeriod=25 Oct 30 17:15:18.856223 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.856191 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:18.929403 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.929370 6202 generic.go:356] "Generic (PLEG): container finished" podID="e9e9f54b-cee4-43c3-96b5-e75eaf17adad" containerID="4a375662eae96b980b90314a24d8f5eb31db8a3618c9a63f466e219f268c1e27" exitCode=60 Oct 30 17:15:18.929880 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.929444 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" event={"ID":"e9e9f54b-cee4-43c3-96b5-e75eaf17adad","Type":"ContainerDied","Data":"4a375662eae96b980b90314a24d8f5eb31db8a3618c9a63f466e219f268c1e27"} Oct 30 17:15:18.929880 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.929515 6202 scope.go:122] "RemoveContainer" containerID="5e80fb6fc6c153cf79cd3d168086d4514af908a310d47b9f3fd670685f69e83e" Oct 30 17:15:18.930058 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.930043 6202 scope.go:122] "RemoveContainer" containerID="4a375662eae96b980b90314a24d8f5eb31db8a3618c9a63f466e219f268c1e27" Oct 30 17:15:18.930446 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:18.930424 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp_odh-ai-gateway-infra(e9e9f54b-cee4-43c3-96b5-e75eaf17adad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" podUID="e9e9f54b-cee4-43c3-96b5-e75eaf17adad" Oct 30 17:15:18.933568 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.933545 6202 generic.go:356] "Generic (PLEG): container finished" podID="3784ac5c-2448-4e4a-b17d-80717638a281" containerID="cee1267e623a1799ea860745165487242c768ec9c8c9803cbc02c8c9a1ea94c2" exitCode=60 Oct 30 17:15:18.933685 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.933632 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" event={"ID":"3784ac5c-2448-4e4a-b17d-80717638a281","Type":"ContainerDied","Data":"cee1267e623a1799ea860745165487242c768ec9c8c9803cbc02c8c9a1ea94c2"} Oct 30 17:15:18.934084 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.934051 6202 scope.go:122] "RemoveContainer" containerID="cee1267e623a1799ea860745165487242c768ec9c8c9803cbc02c8c9a1ea94c2" Oct 30 17:15:18.934457 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:18.934423 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5_odh-ai-gateway-infra(3784ac5c-2448-4e4a-b17d-80717638a281)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" podUID="3784ac5c-2448-4e4a-b17d-80717638a281" Oct 30 17:15:18.935197 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.935178 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" Oct 30 17:15:18.935297 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.935221 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg" event={"ID":"67afc1c4-1c1f-4c7b-9de3-df2899b44392","Type":"ContainerDied","Data":"939ed0173eb3b7d0a780a054f83fc361b0821d80eb28e70c161a1cc83ae5b390"} Oct 30 17:15:18.937540 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.937516 6202 generic.go:356] "Generic (PLEG): container finished" podID="3fa542e0-72f9-4e5b-ab62-25fab055df11" containerID="de7d01cebc8fdcd0c9bb1330085ae1686d76e160c893a2feac64fe1d2de467b9" exitCode=60 Oct 30 17:15:18.937653 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.937582 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" event={"ID":"3fa542e0-72f9-4e5b-ab62-25fab055df11","Type":"ContainerDied","Data":"de7d01cebc8fdcd0c9bb1330085ae1686d76e160c893a2feac64fe1d2de467b9"} Oct 30 17:15:18.937966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.937949 6202 scope.go:122] "RemoveContainer" containerID="de7d01cebc8fdcd0c9bb1330085ae1686d76e160c893a2feac64fe1d2de467b9" Oct 30 17:15:18.938227 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:18.938205 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2_odh-ai-gateway-infra(3fa542e0-72f9-4e5b-ab62-25fab055df11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" podUID="3fa542e0-72f9-4e5b-ab62-25fab055df11" Oct 30 17:15:18.938581 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.938554 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p"] Oct 30 17:15:18.939964 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.939944 6202 generic.go:356] "Generic (PLEG): container finished" podID="fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf" containerID="6b19b84f34808fc38c6340b4cf8de3f9554c179b054620736c6c8fbac2d7edf2" exitCode=60 Oct 30 17:15:18.940052 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.940017 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" event={"ID":"fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf","Type":"ContainerDied","Data":"6b19b84f34808fc38c6340b4cf8de3f9554c179b054620736c6c8fbac2d7edf2"} Oct 30 17:15:18.940388 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.940368 6202 scope.go:122] "RemoveContainer" containerID="6b19b84f34808fc38c6340b4cf8de3f9554c179b054620736c6c8fbac2d7edf2" Oct 30 17:15:18.940703 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:18.940632 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn_odh-ai-gateway-infra(fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" podUID="fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf" Oct 30 17:15:18.941863 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.941846 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" Oct 30 17:15:18.941982 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.941878 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h" event={"ID":"7ad83a2b-ebfd-4902-9b86-bfdc8552f763","Type":"ContainerDied","Data":"332a3aadf67ba39102f5b45a5746223405c41aeb4960e8c5c10b90ecb0b97c19"} Oct 30 17:15:18.943353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.943332 6202 generic.go:356] "Generic (PLEG): container finished" podID="84ccd5fe-cc47-482f-b2cb-e315080fae2a" containerID="c587cbb17d17c9cc9d611a6300d579abe2a9b4994f187f7d0c92f9620b070a0d" exitCode=0 Oct 30 17:15:18.943441 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.943384 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" event={"ID":"84ccd5fe-cc47-482f-b2cb-e315080fae2a","Type":"ContainerDied","Data":"c587cbb17d17c9cc9d611a6300d579abe2a9b4994f187f7d0c92f9620b070a0d"} Oct 30 17:15:18.943441 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.943413 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" event={"ID":"84ccd5fe-cc47-482f-b2cb-e315080fae2a","Type":"ContainerDied","Data":"3cf847654dcdb6998e22326dacd51f158a2e396d3c7e7362f053aae61dd0497b"} Oct 30 17:15:18.943441 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.943435 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cf847654dcdb6998e22326dacd51f158a2e396d3c7e7362f053aae61dd0497b" Oct 30 17:15:18.967673 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.967645 6202 scope.go:122] "RemoveContainer" containerID="c94c333e9ca1242a28d9cc4c2e9bfc0c759245769ab27d45e5ed31551641a7a8" Oct 30 17:15:18.981878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.980154 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:15:18.996585 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:18.996563 6202 scope.go:122] "RemoveContainer" containerID="667a2120048c09842b1ab9d39d586ee5094faf381d149f0a417919593042bc3a" Oct 30 17:15:19.008192 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.008151 6202 scope.go:122] "RemoveContainer" containerID="ef43905129b1ca42dc0a7fd4ef64242d46006c84f0d04716561a7d7121c68b57" Oct 30 17:15:19.029356 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.029321 6202 scope.go:122] "RemoveContainer" containerID="6837bb970432bcc79fefd716b8d2b2d1d312270b5ead10fd1b9f64699512103c" Oct 30 17:15:19.053049 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.052956 6202 scope.go:122] "RemoveContainer" containerID="c720f991df0fbbcaeaeee082837d78551b8cccf19277e78031121734c63d03db" Oct 30 17:15:19.079837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.078322 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-podinfo\") pod \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " Oct 30 17:15:19.079837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.078393 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-envoy\") pod \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " Oct 30 17:15:19.079837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.078422 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-workload-certs\") pod \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " Oct 30 17:15:19.079837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.078521 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-workload-socket\") pod \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " Oct 30 17:15:19.079837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.078578 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istiod-ca-cert\") pod \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " Oct 30 17:15:19.079837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.078629 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-data\") pod \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " Oct 30 17:15:19.079837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.078718 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-token\" (UniqueName: \"kubernetes.io/projected/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-token\") pod \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " Oct 30 17:15:19.079837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.078754 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-credential-socket\") pod \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " Oct 30 17:15:19.079837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.078844 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84ccd5fe-cc47-482f-b2cb-e315080fae2a-kube-api-access-k7pr6\" (UniqueName: \"kubernetes.io/projected/84ccd5fe-cc47-482f-b2cb-e315080fae2a-kube-api-access-k7pr6\") pod \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\" (UID: \"84ccd5fe-cc47-482f-b2cb-e315080fae2a\") " Oct 30 17:15:19.079837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.079418 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-workload-socket" pod "84ccd5fe-cc47-482f-b2cb-e315080fae2a" (UID: "84ccd5fe-cc47-482f-b2cb-e315080fae2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:19.079837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.079543 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-data" pod "84ccd5fe-cc47-482f-b2cb-e315080fae2a" (UID: "84ccd5fe-cc47-482f-b2cb-e315080fae2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:19.079837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.079859 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istiod-ca-cert" pod "84ccd5fe-cc47-482f-b2cb-e315080fae2a" (UID: "84ccd5fe-cc47-482f-b2cb-e315080fae2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:19.080610 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.080098 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-credential-socket" pod "84ccd5fe-cc47-482f-b2cb-e315080fae2a" (UID: "84ccd5fe-cc47-482f-b2cb-e315080fae2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:19.081460 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.080722 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-workload-certs" pod "84ccd5fe-cc47-482f-b2cb-e315080fae2a" (UID: "84ccd5fe-cc47-482f-b2cb-e315080fae2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:19.083949 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.082382 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.083949 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.082410 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.083949 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.082425 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.083949 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.082439 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.083949 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.082451 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.086681 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.086647 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ccd5fe-cc47-482f-b2cb-e315080fae2a-kube-api-access-k7pr6" pod "84ccd5fe-cc47-482f-b2cb-e315080fae2a" (UID: "84ccd5fe-cc47-482f-b2cb-e315080fae2a"). InnerVolumeSpecName "kube-api-access-k7pr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:19.093134 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.093084 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-podinfo" pod "84ccd5fe-cc47-482f-b2cb-e315080fae2a" (UID: "84ccd5fe-cc47-482f-b2cb-e315080fae2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:15:19.093885 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.093838 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-envoy" pod "84ccd5fe-cc47-482f-b2cb-e315080fae2a" (UID: "84ccd5fe-cc47-482f-b2cb-e315080fae2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:19.095000 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.094976 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-token" pod "84ccd5fe-cc47-482f-b2cb-e315080fae2a" (UID: "84ccd5fe-cc47-482f-b2cb-e315080fae2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:19.125624 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.125599 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" Oct 30 17:15:19.184830 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.184720 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.184830 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.184757 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7pr6\" (UniqueName: \"kubernetes.io/projected/84ccd5fe-cc47-482f-b2cb-e315080fae2a-kube-api-access-k7pr6\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.184830 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.184791 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.184830 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.184803 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ccd5fe-cc47-482f-b2cb-e315080fae2a-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.224311 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.224276 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h"] Oct 30 17:15:19.229194 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.229161 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-ae5afd-29889675-4lk7h"] Oct 30 17:15:19.270465 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.270358 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg"] Oct 30 17:15:19.274351 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.274321 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-de18d5-29889675-fnpkg"] Oct 30 17:15:19.285914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.285878 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11be79af-8b98-43ef-8880-3dc9ac2c3a6f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11be79af-8b98-43ef-8880-3dc9ac2c3a6f-maas-api-service-ca\") pod \"11be79af-8b98-43ef-8880-3dc9ac2c3a6f\" (UID: \"11be79af-8b98-43ef-8880-3dc9ac2c3a6f\") " Oct 30 17:15:19.286089 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.285941 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11be79af-8b98-43ef-8880-3dc9ac2c3a6f-kube-api-access-7vzm7\" (UniqueName: \"kubernetes.io/projected/11be79af-8b98-43ef-8880-3dc9ac2c3a6f-kube-api-access-7vzm7\") pod \"11be79af-8b98-43ef-8880-3dc9ac2c3a6f\" (UID: \"11be79af-8b98-43ef-8880-3dc9ac2c3a6f\") " Oct 30 17:15:19.286342 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.286317 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11be79af-8b98-43ef-8880-3dc9ac2c3a6f-maas-api-service-ca" pod "11be79af-8b98-43ef-8880-3dc9ac2c3a6f" (UID: "11be79af-8b98-43ef-8880-3dc9ac2c3a6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:19.286598 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.286578 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11be79af-8b98-43ef-8880-3dc9ac2c3a6f-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.288349 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.288316 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11be79af-8b98-43ef-8880-3dc9ac2c3a6f-kube-api-access-7vzm7" pod "11be79af-8b98-43ef-8880-3dc9ac2c3a6f" (UID: "11be79af-8b98-43ef-8880-3dc9ac2c3a6f"). InnerVolumeSpecName "kube-api-access-7vzm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:19.388133 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.388097 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vzm7\" (UniqueName: \"kubernetes.io/projected/11be79af-8b98-43ef-8880-3dc9ac2c3a6f-kube-api-access-7vzm7\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.403798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.403363 6202 scope.go:122] "RemoveContainer" containerID="9caae8bc289737557f5ec4b62c13f34799ece22ae8a407363f2c493b339de7b8" Oct 30 17:15:19.956415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.956371 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" event={"ID":"9a22ef35-5fa6-4126-9677-94883fe7fe72","Type":"ContainerStarted","Data":"8974ffa474e8332801fc3e4463f7de54f538137227de2ec1966ca612dcfbb9b1"} Oct 30 17:15:19.960137 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.960109 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" Oct 30 17:15:19.960279 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.960109 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p" event={"ID":"11be79af-8b98-43ef-8880-3dc9ac2c3a6f","Type":"ContainerDied","Data":"e029b9b9700f433febfa2f4f4ce5145b634d31e52c65598353924b5e270d445a"} Oct 30 17:15:19.960378 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.960291 6202 scope.go:122] "RemoveContainer" containerID="10585cd1bf423066e75a9710711dc493522b443d80ad81bd61a4ac0cafc60cb8" Oct 30 17:15:19.964602 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.964580 6202 generic.go:356] "Generic (PLEG): container finished" podID="4c9ed38f-5c3a-4ea8-8969-0d2b506cc260" containerID="1c6251effa5bdb02a8f90a794f5939087a2e5d9f4d5bf6037d4d4750983ec40f" exitCode=60 Oct 30 17:15:19.964747 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.964641 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" event={"ID":"4c9ed38f-5c3a-4ea8-8969-0d2b506cc260","Type":"ContainerDied","Data":"1c6251effa5bdb02a8f90a794f5939087a2e5d9f4d5bf6037d4d4750983ec40f"} Oct 30 17:15:19.965065 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.965048 6202 scope.go:122] "RemoveContainer" containerID="1c6251effa5bdb02a8f90a794f5939087a2e5d9f4d5bf6037d4d4750983ec40f" Oct 30 17:15:19.965350 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:19.965330 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57_odh-ai-gateway-infra(4c9ed38f-5c3a-4ea8-8969-0d2b506cc260)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" podUID="4c9ed38f-5c3a-4ea8-8969-0d2b506cc260" Oct 30 17:15:19.968534 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.968507 6202 generic.go:356] "Generic (PLEG): container finished" podID="73d7018a-d448-4e25-afc4-2a002931f75f" containerID="fa76854b89f68db714be94c8be66432bde17359a1b74f4e227acb5dce5562517" exitCode=60 Oct 30 17:15:19.968670 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.968531 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" event={"ID":"73d7018a-d448-4e25-afc4-2a002931f75f","Type":"ContainerDied","Data":"fa76854b89f68db714be94c8be66432bde17359a1b74f4e227acb5dce5562517"} Oct 30 17:15:19.968951 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.968932 6202 scope.go:122] "RemoveContainer" containerID="fa76854b89f68db714be94c8be66432bde17359a1b74f4e227acb5dce5562517" Oct 30 17:15:19.969267 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:19.969247 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d_odh-ai-gateway-infra(73d7018a-d448-4e25-afc4-2a002931f75f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" podUID="73d7018a-d448-4e25-afc4-2a002931f75f" Oct 30 17:15:19.976542 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.976518 6202 generic.go:356] "Generic (PLEG): container finished" podID="3fdeaeb1-059f-49ac-864d-0b5a4a504a50" containerID="9d0925b9c4fb6d850a207e6bedbd91f514e7d7176c0e13a4d80ecfbd247d236d" exitCode=60 Oct 30 17:15:19.976675 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.976595 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" event={"ID":"3fdeaeb1-059f-49ac-864d-0b5a4a504a50","Type":"ContainerDied","Data":"9d0925b9c4fb6d850a207e6bedbd91f514e7d7176c0e13a4d80ecfbd247d236d"} Oct 30 17:15:19.976675 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.976612 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5" Oct 30 17:15:19.977321 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.977284 6202 scope.go:122] "RemoveContainer" containerID="9d0925b9c4fb6d850a207e6bedbd91f514e7d7176c0e13a4d80ecfbd247d236d" Oct 30 17:15:19.977609 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:19.977588 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk_odh-ai-gateway-infra(3fdeaeb1-059f-49ac-864d-0b5a4a504a50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" podUID="3fdeaeb1-059f-49ac-864d-0b5a4a504a50" Oct 30 17:15:19.982831 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.982808 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp"] Oct 30 17:15:19.984121 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:19.984103 6202 scope.go:122] "RemoveContainer" containerID="b1ed244c488a8a779985aebc87dfb03a349d5f52c86f0ae9fe8df8ebea069721" Oct 30 17:15:20.000753 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.000724 6202 scope.go:122] "RemoveContainer" containerID="dba6f2bfecc138ef8fa968da7b3a6d6b832b7a48eb9898361827e2a46a79f896" Oct 30 17:15:20.027756 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.027731 6202 scope.go:122] "RemoveContainer" containerID="c7d66d14e5144be8eb149f90d4f1b959dcd10f7c8d628a19f4b4ded0e1828909" Oct 30 17:15:20.031497 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.031470 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2"] Oct 30 17:15:20.093791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.092999 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn"] Oct 30 17:15:20.144560 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.144531 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5"] Oct 30 17:15:20.165579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.160576 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5"] Oct 30 17:15:20.178651 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.178620 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-9rgp5"] Oct 30 17:15:20.206468 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.206433 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p"] Oct 30 17:15:20.207683 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.207656 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" Oct 30 17:15:20.210978 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.210951 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05810b-29889675-9h65p"] Oct 30 17:15:20.302652 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.302618 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" Oct 30 17:15:20.307735 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.306948 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e9f54b-cee4-43c3-96b5-e75eaf17adad-kube-api-access-s5tgw\" (UniqueName: \"kubernetes.io/projected/e9e9f54b-cee4-43c3-96b5-e75eaf17adad-kube-api-access-s5tgw\") pod \"e9e9f54b-cee4-43c3-96b5-e75eaf17adad\" (UID: \"e9e9f54b-cee4-43c3-96b5-e75eaf17adad\") " Oct 30 17:15:20.307735 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.307130 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e9f54b-cee4-43c3-96b5-e75eaf17adad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e9f54b-cee4-43c3-96b5-e75eaf17adad-maas-api-service-ca\") pod \"e9e9f54b-cee4-43c3-96b5-e75eaf17adad\" (UID: \"e9e9f54b-cee4-43c3-96b5-e75eaf17adad\") " Oct 30 17:15:20.307938 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.307755 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e9f54b-cee4-43c3-96b5-e75eaf17adad-maas-api-service-ca" pod "e9e9f54b-cee4-43c3-96b5-e75eaf17adad" (UID: "e9e9f54b-cee4-43c3-96b5-e75eaf17adad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:20.311459 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.311426 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e9f54b-cee4-43c3-96b5-e75eaf17adad-kube-api-access-s5tgw" pod "e9e9f54b-cee4-43c3-96b5-e75eaf17adad" (UID: "e9e9f54b-cee4-43c3-96b5-e75eaf17adad"). InnerVolumeSpecName "kube-api-access-s5tgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:20.378155 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.378122 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" Oct 30 17:15:20.403161 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.403132 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" Oct 30 17:15:20.403575 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.403556 6202 scope.go:122] "RemoveContainer" containerID="0f5cb560a320851bd3d2879c92836cb73e18696874c7005e55bed908fceaac23" Oct 30 17:15:20.408743 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.408718 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa542e0-72f9-4e5b-ab62-25fab055df11-kube-api-access-7hjtt\" (UniqueName: \"kubernetes.io/projected/3fa542e0-72f9-4e5b-ab62-25fab055df11-kube-api-access-7hjtt\") pod \"3fa542e0-72f9-4e5b-ab62-25fab055df11\" (UID: \"3fa542e0-72f9-4e5b-ab62-25fab055df11\") " Oct 30 17:15:20.408895 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.408876 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa542e0-72f9-4e5b-ab62-25fab055df11-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa542e0-72f9-4e5b-ab62-25fab055df11-maas-api-service-ca\") pod \"3fa542e0-72f9-4e5b-ab62-25fab055df11\" (UID: \"3fa542e0-72f9-4e5b-ab62-25fab055df11\") " Oct 30 17:15:20.409381 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.409361 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5tgw\" (UniqueName: \"kubernetes.io/projected/e9e9f54b-cee4-43c3-96b5-e75eaf17adad-kube-api-access-s5tgw\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.409381 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.409368 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa542e0-72f9-4e5b-ab62-25fab055df11-maas-api-service-ca" pod "3fa542e0-72f9-4e5b-ab62-25fab055df11" (UID: "3fa542e0-72f9-4e5b-ab62-25fab055df11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:20.409506 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.409385 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e9f54b-cee4-43c3-96b5-e75eaf17adad-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.411344 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.411316 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa542e0-72f9-4e5b-ab62-25fab055df11-kube-api-access-7hjtt" pod "3fa542e0-72f9-4e5b-ab62-25fab055df11" (UID: "3fa542e0-72f9-4e5b-ab62-25fab055df11"). InnerVolumeSpecName "kube-api-access-7hjtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:20.413599 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.413576 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11be79af-8b98-43ef-8880-3dc9ac2c3a6f" path="/var/lib/kubelet/pods/11be79af-8b98-43ef-8880-3dc9ac2c3a6f/volumes" Oct 30 17:15:20.414028 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.414013 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67afc1c4-1c1f-4c7b-9de3-df2899b44392" path="/var/lib/kubelet/pods/67afc1c4-1c1f-4c7b-9de3-df2899b44392/volumes" Oct 30 17:15:20.414391 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.414377 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad83a2b-ebfd-4902-9b86-bfdc8552f763" path="/var/lib/kubelet/pods/7ad83a2b-ebfd-4902-9b86-bfdc8552f763/volumes" Oct 30 17:15:20.415118 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.415102 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84ccd5fe-cc47-482f-b2cb-e315080fae2a" path="/var/lib/kubelet/pods/84ccd5fe-cc47-482f-b2cb-e315080fae2a/volumes" Oct 30 17:15:20.510533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.510504 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3784ac5c-2448-4e4a-b17d-80717638a281-kube-api-access-gb9rv\" (UniqueName: \"kubernetes.io/projected/3784ac5c-2448-4e4a-b17d-80717638a281-kube-api-access-gb9rv\") pod \"3784ac5c-2448-4e4a-b17d-80717638a281\" (UID: \"3784ac5c-2448-4e4a-b17d-80717638a281\") " Oct 30 17:15:20.510699 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.510657 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3784ac5c-2448-4e4a-b17d-80717638a281-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3784ac5c-2448-4e4a-b17d-80717638a281-maas-api-service-ca\") pod \"3784ac5c-2448-4e4a-b17d-80717638a281\" (UID: \"3784ac5c-2448-4e4a-b17d-80717638a281\") " Oct 30 17:15:20.510752 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.510722 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf-maas-api-service-ca\") pod \"fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf\" (UID: \"fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf\") " Oct 30 17:15:20.510867 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.510847 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf-kube-api-access-g554k\" (UniqueName: \"kubernetes.io/projected/fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf-kube-api-access-g554k\") pod \"fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf\" (UID: \"fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf\") " Oct 30 17:15:20.511695 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.511209 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3784ac5c-2448-4e4a-b17d-80717638a281-maas-api-service-ca" pod "3784ac5c-2448-4e4a-b17d-80717638a281" (UID: "3784ac5c-2448-4e4a-b17d-80717638a281"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:20.511695 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.511265 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf-maas-api-service-ca" pod "fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf" (UID: "fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:20.511947 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.511826 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hjtt\" (UniqueName: \"kubernetes.io/projected/3fa542e0-72f9-4e5b-ab62-25fab055df11-kube-api-access-7hjtt\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.512022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.511963 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa542e0-72f9-4e5b-ab62-25fab055df11-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.512022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.511998 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3784ac5c-2448-4e4a-b17d-80717638a281-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.512022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.512012 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.513576 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.513548 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3784ac5c-2448-4e4a-b17d-80717638a281-kube-api-access-gb9rv" pod "3784ac5c-2448-4e4a-b17d-80717638a281" (UID: "3784ac5c-2448-4e4a-b17d-80717638a281"). InnerVolumeSpecName "kube-api-access-gb9rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:20.514108 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.514080 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf-kube-api-access-g554k" pod "fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf" (UID: "fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf"). InnerVolumeSpecName "kube-api-access-g554k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:20.615807 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.613384 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb9rv\" (UniqueName: \"kubernetes.io/projected/3784ac5c-2448-4e4a-b17d-80717638a281-kube-api-access-gb9rv\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.615807 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.613420 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g554k\" (UniqueName: \"kubernetes.io/projected/fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf-kube-api-access-g554k\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.652804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.652748 6202 patch_prober.go:29] interesting pod/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:15:20.652996 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.652827 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" podUID="6d43f09c-faf8-49d7-8c86-0cfd033cff9d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:15:20.986997 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.986962 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" Oct 30 17:15:20.987435 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.986962 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2" event={"ID":"3fa542e0-72f9-4e5b-ab62-25fab055df11","Type":"ContainerDied","Data":"18a5389450a13abc6107a2e92120349f6d3b5ba3eec8f4ada28355ddf9397f41"} Oct 30 17:15:20.987435 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.987130 6202 scope.go:122] "RemoveContainer" containerID="de7d01cebc8fdcd0c9bb1330085ae1686d76e160c893a2feac64fe1d2de467b9" Oct 30 17:15:20.988921 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.988892 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" event={"ID":"fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf","Type":"ContainerDied","Data":"8b8c0f552f73462679b062a1c90fd0bf186ea5348ff6402cc8aa95c2fce91197"} Oct 30 17:15:20.988921 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.988916 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn" Oct 30 17:15:20.999131 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.999108 6202 generic.go:356] "Generic (PLEG): container finished" podID="9a22ef35-5fa6-4126-9677-94883fe7fe72" containerID="8974ffa474e8332801fc3e4463f7de54f538137227de2ec1966ca612dcfbb9b1" exitCode=60 Oct 30 17:15:20.999276 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.999175 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" event={"ID":"9a22ef35-5fa6-4126-9677-94883fe7fe72","Type":"ContainerDied","Data":"8974ffa474e8332801fc3e4463f7de54f538137227de2ec1966ca612dcfbb9b1"} Oct 30 17:15:21.000347 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:20.999628 6202 scope.go:122] "RemoveContainer" containerID="8974ffa474e8332801fc3e4463f7de54f538137227de2ec1966ca612dcfbb9b1" Oct 30 17:15:21.000347 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:21.000041 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7_odh-ai-gateway-infra(9a22ef35-5fa6-4126-9677-94883fe7fe72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" podUID="9a22ef35-5fa6-4126-9677-94883fe7fe72" Oct 30 17:15:21.000347 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.000336 6202 scope.go:122] "RemoveContainer" containerID="6b19b84f34808fc38c6340b4cf8de3f9554c179b054620736c6c8fbac2d7edf2" Oct 30 17:15:21.000877 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.000855 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" Oct 30 17:15:21.000983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.000853 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp" event={"ID":"e9e9f54b-cee4-43c3-96b5-e75eaf17adad","Type":"ContainerDied","Data":"e0e61989e1a3e0fcbcd9f3fbe74adbafb2330ef510b5c843f7c8fc3f8ecca74b"} Oct 30 17:15:21.005815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.003854 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" event={"ID":"c4193469-f0ea-48f5-af10-3479e0980736","Type":"ContainerStarted","Data":"2a6b36f734ba02e5e3a36694efc08b43d00bec88429338eae0904ce86940df1a"} Oct 30 17:15:21.008426 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.008395 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" event={"ID":"3784ac5c-2448-4e4a-b17d-80717638a281","Type":"ContainerDied","Data":"d10621e22540be35a4c59203a76f074eec39cff3f3a14314bfa78b584ad5b54d"} Oct 30 17:15:21.009003 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.008495 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5" Oct 30 17:15:21.014133 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.014093 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7"] Oct 30 17:15:21.014535 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.014520 6202 scope.go:122] "RemoveContainer" containerID="9caae8bc289737557f5ec4b62c13f34799ece22ae8a407363f2c493b339de7b8" Oct 30 17:15:21.030169 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.030128 6202 scope.go:122] "RemoveContainer" containerID="4a375662eae96b980b90314a24d8f5eb31db8a3618c9a63f466e219f268c1e27" Oct 30 17:15:21.045810 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.045750 6202 scope.go:122] "RemoveContainer" containerID="cee1267e623a1799ea860745165487242c768ec9c8c9803cbc02c8c9a1ea94c2" Oct 30 17:15:21.049338 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.049305 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2"] Oct 30 17:15:21.056668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.056606 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-97a47d-29889675-httg2"] Oct 30 17:15:21.072673 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.072595 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d"] Oct 30 17:15:21.112855 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.112818 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk"] Oct 30 17:15:21.155948 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.155896 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn"] Oct 30 17:15:21.161849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.161816 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c8db42-29889675-dldbn"] Oct 30 17:15:21.264237 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.264113 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp"] Oct 30 17:15:21.326405 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.326375 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" Oct 30 17:15:21.350451 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.350409 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889675-m4bjp"] Oct 30 17:15:21.364120 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.364085 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57"] Oct 30 17:15:21.393426 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.393388 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5"] Oct 30 17:15:21.397008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.396977 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889675-5fgg5"] Oct 30 17:15:21.404843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.404819 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" Oct 30 17:15:21.428988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.428961 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d7018a-d448-4e25-afc4-2a002931f75f-kube-api-access-qm78r\" (UniqueName: \"kubernetes.io/projected/73d7018a-d448-4e25-afc4-2a002931f75f-kube-api-access-qm78r\") pod \"73d7018a-d448-4e25-afc4-2a002931f75f\" (UID: \"73d7018a-d448-4e25-afc4-2a002931f75f\") " Oct 30 17:15:21.429246 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.429229 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73d7018a-d448-4e25-afc4-2a002931f75f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d7018a-d448-4e25-afc4-2a002931f75f-maas-api-service-ca\") pod \"73d7018a-d448-4e25-afc4-2a002931f75f\" (UID: \"73d7018a-d448-4e25-afc4-2a002931f75f\") " Oct 30 17:15:21.429996 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.429607 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d7018a-d448-4e25-afc4-2a002931f75f-maas-api-service-ca" pod "73d7018a-d448-4e25-afc4-2a002931f75f" (UID: "73d7018a-d448-4e25-afc4-2a002931f75f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:21.429996 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.429871 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d7018a-d448-4e25-afc4-2a002931f75f-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.432751 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.432724 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d7018a-d448-4e25-afc4-2a002931f75f-kube-api-access-qm78r" pod "73d7018a-d448-4e25-afc4-2a002931f75f" (UID: "73d7018a-d448-4e25-afc4-2a002931f75f"). InnerVolumeSpecName "kube-api-access-qm78r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:21.531427 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.531334 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fdeaeb1-059f-49ac-864d-0b5a4a504a50-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdeaeb1-059f-49ac-864d-0b5a4a504a50-maas-api-service-ca\") pod \"3fdeaeb1-059f-49ac-864d-0b5a4a504a50\" (UID: \"3fdeaeb1-059f-49ac-864d-0b5a4a504a50\") " Oct 30 17:15:21.531803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.531756 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fdeaeb1-059f-49ac-864d-0b5a4a504a50-maas-api-service-ca" pod "3fdeaeb1-059f-49ac-864d-0b5a4a504a50" (UID: "3fdeaeb1-059f-49ac-864d-0b5a4a504a50"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:21.531960 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.531855 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fdeaeb1-059f-49ac-864d-0b5a4a504a50-kube-api-access-x5n2r\" (UniqueName: \"kubernetes.io/projected/3fdeaeb1-059f-49ac-864d-0b5a4a504a50-kube-api-access-x5n2r\") pod \"3fdeaeb1-059f-49ac-864d-0b5a4a504a50\" (UID: \"3fdeaeb1-059f-49ac-864d-0b5a4a504a50\") " Oct 30 17:15:21.532364 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.532348 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdeaeb1-059f-49ac-864d-0b5a4a504a50-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.532474 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.532366 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm78r\" (UniqueName: \"kubernetes.io/projected/73d7018a-d448-4e25-afc4-2a002931f75f-kube-api-access-qm78r\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.534843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.534815 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fdeaeb1-059f-49ac-864d-0b5a4a504a50-kube-api-access-x5n2r" pod "3fdeaeb1-059f-49ac-864d-0b5a4a504a50" (UID: "3fdeaeb1-059f-49ac-864d-0b5a4a504a50"). InnerVolumeSpecName "kube-api-access-x5n2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:21.555830 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.555790 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" podUID="5af8f435-0196-487a-b4ce-017b4adc6f50" containerName="payload-processing" containerID="cri-o://035d3b02e7f92fb3ec0e67c3fd016ad8d112db23e218c95bdae212a47ec49576" gracePeriod=25 Oct 30 17:15:21.582669 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.582642 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94"] Oct 30 17:15:21.583630 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.583611 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" Oct 30 17:15:21.601344 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.601312 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" podUID="122dfb4c-4008-4ee9-881f-d0e8a4525e05" containerName="payload-pre-processing" containerID="cri-o://d706f7da72290bcdb59666c58cfbd2b11a15091f76b61f22c8338f1c24489981" gracePeriod=25 Oct 30 17:15:21.634360 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.634321 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5n2r\" (UniqueName: \"kubernetes.io/projected/3fdeaeb1-059f-49ac-864d-0b5a4a504a50-kube-api-access-x5n2r\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.711211 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.711180 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:21.735809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.735782 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c9ed38f-5c3a-4ea8-8969-0d2b506cc260-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9ed38f-5c3a-4ea8-8969-0d2b506cc260-maas-api-service-ca\") pod \"4c9ed38f-5c3a-4ea8-8969-0d2b506cc260\" (UID: \"4c9ed38f-5c3a-4ea8-8969-0d2b506cc260\") " Oct 30 17:15:21.735966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.735869 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9ed38f-5c3a-4ea8-8969-0d2b506cc260-kube-api-access-t8hcp\" (UniqueName: \"kubernetes.io/projected/4c9ed38f-5c3a-4ea8-8969-0d2b506cc260-kube-api-access-t8hcp\") pod \"4c9ed38f-5c3a-4ea8-8969-0d2b506cc260\" (UID: \"4c9ed38f-5c3a-4ea8-8969-0d2b506cc260\") " Oct 30 17:15:21.736138 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.736115 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9ed38f-5c3a-4ea8-8969-0d2b506cc260-maas-api-service-ca" pod "4c9ed38f-5c3a-4ea8-8969-0d2b506cc260" (UID: "4c9ed38f-5c3a-4ea8-8969-0d2b506cc260"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:21.736544 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.736523 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9ed38f-5c3a-4ea8-8969-0d2b506cc260-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.738247 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.738223 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9ed38f-5c3a-4ea8-8969-0d2b506cc260-kube-api-access-t8hcp" pod "4c9ed38f-5c3a-4ea8-8969-0d2b506cc260" (UID: "4c9ed38f-5c3a-4ea8-8969-0d2b506cc260"). InnerVolumeSpecName "kube-api-access-t8hcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:21.838563 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.838497 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8hcp\" (UniqueName: \"kubernetes.io/projected/4c9ed38f-5c3a-4ea8-8969-0d2b506cc260-kube-api-access-t8hcp\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.931083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.931044 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" podUID="122dfb4c-4008-4ee9-881f-d0e8a4525e05" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:15:21.931481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:21.931450 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" podUID="5af8f435-0196-487a-b4ce-017b4adc6f50" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:15:22.038427 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.038388 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" event={"ID":"3fdeaeb1-059f-49ac-864d-0b5a4a504a50","Type":"ContainerDied","Data":"838b2437bd1d8f25eaeafd44a1517124298697ef2dade9130555cc358c905679"} Oct 30 17:15:22.038985 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.038462 6202 scope.go:122] "RemoveContainer" containerID="9d0925b9c4fb6d850a207e6bedbd91f514e7d7176c0e13a4d80ecfbd247d236d" Oct 30 17:15:22.038985 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.038580 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk" Oct 30 17:15:22.109130 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.108438 6202 generic.go:356] "Generic (PLEG): container finished" podID="c4193469-f0ea-48f5-af10-3479e0980736" containerID="2a6b36f734ba02e5e3a36694efc08b43d00bec88429338eae0904ce86940df1a" exitCode=60 Oct 30 17:15:22.109130 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.108787 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" event={"ID":"c4193469-f0ea-48f5-af10-3479e0980736","Type":"ContainerDied","Data":"2a6b36f734ba02e5e3a36694efc08b43d00bec88429338eae0904ce86940df1a"} Oct 30 17:15:22.109130 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.108851 6202 scope.go:122] "RemoveContainer" containerID="0f5cb560a320851bd3d2879c92836cb73e18696874c7005e55bed908fceaac23" Oct 30 17:15:22.127193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.125603 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" event={"ID":"4c9ed38f-5c3a-4ea8-8969-0d2b506cc260","Type":"ContainerDied","Data":"2ca3b0fce1c834086e2f63abbdbbb8751dd2db75df616f5ebc68bd22d2979e7f"} Oct 30 17:15:22.127193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.125699 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57" Oct 30 17:15:22.146346 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.146321 6202 scope.go:122] "RemoveContainer" containerID="1c6251effa5bdb02a8f90a794f5939087a2e5d9f4d5bf6037d4d4750983ec40f" Oct 30 17:15:22.149300 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.149264 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" event={"ID":"73d7018a-d448-4e25-afc4-2a002931f75f","Type":"ContainerDied","Data":"6c157a29718e2625e20156d43b340f34a8447a0fd61d1ada9b94a08ef35bc7e5"} Oct 30 17:15:22.149615 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.149582 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d" Oct 30 17:15:22.155471 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.155437 6202 generic.go:356] "Generic (PLEG): container finished" podID="6d43f09c-faf8-49d7-8c86-0cfd033cff9d" containerID="ace4c0667da616ff9ae2e597cad10f3c3bc37c6f062ebb8796fc8b930bff50db" exitCode=0 Oct 30 17:15:22.155587 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.155550 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" event={"ID":"6d43f09c-faf8-49d7-8c86-0cfd033cff9d","Type":"ContainerDied","Data":"ace4c0667da616ff9ae2e597cad10f3c3bc37c6f062ebb8796fc8b930bff50db"} Oct 30 17:15:22.160156 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.160120 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:15:22.161020 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.160999 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk"] Oct 30 17:15:22.166315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.166285 6202 scope.go:122] "RemoveContainer" containerID="fa76854b89f68db714be94c8be66432bde17359a1b74f4e227acb5dce5562517" Oct 30 17:15:22.167947 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.167841 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dc74d5-29889675-zqttk"] Oct 30 17:15:22.184743 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.183595 6202 scope.go:122] "RemoveContainer" containerID="5e54e4c068085e8c968eacc72829d9c0169a8d4335b73138062df9664f8148ae" Oct 30 17:15:22.257439 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.257405 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57"] Oct 30 17:15:22.262296 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.262267 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b4214a-29889675-nqz57"] Oct 30 17:15:22.262530 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.262505 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-workload-certs\") pod \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " Oct 30 17:15:22.262620 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.262604 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-workload-socket\") pod \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " Oct 30 17:15:22.262695 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.262641 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-podinfo\") pod \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " Oct 30 17:15:22.262744 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.262691 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-data\") pod \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " Oct 30 17:15:22.262812 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.262743 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-envoy\") pod \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " Oct 30 17:15:22.262877 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.262820 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-kube-api-access-ww9rz\" (UniqueName: \"kubernetes.io/projected/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-kube-api-access-ww9rz\") pod \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " Oct 30 17:15:22.262936 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.262869 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-workload-socket" pod "6d43f09c-faf8-49d7-8c86-0cfd033cff9d" (UID: "6d43f09c-faf8-49d7-8c86-0cfd033cff9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:22.262936 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.262892 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-token\" (UniqueName: \"kubernetes.io/projected/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-token\") pod \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " Oct 30 17:15:22.263043 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.262946 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-credential-socket\") pod \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " Oct 30 17:15:22.263043 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.263022 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istiod-ca-cert\") pod \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\" (UID: \"6d43f09c-faf8-49d7-8c86-0cfd033cff9d\") " Oct 30 17:15:22.263240 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.263217 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-workload-certs" pod "6d43f09c-faf8-49d7-8c86-0cfd033cff9d" (UID: "6d43f09c-faf8-49d7-8c86-0cfd033cff9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:22.263307 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.263252 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-data" pod "6d43f09c-faf8-49d7-8c86-0cfd033cff9d" (UID: "6d43f09c-faf8-49d7-8c86-0cfd033cff9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:22.263444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.263426 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-credential-socket" pod "6d43f09c-faf8-49d7-8c86-0cfd033cff9d" (UID: "6d43f09c-faf8-49d7-8c86-0cfd033cff9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:22.263638 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.263612 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istiod-ca-cert" pod "6d43f09c-faf8-49d7-8c86-0cfd033cff9d" (UID: "6d43f09c-faf8-49d7-8c86-0cfd033cff9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:22.264066 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.264046 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.264180 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.264070 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.264180 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.264085 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.264180 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.264099 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.264180 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.264111 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.266662 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.266639 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-token" pod "6d43f09c-faf8-49d7-8c86-0cfd033cff9d" (UID: "6d43f09c-faf8-49d7-8c86-0cfd033cff9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:22.267648 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.267626 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-envoy" pod "6d43f09c-faf8-49d7-8c86-0cfd033cff9d" (UID: "6d43f09c-faf8-49d7-8c86-0cfd033cff9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:22.267753 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.267736 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-kube-api-access-ww9rz" pod "6d43f09c-faf8-49d7-8c86-0cfd033cff9d" (UID: "6d43f09c-faf8-49d7-8c86-0cfd033cff9d"). InnerVolumeSpecName "kube-api-access-ww9rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:22.268644 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.268622 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-podinfo" pod "6d43f09c-faf8-49d7-8c86-0cfd033cff9d" (UID: "6d43f09c-faf8-49d7-8c86-0cfd033cff9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:15:22.290342 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.288273 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" Oct 30 17:15:22.302816 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.297667 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d"] Oct 30 17:15:22.302816 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.301225 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889675-2mh9d"] Oct 30 17:15:22.309511 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.309482 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" Oct 30 17:15:22.365511 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.365432 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.365511 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.365456 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.365511 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.365465 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww9rz\" (UniqueName: \"kubernetes.io/projected/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-kube-api-access-ww9rz\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.365511 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.365476 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d43f09c-faf8-49d7-8c86-0cfd033cff9d-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.416899 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.416853 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3784ac5c-2448-4e4a-b17d-80717638a281" path="/var/lib/kubelet/pods/3784ac5c-2448-4e4a-b17d-80717638a281/volumes" Oct 30 17:15:22.417613 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.417588 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa542e0-72f9-4e5b-ab62-25fab055df11" path="/var/lib/kubelet/pods/3fa542e0-72f9-4e5b-ab62-25fab055df11/volumes" Oct 30 17:15:22.418180 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.418157 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fdeaeb1-059f-49ac-864d-0b5a4a504a50" path="/var/lib/kubelet/pods/3fdeaeb1-059f-49ac-864d-0b5a4a504a50/volumes" Oct 30 17:15:22.419321 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.419281 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c9ed38f-5c3a-4ea8-8969-0d2b506cc260" path="/var/lib/kubelet/pods/4c9ed38f-5c3a-4ea8-8969-0d2b506cc260/volumes" Oct 30 17:15:22.419928 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.419910 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d7018a-d448-4e25-afc4-2a002931f75f" path="/var/lib/kubelet/pods/73d7018a-d448-4e25-afc4-2a002931f75f/volumes" Oct 30 17:15:22.420445 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.420426 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e9f54b-cee4-43c3-96b5-e75eaf17adad" path="/var/lib/kubelet/pods/e9e9f54b-cee4-43c3-96b5-e75eaf17adad/volumes" Oct 30 17:15:22.421397 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.421378 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf" path="/var/lib/kubelet/pods/fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf/volumes" Oct 30 17:15:22.466323 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.466289 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a22ef35-5fa6-4126-9677-94883fe7fe72-kube-api-access-p6r72\" (UniqueName: \"kubernetes.io/projected/9a22ef35-5fa6-4126-9677-94883fe7fe72-kube-api-access-p6r72\") pod \"9a22ef35-5fa6-4126-9677-94883fe7fe72\" (UID: \"9a22ef35-5fa6-4126-9677-94883fe7fe72\") " Oct 30 17:15:22.466449 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.466399 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4193469-f0ea-48f5-af10-3479e0980736-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4193469-f0ea-48f5-af10-3479e0980736-maas-api-service-ca\") pod \"c4193469-f0ea-48f5-af10-3479e0980736\" (UID: \"c4193469-f0ea-48f5-af10-3479e0980736\") " Oct 30 17:15:22.466449 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.466437 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a22ef35-5fa6-4126-9677-94883fe7fe72-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a22ef35-5fa6-4126-9677-94883fe7fe72-maas-api-service-ca\") pod \"9a22ef35-5fa6-4126-9677-94883fe7fe72\" (UID: \"9a22ef35-5fa6-4126-9677-94883fe7fe72\") " Oct 30 17:15:22.466597 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.466578 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4193469-f0ea-48f5-af10-3479e0980736-kube-api-access-6z6r8\" (UniqueName: \"kubernetes.io/projected/c4193469-f0ea-48f5-af10-3479e0980736-kube-api-access-6z6r8\") pod \"c4193469-f0ea-48f5-af10-3479e0980736\" (UID: \"c4193469-f0ea-48f5-af10-3479e0980736\") " Oct 30 17:15:22.466881 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.466852 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4193469-f0ea-48f5-af10-3479e0980736-maas-api-service-ca" pod "c4193469-f0ea-48f5-af10-3479e0980736" (UID: "c4193469-f0ea-48f5-af10-3479e0980736"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:22.467333 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.467294 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a22ef35-5fa6-4126-9677-94883fe7fe72-maas-api-service-ca" pod "9a22ef35-5fa6-4126-9677-94883fe7fe72" (UID: "9a22ef35-5fa6-4126-9677-94883fe7fe72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:22.468222 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.468193 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4193469-f0ea-48f5-af10-3479e0980736-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.468330 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.468229 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a22ef35-5fa6-4126-9677-94883fe7fe72-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.469826 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.469367 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a22ef35-5fa6-4126-9677-94883fe7fe72-kube-api-access-p6r72" pod "9a22ef35-5fa6-4126-9677-94883fe7fe72" (UID: "9a22ef35-5fa6-4126-9677-94883fe7fe72"). InnerVolumeSpecName "kube-api-access-p6r72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:22.472757 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.472730 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4193469-f0ea-48f5-af10-3479e0980736-kube-api-access-6z6r8" pod "c4193469-f0ea-48f5-af10-3479e0980736" (UID: "c4193469-f0ea-48f5-af10-3479e0980736"). InnerVolumeSpecName "kube-api-access-6z6r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:22.569344 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.569307 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6r72\" (UniqueName: \"kubernetes.io/projected/9a22ef35-5fa6-4126-9677-94883fe7fe72-kube-api-access-p6r72\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.569344 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:22.569343 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z6r8\" (UniqueName: \"kubernetes.io/projected/c4193469-f0ea-48f5-af10-3479e0980736-kube-api-access-6z6r8\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.193239 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:23.187184 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" Oct 30 17:15:23.193239 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:23.187316 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj" event={"ID":"6d43f09c-faf8-49d7-8c86-0cfd033cff9d","Type":"ContainerDied","Data":"edeb5c862b4b27f9ccfb1191be3e5bdceb0a432c82e2a405163a486ccee525c9"} Oct 30 17:15:23.193239 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:23.187422 6202 scope.go:122] "RemoveContainer" containerID="ace4c0667da616ff9ae2e597cad10f3c3bc37c6f062ebb8796fc8b930bff50db" Oct 30 17:15:23.193239 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:23.191601 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" event={"ID":"9a22ef35-5fa6-4126-9677-94883fe7fe72","Type":"ContainerDied","Data":"73d7095bd6a5d5ed00cec4c71580c2da52c972ec594d9fc4df5707ce0dcbfd0d"} Oct 30 17:15:23.193239 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:23.192179 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7" Oct 30 17:15:23.194378 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:23.194289 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" event={"ID":"c4193469-f0ea-48f5-af10-3479e0980736","Type":"ContainerDied","Data":"5930a4ec63227c5c6daeb5de428504403448664446a1e74713a877b7949f9a02"} Oct 30 17:15:23.194435 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:23.194411 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94" Oct 30 17:15:23.218834 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:23.218811 6202 scope.go:122] "RemoveContainer" containerID="8974ffa474e8332801fc3e4463f7de54f538137227de2ec1966ca612dcfbb9b1" Oct 30 17:15:23.257203 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:23.257129 6202 scope.go:122] "RemoveContainer" containerID="2a6b36f734ba02e5e3a36694efc08b43d00bec88429338eae0904ce86940df1a" Oct 30 17:15:23.288592 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:23.288554 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj"] Oct 30 17:15:23.294813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:23.293498 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:23.301482 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:23.301457 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-55e83e-openshift-default-75dbc9bb97-scmcj"] Oct 30 17:15:23.328245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:23.328212 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94"] Oct 30 17:15:23.330059 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:23.330037 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889675-scv94"] Oct 30 17:15:23.366818 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:23.366746 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7"] Oct 30 17:15:23.370881 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:23.370851 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9e533f-29889675-j97g7"] Oct 30 17:15:23.859232 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:23.859199 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:23.974383 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:23.974348 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:24.089911 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:24.089876 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:24.414237 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:24.414201 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d43f09c-faf8-49d7-8c86-0cfd033cff9d" path="/var/lib/kubelet/pods/6d43f09c-faf8-49d7-8c86-0cfd033cff9d/volumes" Oct 30 17:15:24.414717 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:24.414703 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a22ef35-5fa6-4126-9677-94883fe7fe72" path="/var/lib/kubelet/pods/9a22ef35-5fa6-4126-9677-94883fe7fe72/volumes" Oct 30 17:15:24.415185 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:24.415169 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4193469-f0ea-48f5-af10-3479e0980736" path="/var/lib/kubelet/pods/c4193469-f0ea-48f5-af10-3479e0980736/volumes" Oct 30 17:15:24.468638 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:24.468609 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:25.874223 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:25.874184 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:27.525024 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:27.524989 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:27.696814 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:27.696752 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" podUID="2d98a198-b4de-422f-a2f4-df5e059d105d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:15:29.024095 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:29.024063 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:29.538575 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:29.538541 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:29.652415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:29.652379 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:29.690059 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:29.690021 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f"] Oct 30 17:15:29.717809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:29.712090 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3ed10a-vtr5f"] Oct 30 17:15:30.206238 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:30.206207 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:30.413453 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:30.413417 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d68156cc-8c00-43ec-a4eb-b0613b136629" path="/var/lib/kubelet/pods/d68156cc-8c00-43ec-a4eb-b0613b136629/volumes" Oct 30 17:15:30.981613 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:30.981564 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:31.135969 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.135938 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:31.352360 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.352332 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:31.610987 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.610903 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:31.840271 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.840234 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w"] Oct 30 17:15:31.840959 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.840943 6202 cpu_manager.go:414] "containerMap: removing container" podUID="bbcf3ac0-0197-4960-8672-2fd5d9053729" containerName="istio-proxy" Oct 30 17:15:31.841011 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.840960 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbcf3ac0-0197-4960-8672-2fd5d9053729" containerName="istio-proxy" Oct 30 17:15:31.841011 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.840975 6202 cpu_manager.go:414] "containerMap: removing container" podUID="fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf" containerName="cleanup" Oct 30 17:15:31.841011 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.840980 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf" containerName="cleanup" Oct 30 17:15:31.841011 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.840991 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3784ac5c-2448-4e4a-b17d-80717638a281" containerName="cleanup" Oct 30 17:15:31.841011 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.840996 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3784ac5c-2448-4e4a-b17d-80717638a281" containerName="cleanup" Oct 30 17:15:31.841011 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841002 6202 cpu_manager.go:414] "containerMap: removing container" podUID="fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf" containerName="cleanup" Oct 30 17:15:31.841011 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841007 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841015 6202 cpu_manager.go:414] "containerMap: removing container" podUID="d68156cc-8c00-43ec-a4eb-b0613b136629" containerName="revoke-keys" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841020 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d68156cc-8c00-43ec-a4eb-b0613b136629" containerName="revoke-keys" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841027 6202 cpu_manager.go:414] "containerMap: removing container" podUID="11be79af-8b98-43ef-8880-3dc9ac2c3a6f" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841031 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11be79af-8b98-43ef-8880-3dc9ac2c3a6f" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841039 6202 cpu_manager.go:414] "containerMap: removing container" podUID="67afc1c4-1c1f-4c7b-9de3-df2899b44392" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841044 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67afc1c4-1c1f-4c7b-9de3-df2899b44392" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841052 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9a22ef35-5fa6-4126-9677-94883fe7fe72" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841056 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a22ef35-5fa6-4126-9677-94883fe7fe72" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841063 6202 cpu_manager.go:414] "containerMap: removing container" podUID="6d43f09c-faf8-49d7-8c86-0cfd033cff9d" containerName="istio-proxy" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841067 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d43f09c-faf8-49d7-8c86-0cfd033cff9d" containerName="istio-proxy" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841072 6202 cpu_manager.go:414] "containerMap: removing container" podUID="7ad83a2b-ebfd-4902-9b86-bfdc8552f763" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841077 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad83a2b-ebfd-4902-9b86-bfdc8552f763" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841086 6202 cpu_manager.go:414] "containerMap: removing container" podUID="73d7018a-d448-4e25-afc4-2a002931f75f" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841091 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d7018a-d448-4e25-afc4-2a002931f75f" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841097 6202 cpu_manager.go:414] "containerMap: removing container" podUID="e9e9f54b-cee4-43c3-96b5-e75eaf17adad" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841101 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e9f54b-cee4-43c3-96b5-e75eaf17adad" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841107 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3fdeaeb1-059f-49ac-864d-0b5a4a504a50" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841111 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdeaeb1-059f-49ac-864d-0b5a4a504a50" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841119 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c4193469-f0ea-48f5-af10-3479e0980736" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841123 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4193469-f0ea-48f5-af10-3479e0980736" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841129 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3fa542e0-72f9-4e5b-ab62-25fab055df11" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841133 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa542e0-72f9-4e5b-ab62-25fab055df11" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841147 6202 cpu_manager.go:414] "containerMap: removing container" podUID="4c9ed38f-5c3a-4ea8-8969-0d2b506cc260" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841151 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9ed38f-5c3a-4ea8-8969-0d2b506cc260" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841159 6202 cpu_manager.go:414] "containerMap: removing container" podUID="e9e9f54b-cee4-43c3-96b5-e75eaf17adad" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841163 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e9f54b-cee4-43c3-96b5-e75eaf17adad" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841169 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3fdeaeb1-059f-49ac-864d-0b5a4a504a50" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841173 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdeaeb1-059f-49ac-864d-0b5a4a504a50" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841179 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c4193469-f0ea-48f5-af10-3479e0980736" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841183 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4193469-f0ea-48f5-af10-3479e0980736" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841189 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3fa542e0-72f9-4e5b-ab62-25fab055df11" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841192 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa542e0-72f9-4e5b-ab62-25fab055df11" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841202 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3784ac5c-2448-4e4a-b17d-80717638a281" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841206 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3784ac5c-2448-4e4a-b17d-80717638a281" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841213 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9a22ef35-5fa6-4126-9677-94883fe7fe72" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841217 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a22ef35-5fa6-4126-9677-94883fe7fe72" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841225 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3fdeaeb1-059f-49ac-864d-0b5a4a504a50" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841229 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdeaeb1-059f-49ac-864d-0b5a4a504a50" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841237 6202 cpu_manager.go:414] "containerMap: removing container" podUID="bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" containerName="istio-proxy" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841242 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" containerName="istio-proxy" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841249 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c4193469-f0ea-48f5-af10-3479e0980736" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841253 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4193469-f0ea-48f5-af10-3479e0980736" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841261 6202 cpu_manager.go:414] "containerMap: removing container" podUID="4c9ed38f-5c3a-4ea8-8969-0d2b506cc260" containerName="cleanup" Oct 30 17:15:31.841253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841265 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9ed38f-5c3a-4ea8-8969-0d2b506cc260" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841277 6202 cpu_manager.go:414] "containerMap: removing container" podUID="bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" containerName="istio-proxy" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841282 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" containerName="istio-proxy" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841290 6202 cpu_manager.go:414] "containerMap: removing container" podUID="84ccd5fe-cc47-482f-b2cb-e315080fae2a" containerName="istio-proxy" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841294 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84ccd5fe-cc47-482f-b2cb-e315080fae2a" containerName="istio-proxy" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841307 6202 cpu_manager.go:414] "containerMap: removing container" podUID="11be79af-8b98-43ef-8880-3dc9ac2c3a6f" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841311 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11be79af-8b98-43ef-8880-3dc9ac2c3a6f" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841316 6202 cpu_manager.go:414] "containerMap: removing container" podUID="7ad83a2b-ebfd-4902-9b86-bfdc8552f763" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841321 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad83a2b-ebfd-4902-9b86-bfdc8552f763" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841329 6202 cpu_manager.go:414] "containerMap: removing container" podUID="67afc1c4-1c1f-4c7b-9de3-df2899b44392" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841333 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67afc1c4-1c1f-4c7b-9de3-df2899b44392" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841339 6202 cpu_manager.go:414] "containerMap: removing container" podUID="7ad83a2b-ebfd-4902-9b86-bfdc8552f763" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841342 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad83a2b-ebfd-4902-9b86-bfdc8552f763" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841350 6202 cpu_manager.go:414] "containerMap: removing container" podUID="73d7018a-d448-4e25-afc4-2a002931f75f" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841354 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d7018a-d448-4e25-afc4-2a002931f75f" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841360 6202 cpu_manager.go:414] "containerMap: removing container" podUID="4c9ed38f-5c3a-4ea8-8969-0d2b506cc260" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841364 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9ed38f-5c3a-4ea8-8969-0d2b506cc260" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841452 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdeaeb1-059f-49ac-864d-0b5a4a504a50" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841460 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4193469-f0ea-48f5-af10-3479e0980736" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841466 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa542e0-72f9-4e5b-ab62-25fab055df11" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841472 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3784ac5c-2448-4e4a-b17d-80717638a281" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841477 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a22ef35-5fa6-4126-9677-94883fe7fe72" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841482 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841487 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdeaeb1-059f-49ac-864d-0b5a4a504a50" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841492 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d43f09c-faf8-49d7-8c86-0cfd033cff9d" containerName="istio-proxy" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841499 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" containerName="istio-proxy" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841505 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9ed38f-5c3a-4ea8-8969-0d2b506cc260" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841510 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="67afc1c4-1c1f-4c7b-9de3-df2899b44392" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841516 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a22ef35-5fa6-4126-9677-94883fe7fe72" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841520 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="11be79af-8b98-43ef-8880-3dc9ac2c3a6f" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841525 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcdb58e9-32c9-4901-8878-b2e85c2a1ee9" containerName="istio-proxy" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841530 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d7018a-d448-4e25-afc4-2a002931f75f" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841536 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="84ccd5fe-cc47-482f-b2cb-e315080fae2a" containerName="istio-proxy" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841541 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa542e0-72f9-4e5b-ab62-25fab055df11" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841547 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="11be79af-8b98-43ef-8880-3dc9ac2c3a6f" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841552 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad83a2b-ebfd-4902-9b86-bfdc8552f763" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841557 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="67afc1c4-1c1f-4c7b-9de3-df2899b44392" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841562 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbcf3ac0-0197-4960-8672-2fd5d9053729" containerName="istio-proxy" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841568 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad83a2b-ebfd-4902-9b86-bfdc8552f763" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841573 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d7018a-d448-4e25-afc4-2a002931f75f" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841578 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbcf3ac0-0197-4960-8672-2fd5d9053729" containerName="istio-proxy" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841583 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841588 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="d68156cc-8c00-43ec-a4eb-b0613b136629" containerName="revoke-keys" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841594 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="67afc1c4-1c1f-4c7b-9de3-df2899b44392" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841599 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad83a2b-ebfd-4902-9b86-bfdc8552f763" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841606 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d7018a-d448-4e25-afc4-2a002931f75f" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841610 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e9f54b-cee4-43c3-96b5-e75eaf17adad" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841615 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4193469-f0ea-48f5-af10-3479e0980736" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841621 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3784ac5c-2448-4e4a-b17d-80717638a281" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841626 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e9f54b-cee4-43c3-96b5-e75eaf17adad" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841631 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9ed38f-5c3a-4ea8-8969-0d2b506cc260" containerName="cleanup" Oct 30 17:15:31.842983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.841636 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e9f54b-cee4-43c3-96b5-e75eaf17adad" containerName="cleanup" Oct 30 17:15:31.845487 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.845471 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w" Oct 30 17:15:31.865757 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.865690 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w"] Oct 30 17:15:31.873627 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.873597 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:15:31.873627 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.873598 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:15:31.873858 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.873666 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:15:31.873858 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.873753 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:15:31.927932 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.927898 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a40eabb7-728d-44aa-bb47-5ec098f715e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w\" (UID: \"a40eabb7-728d-44aa-bb47-5ec098f715e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w" Oct 30 17:15:31.928142 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.928061 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a40eabb7-728d-44aa-bb47-5ec098f715e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w\" (UID: \"a40eabb7-728d-44aa-bb47-5ec098f715e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w" Oct 30 17:15:31.928142 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.928092 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7xn2\" (UniqueName: \"kubernetes.io/projected/a40eabb7-728d-44aa-bb47-5ec098f715e6-kube-api-access-b7xn2\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w\" (UID: \"a40eabb7-728d-44aa-bb47-5ec098f715e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w" Oct 30 17:15:31.930499 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.930470 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" podUID="122dfb4c-4008-4ee9-881f-d0e8a4525e05" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:15:31.931477 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:31.931450 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" podUID="5af8f435-0196-487a-b4ce-017b4adc6f50" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:15:32.029718 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:32.029682 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a40eabb7-728d-44aa-bb47-5ec098f715e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w\" (UID: \"a40eabb7-728d-44aa-bb47-5ec098f715e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w" Oct 30 17:15:32.029922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:32.029803 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a40eabb7-728d-44aa-bb47-5ec098f715e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w\" (UID: \"a40eabb7-728d-44aa-bb47-5ec098f715e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w" Oct 30 17:15:32.029922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:32.029823 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7xn2\" (UniqueName: \"kubernetes.io/projected/a40eabb7-728d-44aa-bb47-5ec098f715e6-kube-api-access-b7xn2\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w\" (UID: \"a40eabb7-728d-44aa-bb47-5ec098f715e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w" Oct 30 17:15:32.030535 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:32.030513 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a40eabb7-728d-44aa-bb47-5ec098f715e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w\" (UID: \"a40eabb7-728d-44aa-bb47-5ec098f715e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w" Oct 30 17:15:32.032435 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:32.032411 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a40eabb7-728d-44aa-bb47-5ec098f715e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w\" (UID: \"a40eabb7-728d-44aa-bb47-5ec098f715e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w" Oct 30 17:15:32.057003 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:32.056970 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7xn2\" (UniqueName: \"kubernetes.io/projected/a40eabb7-728d-44aa-bb47-5ec098f715e6-kube-api-access-b7xn2\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w\" (UID: \"a40eabb7-728d-44aa-bb47-5ec098f715e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w" Oct 30 17:15:32.172570 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:32.172493 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w" Oct 30 17:15:32.474694 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:15:32.474660 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40eabb7_728d_44aa_bb47_5ec098f715e6.slice/crio-1bc29e172872e83789448f5decda9f3ed09c4ca2881c23a0bf5df312fee88970 WatchSource:0}: Error finding container 1bc29e172872e83789448f5decda9f3ed09c4ca2881c23a0bf5df312fee88970: Status 404 returned error can't find the container with id 1bc29e172872e83789448f5decda9f3ed09c4ca2881c23a0bf5df312fee88970 Oct 30 17:15:32.475147 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:32.475060 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w"] Oct 30 17:15:33.327732 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:33.327693 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w" event={"ID":"a40eabb7-728d-44aa-bb47-5ec098f715e6","Type":"ContainerStarted","Data":"5a88dc218b0d16e9409f1344679215996b639e59bbd23e175f167b53e4a21a4f"} Oct 30 17:15:33.327935 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:33.327747 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w" event={"ID":"a40eabb7-728d-44aa-bb47-5ec098f715e6","Type":"ContainerStarted","Data":"1bc29e172872e83789448f5decda9f3ed09c4ca2881c23a0bf5df312fee88970"} Oct 30 17:15:33.378085 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:33.378011 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w" podStartSLOduration=2.377992245 podStartE2EDuration="2.377992245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:33.37010225 +0000 UTC m=+3247.080557023" watchObservedRunningTime="2026-10-30 17:15:33.377992245 +0000 UTC m=+3247.088446991" Oct 30 17:15:33.627716 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:33.627441 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:34.248269 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:34.248231 6202 scope.go:122] "RemoveContainer" containerID="c587cbb17d17c9cc9d611a6300d579abe2a9b4994f187f7d0c92f9620b070a0d" Oct 30 17:15:34.262027 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:34.262002 6202 scope.go:122] "RemoveContainer" containerID="280dc6bee98e4a2f25b348bd9b2597d560f29cce75f2330b6d89a08d0b15370b" Oct 30 17:15:34.271991 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:34.271964 6202 scope.go:122] "RemoveContainer" containerID="e99400e25b36a5d64a0739a06dd1be37be205d7fb28dc16bf812859af8b46231" Oct 30 17:15:34.287874 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:34.287843 6202 scope.go:122] "RemoveContainer" containerID="0c526cb1090d03e6480ec3b5b9c5f45d21c2f395c5ca863da1b877c29e7063b7" Oct 30 17:15:34.298623 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:34.298599 6202 scope.go:122] "RemoveContainer" containerID="0747f8c08fcdd36e17d4eff875c601746683d848cfe8189e0c23d0abac6920b5" Oct 30 17:15:34.308347 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:34.308326 6202 scope.go:122] "RemoveContainer" containerID="3d1a586e7c2ef804c9a6a10576aa413db9b40b9e2d6884182dbacb48fc66e236" Oct 30 17:15:34.319877 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:34.319857 6202 scope.go:122] "RemoveContainer" containerID="74e95b718948f2ba46c804eff67fffee6281c7540f591999e4fb27cb30ebde4b" Oct 30 17:15:34.342527 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:34.342497 6202 generic.go:356] "Generic (PLEG): container finished" podID="a40eabb7-728d-44aa-bb47-5ec098f715e6" containerID="5a88dc218b0d16e9409f1344679215996b639e59bbd23e175f167b53e4a21a4f" exitCode=0 Oct 30 17:15:34.342621 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:34.342589 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w" event={"ID":"a40eabb7-728d-44aa-bb47-5ec098f715e6","Type":"ContainerDied","Data":"5a88dc218b0d16e9409f1344679215996b639e59bbd23e175f167b53e4a21a4f"} Oct 30 17:15:35.377451 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:35.377265 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:35.541431 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:35.541400 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w" Oct 30 17:15:35.582684 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:35.582651 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a40eabb7-728d-44aa-bb47-5ec098f715e6-kube-api-access-b7xn2\" (UniqueName: \"kubernetes.io/projected/a40eabb7-728d-44aa-bb47-5ec098f715e6-kube-api-access-b7xn2\") pod \"a40eabb7-728d-44aa-bb47-5ec098f715e6\" (UID: \"a40eabb7-728d-44aa-bb47-5ec098f715e6\") " Oct 30 17:15:35.582875 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:35.582830 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a40eabb7-728d-44aa-bb47-5ec098f715e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a40eabb7-728d-44aa-bb47-5ec098f715e6-maas-api-xks-ca\") pod \"a40eabb7-728d-44aa-bb47-5ec098f715e6\" (UID: \"a40eabb7-728d-44aa-bb47-5ec098f715e6\") " Oct 30 17:15:35.582946 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:35.582896 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a40eabb7-728d-44aa-bb47-5ec098f715e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a40eabb7-728d-44aa-bb47-5ec098f715e6-maas-api-openshift-ca\") pod \"a40eabb7-728d-44aa-bb47-5ec098f715e6\" (UID: \"a40eabb7-728d-44aa-bb47-5ec098f715e6\") " Oct 30 17:15:35.583294 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:35.583265 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a40eabb7-728d-44aa-bb47-5ec098f715e6-maas-api-openshift-ca" pod "a40eabb7-728d-44aa-bb47-5ec098f715e6" (UID: "a40eabb7-728d-44aa-bb47-5ec098f715e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:35.583584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:35.583431 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a40eabb7-728d-44aa-bb47-5ec098f715e6-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:35.585174 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:35.585117 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a40eabb7-728d-44aa-bb47-5ec098f715e6-kube-api-access-b7xn2" pod "a40eabb7-728d-44aa-bb47-5ec098f715e6" (UID: "a40eabb7-728d-44aa-bb47-5ec098f715e6"). InnerVolumeSpecName "kube-api-access-b7xn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:35.585418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:35.585395 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a40eabb7-728d-44aa-bb47-5ec098f715e6-maas-api-xks-ca" pod "a40eabb7-728d-44aa-bb47-5ec098f715e6" (UID: "a40eabb7-728d-44aa-bb47-5ec098f715e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:35.684571 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:35.684487 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a40eabb7-728d-44aa-bb47-5ec098f715e6-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:35.684571 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:35.684517 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7xn2\" (UniqueName: \"kubernetes.io/projected/a40eabb7-728d-44aa-bb47-5ec098f715e6-kube-api-access-b7xn2\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:36.364814 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:36.364783 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w" Oct 30 17:15:36.365010 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:36.364812 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w" event={"ID":"a40eabb7-728d-44aa-bb47-5ec098f715e6","Type":"ContainerDied","Data":"1bc29e172872e83789448f5decda9f3ed09c4ca2881c23a0bf5df312fee88970"} Oct 30 17:15:36.365010 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:36.364871 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bc29e172872e83789448f5decda9f3ed09c4ca2881c23a0bf5df312fee88970" Oct 30 17:15:36.997358 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:36.997318 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:37.112845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.112623 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:37.366365 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.366331 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:37.696915 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.696823 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" podUID="2d98a198-b4de-422f-a2f4-df5e059d105d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:15:37.812274 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.812239 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:37.901956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.901916 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb"] Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903408 6202 cpu_manager.go:414] "containerMap: removing container" podUID="73d7018a-d448-4e25-afc4-2a002931f75f" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903438 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d7018a-d448-4e25-afc4-2a002931f75f" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903462 6202 cpu_manager.go:414] "containerMap: removing container" podUID="bbcf3ac0-0197-4960-8672-2fd5d9053729" containerName="istio-proxy" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903470 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbcf3ac0-0197-4960-8672-2fd5d9053729" containerName="istio-proxy" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903485 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3784ac5c-2448-4e4a-b17d-80717638a281" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903493 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3784ac5c-2448-4e4a-b17d-80717638a281" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903506 6202 cpu_manager.go:414] "containerMap: removing container" podUID="fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903513 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903530 6202 cpu_manager.go:414] "containerMap: removing container" podUID="11be79af-8b98-43ef-8880-3dc9ac2c3a6f" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903537 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11be79af-8b98-43ef-8880-3dc9ac2c3a6f" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903550 6202 cpu_manager.go:414] "containerMap: removing container" podUID="67afc1c4-1c1f-4c7b-9de3-df2899b44392" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903558 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67afc1c4-1c1f-4c7b-9de3-df2899b44392" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903572 6202 cpu_manager.go:414] "containerMap: removing container" podUID="6d43f09c-faf8-49d7-8c86-0cfd033cff9d" containerName="istio-proxy" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903580 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d43f09c-faf8-49d7-8c86-0cfd033cff9d" containerName="istio-proxy" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903596 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a40eabb7-728d-44aa-bb47-5ec098f715e6" containerName="revoke-keys" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903604 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a40eabb7-728d-44aa-bb47-5ec098f715e6" containerName="revoke-keys" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903633 6202 cpu_manager.go:414] "containerMap: removing container" podUID="e9e9f54b-cee4-43c3-96b5-e75eaf17adad" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903642 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e9f54b-cee4-43c3-96b5-e75eaf17adad" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903654 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3fa542e0-72f9-4e5b-ab62-25fab055df11" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903662 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa542e0-72f9-4e5b-ab62-25fab055df11" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903679 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9a22ef35-5fa6-4126-9677-94883fe7fe72" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903687 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a22ef35-5fa6-4126-9677-94883fe7fe72" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903905 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa1b0d59-0fa9-4637-b66b-2b3512e7ceaf" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903923 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a40eabb7-728d-44aa-bb47-5ec098f715e6" containerName="revoke-keys" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903934 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3784ac5c-2448-4e4a-b17d-80717638a281" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903984 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4193469-f0ea-48f5-af10-3479e0980736" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.903995 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa542e0-72f9-4e5b-ab62-25fab055df11" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.904007 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdeaeb1-059f-49ac-864d-0b5a4a504a50" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.904017 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d43f09c-faf8-49d7-8c86-0cfd033cff9d" containerName="istio-proxy" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.904028 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9ed38f-5c3a-4ea8-8969-0d2b506cc260" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.904041 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a22ef35-5fa6-4126-9677-94883fe7fe72" containerName="cleanup" Oct 30 17:15:37.904375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.904055 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="11be79af-8b98-43ef-8880-3dc9ac2c3a6f" containerName="cleanup" Oct 30 17:15:37.908684 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.908663 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb" Oct 30 17:15:37.924007 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.923975 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb"] Oct 30 17:15:37.933072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.933048 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:15:37.933072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.933048 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:15:37.947895 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.947823 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk"] Oct 30 17:15:37.953814 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.953788 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk" Oct 30 17:15:37.973979 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:37.973252 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk"] Oct 30 17:15:38.006395 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.006362 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:15:38.006786 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.006526 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:15:38.023650 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.023619 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc285144-2c80-4d79-8269-3994f8858ab3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk\" (UID: \"cc285144-2c80-4d79-8269-3994f8858ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk" Oct 30 17:15:38.023832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.023662 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc285144-2c80-4d79-8269-3994f8858ab3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk\" (UID: \"cc285144-2c80-4d79-8269-3994f8858ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk" Oct 30 17:15:38.023892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.023833 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv4xq\" (UniqueName: \"kubernetes.io/projected/cc285144-2c80-4d79-8269-3994f8858ab3-kube-api-access-gv4xq\") pod \"maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk\" (UID: \"cc285144-2c80-4d79-8269-3994f8858ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk" Oct 30 17:15:38.023892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.023862 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfeb39d1-2907-41de-afaa-8c3b0690a828-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb\" (UID: \"cfeb39d1-2907-41de-afaa-8c3b0690a828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb" Oct 30 17:15:38.023962 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.023903 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhfjg\" (UniqueName: \"kubernetes.io/projected/cfeb39d1-2907-41de-afaa-8c3b0690a828-kube-api-access-qhfjg\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb\" (UID: \"cfeb39d1-2907-41de-afaa-8c3b0690a828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb" Oct 30 17:15:38.024032 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.024008 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfeb39d1-2907-41de-afaa-8c3b0690a828-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb\" (UID: \"cfeb39d1-2907-41de-afaa-8c3b0690a828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb" Oct 30 17:15:38.125924 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.125888 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc285144-2c80-4d79-8269-3994f8858ab3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk\" (UID: \"cc285144-2c80-4d79-8269-3994f8858ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk" Oct 30 17:15:38.126130 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.125935 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc285144-2c80-4d79-8269-3994f8858ab3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk\" (UID: \"cc285144-2c80-4d79-8269-3994f8858ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk" Oct 30 17:15:38.126130 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.125994 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv4xq\" (UniqueName: \"kubernetes.io/projected/cc285144-2c80-4d79-8269-3994f8858ab3-kube-api-access-gv4xq\") pod \"maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk\" (UID: \"cc285144-2c80-4d79-8269-3994f8858ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk" Oct 30 17:15:38.126130 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.126016 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfeb39d1-2907-41de-afaa-8c3b0690a828-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb\" (UID: \"cfeb39d1-2907-41de-afaa-8c3b0690a828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb" Oct 30 17:15:38.126130 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.126045 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhfjg\" (UniqueName: \"kubernetes.io/projected/cfeb39d1-2907-41de-afaa-8c3b0690a828-kube-api-access-qhfjg\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb\" (UID: \"cfeb39d1-2907-41de-afaa-8c3b0690a828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb" Oct 30 17:15:38.126130 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.126085 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfeb39d1-2907-41de-afaa-8c3b0690a828-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb\" (UID: \"cfeb39d1-2907-41de-afaa-8c3b0690a828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb" Oct 30 17:15:38.126816 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.126786 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfeb39d1-2907-41de-afaa-8c3b0690a828-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb\" (UID: \"cfeb39d1-2907-41de-afaa-8c3b0690a828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb" Oct 30 17:15:38.126936 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.126831 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc285144-2c80-4d79-8269-3994f8858ab3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk\" (UID: \"cc285144-2c80-4d79-8269-3994f8858ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk" Oct 30 17:15:38.128465 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.128440 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc285144-2c80-4d79-8269-3994f8858ab3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk\" (UID: \"cc285144-2c80-4d79-8269-3994f8858ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk" Oct 30 17:15:38.128580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.128564 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfeb39d1-2907-41de-afaa-8c3b0690a828-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb\" (UID: \"cfeb39d1-2907-41de-afaa-8c3b0690a828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb" Oct 30 17:15:38.150376 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.150353 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv4xq\" (UniqueName: \"kubernetes.io/projected/cc285144-2c80-4d79-8269-3994f8858ab3-kube-api-access-gv4xq\") pod \"maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk\" (UID: \"cc285144-2c80-4d79-8269-3994f8858ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk" Oct 30 17:15:38.150496 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.150446 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhfjg\" (UniqueName: \"kubernetes.io/projected/cfeb39d1-2907-41de-afaa-8c3b0690a828-kube-api-access-qhfjg\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb\" (UID: \"cfeb39d1-2907-41de-afaa-8c3b0690a828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb" Oct 30 17:15:38.232752 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.232663 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb" Oct 30 17:15:38.286426 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.286386 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk" Oct 30 17:15:38.537360 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:15:38.537331 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfeb39d1_2907_41de_afaa_8c3b0690a828.slice/crio-41e203aef5565f29fc224931b7abbebbbf5699eacbbb4fa7556a8898c22c90ee WatchSource:0}: Error finding container 41e203aef5565f29fc224931b7abbebbbf5699eacbbb4fa7556a8898c22c90ee: Status 404 returned error can't find the container with id 41e203aef5565f29fc224931b7abbebbbf5699eacbbb4fa7556a8898c22c90ee Oct 30 17:15:38.538457 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.538426 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb"] Oct 30 17:15:38.606726 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.606692 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk"] Oct 30 17:15:38.617741 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:15:38.617706 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc285144_2c80_4d79_8269_3994f8858ab3.slice/crio-ef3f5af92e5949c9f13b8fab6f19b584e2b50ae78e41ad42f6d09fbe4f73fee1 WatchSource:0}: Error finding container ef3f5af92e5949c9f13b8fab6f19b584e2b50ae78e41ad42f6d09fbe4f73fee1: Status 404 returned error can't find the container with id ef3f5af92e5949c9f13b8fab6f19b584e2b50ae78e41ad42f6d09fbe4f73fee1 Oct 30 17:15:38.850052 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.850022 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" Oct 30 17:15:38.936998 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.936963 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d98a198-b4de-422f-a2f4-df5e059d105d-kube-api-access-lw4gv\" (UniqueName: \"kubernetes.io/projected/2d98a198-b4de-422f-a2f4-df5e059d105d-kube-api-access-lw4gv\") pod \"2d98a198-b4de-422f-a2f4-df5e059d105d\" (UID: \"2d98a198-b4de-422f-a2f4-df5e059d105d\") " Oct 30 17:15:38.937207 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.937085 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d98a198-b4de-422f-a2f4-df5e059d105d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d98a198-b4de-422f-a2f4-df5e059d105d-plugins-config-volume\") pod \"2d98a198-b4de-422f-a2f4-df5e059d105d\" (UID: \"2d98a198-b4de-422f-a2f4-df5e059d105d\") " Oct 30 17:15:38.937558 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.937518 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d98a198-b4de-422f-a2f4-df5e059d105d-plugins-config-volume" pod "2d98a198-b4de-422f-a2f4-df5e059d105d" (UID: "2d98a198-b4de-422f-a2f4-df5e059d105d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:38.940306 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:38.939971 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d98a198-b4de-422f-a2f4-df5e059d105d-kube-api-access-lw4gv" pod "2d98a198-b4de-422f-a2f4-df5e059d105d" (UID: "2d98a198-b4de-422f-a2f4-df5e059d105d"). InnerVolumeSpecName "kube-api-access-lw4gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:39.039313 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:39.039268 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw4gv\" (UniqueName: \"kubernetes.io/projected/2d98a198-b4de-422f-a2f4-df5e059d105d-kube-api-access-lw4gv\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:39.039313 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:39.039300 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d98a198-b4de-422f-a2f4-df5e059d105d-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:39.396488 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:39.396394 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk" event={"ID":"cc285144-2c80-4d79-8269-3994f8858ab3","Type":"ContainerStarted","Data":"3d4e638b6fb26d2c811f8437d9a1ca607c26982a5f5d3eee08b05b24dfdaff01"} Oct 30 17:15:39.396488 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:39.396448 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk" event={"ID":"cc285144-2c80-4d79-8269-3994f8858ab3","Type":"ContainerStarted","Data":"ef3f5af92e5949c9f13b8fab6f19b584e2b50ae78e41ad42f6d09fbe4f73fee1"} Oct 30 17:15:39.397913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:39.397887 6202 generic.go:356] "Generic (PLEG): container finished" podID="2d98a198-b4de-422f-a2f4-df5e059d105d" containerID="e1c17a9be77260afe59bef057caab8c8060c9f1e911ee1b0533718c9f7559cfd" exitCode=0 Oct 30 17:15:39.398024 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:39.397929 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" Oct 30 17:15:39.398024 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:39.397975 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" event={"ID":"2d98a198-b4de-422f-a2f4-df5e059d105d","Type":"ContainerDied","Data":"e1c17a9be77260afe59bef057caab8c8060c9f1e911ee1b0533718c9f7559cfd"} Oct 30 17:15:39.398024 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:39.398017 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7" event={"ID":"2d98a198-b4de-422f-a2f4-df5e059d105d","Type":"ContainerDied","Data":"294643be0c731aff857adc4731a8b609013d71547933207efff32a1259d57af5"} Oct 30 17:15:39.398191 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:39.398037 6202 scope.go:122] "RemoveContainer" containerID="e1c17a9be77260afe59bef057caab8c8060c9f1e911ee1b0533718c9f7559cfd" Oct 30 17:15:39.399839 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:39.399816 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb" event={"ID":"cfeb39d1-2907-41de-afaa-8c3b0690a828","Type":"ContainerStarted","Data":"7e0e59b9fe0042ef53e0c448cc7e3118281be6dfa9ddec9f9ca7e87b4efccee1"} Oct 30 17:15:39.399960 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:39.399866 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb" event={"ID":"cfeb39d1-2907-41de-afaa-8c3b0690a828","Type":"ContainerStarted","Data":"41e203aef5565f29fc224931b7abbebbbf5699eacbbb4fa7556a8898c22c90ee"} Oct 30 17:15:39.413684 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:39.413663 6202 scope.go:122] "RemoveContainer" containerID="e1c17a9be77260afe59bef057caab8c8060c9f1e911ee1b0533718c9f7559cfd" Oct 30 17:15:39.414001 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:39.413981 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c17a9be77260afe59bef057caab8c8060c9f1e911ee1b0533718c9f7559cfd\": container with ID starting with e1c17a9be77260afe59bef057caab8c8060c9f1e911ee1b0533718c9f7559cfd not found: ID does not exist" containerID="e1c17a9be77260afe59bef057caab8c8060c9f1e911ee1b0533718c9f7559cfd" Oct 30 17:15:39.414069 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:39.414008 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c17a9be77260afe59bef057caab8c8060c9f1e911ee1b0533718c9f7559cfd"} err="failed to get container status \"e1c17a9be77260afe59bef057caab8c8060c9f1e911ee1b0533718c9f7559cfd\": rpc error: code = NotFound desc = could not find container \"e1c17a9be77260afe59bef057caab8c8060c9f1e911ee1b0533718c9f7559cfd\": container with ID starting with e1c17a9be77260afe59bef057caab8c8060c9f1e911ee1b0533718c9f7559cfd not found: ID does not exist" Oct 30 17:15:39.441911 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:39.441842 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk" podStartSLOduration=2.44182254 podStartE2EDuration="2.44182254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:39.438907856 +0000 UTC m=+3253.149362613" watchObservedRunningTime="2026-10-30 17:15:39.44182254 +0000 UTC m=+3253.152277285" Oct 30 17:15:39.480444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:39.480410 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7"] Oct 30 17:15:39.486402 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:39.486373 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3ed10a-75c5955cdd-858q7"] Oct 30 17:15:39.526404 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:39.526348 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb" podStartSLOduration=2.526332936 podStartE2EDuration="2.526332936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:39.5245345 +0000 UTC m=+3253.234989244" watchObservedRunningTime="2026-10-30 17:15:39.526332936 +0000 UTC m=+3253.236787679" Oct 30 17:15:39.820822 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:39.820790 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:40.211940 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:40.211901 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc"] Oct 30 17:15:40.212363 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:40.212138 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" podUID="35198b3d-3a56-45c7-8d36-d4de1780a938" containerName="istio-proxy" containerID="cri-o://6f1725e28ca712e879093ba350b2876bb755af261b2838fbf24b4787efd9ab02" gracePeriod=30 Oct 30 17:15:40.409575 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:40.409542 6202 generic.go:356] "Generic (PLEG): container finished" podID="cfeb39d1-2907-41de-afaa-8c3b0690a828" containerID="7e0e59b9fe0042ef53e0c448cc7e3118281be6dfa9ddec9f9ca7e87b4efccee1" exitCode=0 Oct 30 17:15:40.411501 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:40.411464 6202 generic.go:356] "Generic (PLEG): container finished" podID="cc285144-2c80-4d79-8269-3994f8858ab3" containerID="3d4e638b6fb26d2c811f8437d9a1ca607c26982a5f5d3eee08b05b24dfdaff01" exitCode=0 Oct 30 17:15:40.412705 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:40.412685 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d98a198-b4de-422f-a2f4-df5e059d105d" path="/var/lib/kubelet/pods/2d98a198-b4de-422f-a2f4-df5e059d105d/volumes" Oct 30 17:15:40.413093 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:40.413074 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb" event={"ID":"cfeb39d1-2907-41de-afaa-8c3b0690a828","Type":"ContainerDied","Data":"7e0e59b9fe0042ef53e0c448cc7e3118281be6dfa9ddec9f9ca7e87b4efccee1"} Oct 30 17:15:40.413142 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:40.413104 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk" event={"ID":"cc285144-2c80-4d79-8269-3994f8858ab3","Type":"ContainerDied","Data":"3d4e638b6fb26d2c811f8437d9a1ca607c26982a5f5d3eee08b05b24dfdaff01"} Oct 30 17:15:40.888548 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:40.888510 6202 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:15:40.888714 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:40.888575 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" podUID="35198b3d-3a56-45c7-8d36-d4de1780a938" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:15:41.741461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.741420 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb" Oct 30 17:15:41.746038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.746003 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk" Oct 30 17:15:41.884793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.884419 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" Oct 30 17:15:41.894200 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.894170 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfeb39d1-2907-41de-afaa-8c3b0690a828-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfeb39d1-2907-41de-afaa-8c3b0690a828-maas-api-openshift-ca\") pod \"cfeb39d1-2907-41de-afaa-8c3b0690a828\" (UID: \"cfeb39d1-2907-41de-afaa-8c3b0690a828\") " Oct 30 17:15:41.894527 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.894499 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfeb39d1-2907-41de-afaa-8c3b0690a828-maas-api-openshift-ca" pod "cfeb39d1-2907-41de-afaa-8c3b0690a828" (UID: "cfeb39d1-2907-41de-afaa-8c3b0690a828"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:41.894699 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.894678 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfeb39d1-2907-41de-afaa-8c3b0690a828-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfeb39d1-2907-41de-afaa-8c3b0690a828-maas-api-xks-ca\") pod \"cfeb39d1-2907-41de-afaa-8c3b0690a828\" (UID: \"cfeb39d1-2907-41de-afaa-8c3b0690a828\") " Oct 30 17:15:41.894975 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.894955 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc285144-2c80-4d79-8269-3994f8858ab3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc285144-2c80-4d79-8269-3994f8858ab3-maas-api-openshift-ca\") pod \"cc285144-2c80-4d79-8269-3994f8858ab3\" (UID: \"cc285144-2c80-4d79-8269-3994f8858ab3\") " Oct 30 17:15:41.895170 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.895153 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc285144-2c80-4d79-8269-3994f8858ab3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc285144-2c80-4d79-8269-3994f8858ab3-maas-api-xks-ca\") pod \"cc285144-2c80-4d79-8269-3994f8858ab3\" (UID: \"cc285144-2c80-4d79-8269-3994f8858ab3\") " Oct 30 17:15:41.895342 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.895324 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc285144-2c80-4d79-8269-3994f8858ab3-kube-api-access-gv4xq\" (UniqueName: \"kubernetes.io/projected/cc285144-2c80-4d79-8269-3994f8858ab3-kube-api-access-gv4xq\") pod \"cc285144-2c80-4d79-8269-3994f8858ab3\" (UID: \"cc285144-2c80-4d79-8269-3994f8858ab3\") " Oct 30 17:15:41.895508 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.895489 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfeb39d1-2907-41de-afaa-8c3b0690a828-kube-api-access-qhfjg\" (UniqueName: \"kubernetes.io/projected/cfeb39d1-2907-41de-afaa-8c3b0690a828-kube-api-access-qhfjg\") pod \"cfeb39d1-2907-41de-afaa-8c3b0690a828\" (UID: \"cfeb39d1-2907-41de-afaa-8c3b0690a828\") " Oct 30 17:15:41.896321 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.896301 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfeb39d1-2907-41de-afaa-8c3b0690a828-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:41.896627 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.896607 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc285144-2c80-4d79-8269-3994f8858ab3-maas-api-openshift-ca" pod "cc285144-2c80-4d79-8269-3994f8858ab3" (UID: "cc285144-2c80-4d79-8269-3994f8858ab3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:41.900048 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.900017 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc285144-2c80-4d79-8269-3994f8858ab3-kube-api-access-gv4xq" pod "cc285144-2c80-4d79-8269-3994f8858ab3" (UID: "cc285144-2c80-4d79-8269-3994f8858ab3"). InnerVolumeSpecName "kube-api-access-gv4xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:41.901207 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.901179 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc285144-2c80-4d79-8269-3994f8858ab3-maas-api-xks-ca" pod "cc285144-2c80-4d79-8269-3994f8858ab3" (UID: "cc285144-2c80-4d79-8269-3994f8858ab3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:41.901319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.901238 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfeb39d1-2907-41de-afaa-8c3b0690a828-maas-api-xks-ca" pod "cfeb39d1-2907-41de-afaa-8c3b0690a828" (UID: "cfeb39d1-2907-41de-afaa-8c3b0690a828"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:41.901319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.901252 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfeb39d1-2907-41de-afaa-8c3b0690a828-kube-api-access-qhfjg" pod "cfeb39d1-2907-41de-afaa-8c3b0690a828" (UID: "cfeb39d1-2907-41de-afaa-8c3b0690a828"). InnerVolumeSpecName "kube-api-access-qhfjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:41.904939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.904915 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:41.963594 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.963205 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" Oct 30 17:15:41.997473 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.997441 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5af8f435-0196-487a-b4ce-017b4adc6f50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af8f435-0196-487a-b4ce-017b4adc6f50-plugins-config-volume\") pod \"5af8f435-0196-487a-b4ce-017b4adc6f50\" (UID: \"5af8f435-0196-487a-b4ce-017b4adc6f50\") " Oct 30 17:15:41.997686 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.997499 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af8f435-0196-487a-b4ce-017b4adc6f50-kube-api-access-8pjwj\" (UniqueName: \"kubernetes.io/projected/5af8f435-0196-487a-b4ce-017b4adc6f50-kube-api-access-8pjwj\") pod \"5af8f435-0196-487a-b4ce-017b4adc6f50\" (UID: \"5af8f435-0196-487a-b4ce-017b4adc6f50\") " Oct 30 17:15:41.998042 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.997923 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv4xq\" (UniqueName: \"kubernetes.io/projected/cc285144-2c80-4d79-8269-3994f8858ab3-kube-api-access-gv4xq\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:41.998042 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.997941 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhfjg\" (UniqueName: \"kubernetes.io/projected/cfeb39d1-2907-41de-afaa-8c3b0690a828-kube-api-access-qhfjg\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:41.998042 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.997955 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfeb39d1-2907-41de-afaa-8c3b0690a828-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:41.998042 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.997967 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc285144-2c80-4d79-8269-3994f8858ab3-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:41.998042 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.997975 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc285144-2c80-4d79-8269-3994f8858ab3-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:41.998638 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:41.998584 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af8f435-0196-487a-b4ce-017b4adc6f50-plugins-config-volume" pod "5af8f435-0196-487a-b4ce-017b4adc6f50" (UID: "5af8f435-0196-487a-b4ce-017b4adc6f50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:42.000947 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.000927 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af8f435-0196-487a-b4ce-017b4adc6f50-kube-api-access-8pjwj" pod "5af8f435-0196-487a-b4ce-017b4adc6f50" (UID: "5af8f435-0196-487a-b4ce-017b4adc6f50"). InnerVolumeSpecName "kube-api-access-8pjwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:42.099216 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.099180 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122dfb4c-4008-4ee9-881f-d0e8a4525e05-kube-api-access-f4zbl\" (UniqueName: \"kubernetes.io/projected/122dfb4c-4008-4ee9-881f-d0e8a4525e05-kube-api-access-f4zbl\") pod \"122dfb4c-4008-4ee9-881f-d0e8a4525e05\" (UID: \"122dfb4c-4008-4ee9-881f-d0e8a4525e05\") " Oct 30 17:15:42.099442 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.099340 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/122dfb4c-4008-4ee9-881f-d0e8a4525e05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122dfb4c-4008-4ee9-881f-d0e8a4525e05-plugins-config-volume\") pod \"122dfb4c-4008-4ee9-881f-d0e8a4525e05\" (UID: \"122dfb4c-4008-4ee9-881f-d0e8a4525e05\") " Oct 30 17:15:42.099808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.099761 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122dfb4c-4008-4ee9-881f-d0e8a4525e05-plugins-config-volume" pod "122dfb4c-4008-4ee9-881f-d0e8a4525e05" (UID: "122dfb4c-4008-4ee9-881f-d0e8a4525e05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:42.099998 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.099982 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af8f435-0196-487a-b4ce-017b4adc6f50-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.100050 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.100001 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pjwj\" (UniqueName: \"kubernetes.io/projected/5af8f435-0196-487a-b4ce-017b4adc6f50-kube-api-access-8pjwj\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.100050 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.100011 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122dfb4c-4008-4ee9-881f-d0e8a4525e05-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.101534 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.101505 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122dfb4c-4008-4ee9-881f-d0e8a4525e05-kube-api-access-f4zbl" pod "122dfb4c-4008-4ee9-881f-d0e8a4525e05" (UID: "122dfb4c-4008-4ee9-881f-d0e8a4525e05"). InnerVolumeSpecName "kube-api-access-f4zbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:42.201719 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.201638 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4zbl\" (UniqueName: \"kubernetes.io/projected/122dfb4c-4008-4ee9-881f-d0e8a4525e05-kube-api-access-f4zbl\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.261158 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.261118 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml"] Oct 30 17:15:42.290026 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.289998 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg"] Oct 30 17:15:42.391031 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.390996 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:42.431594 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.431567 6202 generic.go:356] "Generic (PLEG): container finished" podID="122dfb4c-4008-4ee9-881f-d0e8a4525e05" containerID="d706f7da72290bcdb59666c58cfbd2b11a15091f76b61f22c8338f1c24489981" exitCode=0 Oct 30 17:15:42.431711 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.431606 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" Oct 30 17:15:42.431711 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.431643 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" event={"ID":"122dfb4c-4008-4ee9-881f-d0e8a4525e05","Type":"ContainerDied","Data":"d706f7da72290bcdb59666c58cfbd2b11a15091f76b61f22c8338f1c24489981"} Oct 30 17:15:42.431711 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.431691 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" event={"ID":"122dfb4c-4008-4ee9-881f-d0e8a4525e05","Type":"ContainerDied","Data":"6b5a24f75b8f79051d124c4d31ecf3be78cc13d344a9f34ef3695cfdc9bd3955"} Oct 30 17:15:42.431880 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.431714 6202 scope.go:122] "RemoveContainer" containerID="d706f7da72290bcdb59666c58cfbd2b11a15091f76b61f22c8338f1c24489981" Oct 30 17:15:42.433592 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.433572 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk" Oct 30 17:15:42.433592 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.433583 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk" event={"ID":"cc285144-2c80-4d79-8269-3994f8858ab3","Type":"ContainerDied","Data":"ef3f5af92e5949c9f13b8fab6f19b584e2b50ae78e41ad42f6d09fbe4f73fee1"} Oct 30 17:15:42.433911 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.433655 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef3f5af92e5949c9f13b8fab6f19b584e2b50ae78e41ad42f6d09fbe4f73fee1" Oct 30 17:15:42.435388 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.435369 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb" event={"ID":"cfeb39d1-2907-41de-afaa-8c3b0690a828","Type":"ContainerDied","Data":"41e203aef5565f29fc224931b7abbebbbf5699eacbbb4fa7556a8898c22c90ee"} Oct 30 17:15:42.435483 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.435391 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb" Oct 30 17:15:42.435483 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.435421 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41e203aef5565f29fc224931b7abbebbbf5699eacbbb4fa7556a8898c22c90ee" Oct 30 17:15:42.437133 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.437114 6202 generic.go:356] "Generic (PLEG): container finished" podID="5af8f435-0196-487a-b4ce-017b4adc6f50" containerID="035d3b02e7f92fb3ec0e67c3fd016ad8d112db23e218c95bdae212a47ec49576" exitCode=0 Oct 30 17:15:42.437219 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.437145 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" Oct 30 17:15:42.437219 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.437149 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" event={"ID":"5af8f435-0196-487a-b4ce-017b4adc6f50","Type":"ContainerDied","Data":"035d3b02e7f92fb3ec0e67c3fd016ad8d112db23e218c95bdae212a47ec49576"} Oct 30 17:15:42.437313 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.437250 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb" event={"ID":"5af8f435-0196-487a-b4ce-017b4adc6f50","Type":"ContainerDied","Data":"d471bc778172d0d7af213f66e4f5bf3248d77b2410c42a899cb4c156ca0c1d46"} Oct 30 17:15:42.444278 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.444258 6202 scope.go:122] "RemoveContainer" containerID="d706f7da72290bcdb59666c58cfbd2b11a15091f76b61f22c8338f1c24489981" Oct 30 17:15:42.444560 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:42.444541 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d706f7da72290bcdb59666c58cfbd2b11a15091f76b61f22c8338f1c24489981\": container with ID starting with d706f7da72290bcdb59666c58cfbd2b11a15091f76b61f22c8338f1c24489981 not found: ID does not exist" containerID="d706f7da72290bcdb59666c58cfbd2b11a15091f76b61f22c8338f1c24489981" Oct 30 17:15:42.444642 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.444568 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d706f7da72290bcdb59666c58cfbd2b11a15091f76b61f22c8338f1c24489981"} err="failed to get container status \"d706f7da72290bcdb59666c58cfbd2b11a15091f76b61f22c8338f1c24489981\": rpc error: code = NotFound desc = could not find container \"d706f7da72290bcdb59666c58cfbd2b11a15091f76b61f22c8338f1c24489981\": container with ID starting with d706f7da72290bcdb59666c58cfbd2b11a15091f76b61f22c8338f1c24489981 not found: ID does not exist" Oct 30 17:15:42.444642 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.444589 6202 scope.go:122] "RemoveContainer" containerID="035d3b02e7f92fb3ec0e67c3fd016ad8d112db23e218c95bdae212a47ec49576" Oct 30 17:15:42.454547 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.454526 6202 scope.go:122] "RemoveContainer" containerID="035d3b02e7f92fb3ec0e67c3fd016ad8d112db23e218c95bdae212a47ec49576" Oct 30 17:15:42.454867 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:42.454843 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035d3b02e7f92fb3ec0e67c3fd016ad8d112db23e218c95bdae212a47ec49576\": container with ID starting with 035d3b02e7f92fb3ec0e67c3fd016ad8d112db23e218c95bdae212a47ec49576 not found: ID does not exist" containerID="035d3b02e7f92fb3ec0e67c3fd016ad8d112db23e218c95bdae212a47ec49576" Oct 30 17:15:42.454949 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.454876 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035d3b02e7f92fb3ec0e67c3fd016ad8d112db23e218c95bdae212a47ec49576"} err="failed to get container status \"035d3b02e7f92fb3ec0e67c3fd016ad8d112db23e218c95bdae212a47ec49576\": rpc error: code = NotFound desc = could not find container \"035d3b02e7f92fb3ec0e67c3fd016ad8d112db23e218c95bdae212a47ec49576\": container with ID starting with 035d3b02e7f92fb3ec0e67c3fd016ad8d112db23e218c95bdae212a47ec49576 not found: ID does not exist" Oct 30 17:15:42.511022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.510993 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz"] Oct 30 17:15:42.511255 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.511197 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:42.515027 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.515001 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz"] Oct 30 17:15:42.555014 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.554981 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb"] Oct 30 17:15:42.557904 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.557876 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-dc74d5-6f556cd588-s49bb"] Oct 30 17:15:42.793956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.793863 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:42.930899 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:42.930847 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dc74d5-cfddcf9f5-5dfxz" podUID="122dfb4c-4008-4ee9-881f-d0e8a4525e05" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.138:50052\" within 1s: context deadline exceeded" Oct 30 17:15:42.931222 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:15:42.930961 6202 logging.go:55] [core] [Channel #20582 SubChannel #20583]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.138:50052", ServerName: "10.133.0.138:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.138:50052: operation was canceled" Oct 30 17:15:43.131859 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:43.131824 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:43.917913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:43.917882 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:44.070835 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:44.070794 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:15:44.411401 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:44.411371 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="122dfb4c-4008-4ee9-881f-d0e8a4525e05" path="/var/lib/kubelet/pods/122dfb4c-4008-4ee9-881f-d0e8a4525e05/volumes" Oct 30 17:15:44.411844 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:44.411828 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af8f435-0196-487a-b4ce-017b4adc6f50" path="/var/lib/kubelet/pods/5af8f435-0196-487a-b4ce-017b4adc6f50/volumes" Oct 30 17:15:45.465339 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.465296 6202 generic.go:356] "Generic (PLEG): container finished" podID="35198b3d-3a56-45c7-8d36-d4de1780a938" containerID="6f1725e28ca712e879093ba350b2876bb755af261b2838fbf24b4787efd9ab02" exitCode=0 Oct 30 17:15:45.465703 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.465360 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" event={"ID":"35198b3d-3a56-45c7-8d36-d4de1780a938","Type":"ContainerDied","Data":"6f1725e28ca712e879093ba350b2876bb755af261b2838fbf24b4787efd9ab02"} Oct 30 17:15:45.489390 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.489321 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 17:15:45.653460 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.653430 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-podinfo\") pod \"35198b3d-3a56-45c7-8d36-d4de1780a938\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " Oct 30 17:15:45.653626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.653472 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-workload-certs\") pod \"35198b3d-3a56-45c7-8d36-d4de1780a938\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " Oct 30 17:15:45.653626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.653509 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35198b3d-3a56-45c7-8d36-d4de1780a938-kube-api-access-8qvsl\" (UniqueName: \"kubernetes.io/projected/35198b3d-3a56-45c7-8d36-d4de1780a938-kube-api-access-8qvsl\") pod \"35198b3d-3a56-45c7-8d36-d4de1780a938\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " Oct 30 17:15:45.653626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.653544 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-credential-socket\") pod \"35198b3d-3a56-45c7-8d36-d4de1780a938\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " Oct 30 17:15:45.653626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.653603 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-data\") pod \"35198b3d-3a56-45c7-8d36-d4de1780a938\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " Oct 30 17:15:45.653909 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.653638 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-token\" (UniqueName: \"kubernetes.io/projected/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-token\") pod \"35198b3d-3a56-45c7-8d36-d4de1780a938\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " Oct 30 17:15:45.653909 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.653664 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35198b3d-3a56-45c7-8d36-d4de1780a938-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35198b3d-3a56-45c7-8d36-d4de1780a938-istiod-ca-cert\") pod \"35198b3d-3a56-45c7-8d36-d4de1780a938\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " Oct 30 17:15:45.653909 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.653758 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-workload-certs" pod "35198b3d-3a56-45c7-8d36-d4de1780a938" (UID: "35198b3d-3a56-45c7-8d36-d4de1780a938"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:45.653909 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.653884 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-workload-socket\") pod \"35198b3d-3a56-45c7-8d36-d4de1780a938\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " Oct 30 17:15:45.654142 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.653958 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-credential-socket" pod "35198b3d-3a56-45c7-8d36-d4de1780a938" (UID: "35198b3d-3a56-45c7-8d36-d4de1780a938"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:45.654142 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.654056 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-envoy\") pod \"35198b3d-3a56-45c7-8d36-d4de1780a938\" (UID: \"35198b3d-3a56-45c7-8d36-d4de1780a938\") " Oct 30 17:15:45.654251 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.654172 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-data" pod "35198b3d-3a56-45c7-8d36-d4de1780a938" (UID: "35198b3d-3a56-45c7-8d36-d4de1780a938"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:45.654251 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.654218 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35198b3d-3a56-45c7-8d36-d4de1780a938-istiod-ca-cert" pod "35198b3d-3a56-45c7-8d36-d4de1780a938" (UID: "35198b3d-3a56-45c7-8d36-d4de1780a938"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:45.654363 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.654307 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-workload-socket" pod "35198b3d-3a56-45c7-8d36-d4de1780a938" (UID: "35198b3d-3a56-45c7-8d36-d4de1780a938"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:45.654848 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.654823 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:45.654972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.654854 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:45.654972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.654870 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:45.654972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.654884 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35198b3d-3a56-45c7-8d36-d4de1780a938-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:45.654972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.654898 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:45.656119 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.656092 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-podinfo" pod "35198b3d-3a56-45c7-8d36-d4de1780a938" (UID: "35198b3d-3a56-45c7-8d36-d4de1780a938"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:15:45.656231 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.656178 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-token" pod "35198b3d-3a56-45c7-8d36-d4de1780a938" (UID: "35198b3d-3a56-45c7-8d36-d4de1780a938"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:45.656382 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.656368 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-envoy" pod "35198b3d-3a56-45c7-8d36-d4de1780a938" (UID: "35198b3d-3a56-45c7-8d36-d4de1780a938"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:45.656618 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.656597 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35198b3d-3a56-45c7-8d36-d4de1780a938-kube-api-access-8qvsl" pod "35198b3d-3a56-45c7-8d36-d4de1780a938" (UID: "35198b3d-3a56-45c7-8d36-d4de1780a938"). InnerVolumeSpecName "kube-api-access-8qvsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:45.756207 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.756128 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:45.756207 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.756156 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:45.756207 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.756165 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qvsl\" (UniqueName: \"kubernetes.io/projected/35198b3d-3a56-45c7-8d36-d4de1780a938-kube-api-access-8qvsl\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:45.756207 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.756175 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35198b3d-3a56-45c7-8d36-d4de1780a938-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:45.769817 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:45.769790 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:46.475002 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:46.474955 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" event={"ID":"35198b3d-3a56-45c7-8d36-d4de1780a938","Type":"ContainerDied","Data":"5cc134960a6db43d8ace16eb4e0a1851ea7d514478982cb0bbb51552830b2755"} Oct 30 17:15:46.475002 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:46.474980 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc" Oct 30 17:15:46.475482 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:46.475033 6202 scope.go:122] "RemoveContainer" containerID="6f1725e28ca712e879093ba350b2876bb755af261b2838fbf24b4787efd9ab02" Oct 30 17:15:46.548834 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:46.548804 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc"] Oct 30 17:15:46.561837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:46.561804 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-cgqjc"] Oct 30 17:15:47.262460 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:47.261665 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" podUID="2dd4faa7-90ff-4815-b7c5-69081a00ed05" containerName="payload-processing" containerID="cri-o://7802d6433d34e774bed4f1d0b9a263d7a00db6863d2314768684bd51f103469a" gracePeriod=25 Oct 30 17:15:47.291075 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:47.291025 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" podUID="7bec536b-a62b-42d7-810d-fa8090168784" containerName="payload-pre-processing" containerID="cri-o://9e9ea2e407ca284194fde9dd557d968c2c2b224c6018ac0ff8e3815fa1b680fd" gracePeriod=25 Oct 30 17:15:47.292192 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:47.292169 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:47.385482 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:47.385443 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn"] Oct 30 17:15:47.385722 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:47.385686 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn" podUID="6a0d47fd-b562-4eb1-b283-23bd21abc026" containerName="payload-processing" containerID="cri-o://2aaccceff6899cac6c28cf12aa3252ee21765ce9a8a8406a82c87e139d018ab1" gracePeriod=30 Oct 30 17:15:47.404751 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:47.404261 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d"] Oct 30 17:15:47.404955 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:47.404725 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d" podUID="1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a" containerName="payload-pre-processing" containerID="cri-o://7fa91a5f3fe2fa7631292c931a2c932dfea0dff88260059ccfbccd9e9f739769" gracePeriod=30 Oct 30 17:15:47.477998 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:47.477967 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:47.598021 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:47.597946 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:47.821682 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:47.821654 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn" Oct 30 17:15:47.845812 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:47.833242 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:47.930713 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:47.930685 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d" Oct 30 17:15:47.956785 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:47.956718 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:48.001673 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.001636 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a0d47fd-b562-4eb1-b283-23bd21abc026-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0d47fd-b562-4eb1-b283-23bd21abc026-plugins-config-volume\") pod \"6a0d47fd-b562-4eb1-b283-23bd21abc026\" (UID: \"6a0d47fd-b562-4eb1-b283-23bd21abc026\") " Oct 30 17:15:48.002520 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.002494 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0d47fd-b562-4eb1-b283-23bd21abc026-kube-api-access-cnns5\" (UniqueName: \"kubernetes.io/projected/6a0d47fd-b562-4eb1-b283-23bd21abc026-kube-api-access-cnns5\") pod \"6a0d47fd-b562-4eb1-b283-23bd21abc026\" (UID: \"6a0d47fd-b562-4eb1-b283-23bd21abc026\") " Oct 30 17:15:48.003135 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.002288 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0d47fd-b562-4eb1-b283-23bd21abc026-plugins-config-volume" pod "6a0d47fd-b562-4eb1-b283-23bd21abc026" (UID: "6a0d47fd-b562-4eb1-b283-23bd21abc026"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:48.003794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.003750 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0d47fd-b562-4eb1-b283-23bd21abc026-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:48.042842 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.042797 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0d47fd-b562-4eb1-b283-23bd21abc026-kube-api-access-cnns5" pod "6a0d47fd-b562-4eb1-b283-23bd21abc026" (UID: "6a0d47fd-b562-4eb1-b283-23bd21abc026"). InnerVolumeSpecName "kube-api-access-cnns5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:48.064193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.064159 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:48.104616 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.104532 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a-kube-api-access-v9gt6\" (UniqueName: \"kubernetes.io/projected/1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a-kube-api-access-v9gt6\") pod \"1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a\" (UID: \"1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a\") " Oct 30 17:15:48.104820 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.104731 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a-plugins-config-volume\") pod \"1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a\" (UID: \"1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a\") " Oct 30 17:15:48.105084 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.105065 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnns5\" (UniqueName: \"kubernetes.io/projected/6a0d47fd-b562-4eb1-b283-23bd21abc026-kube-api-access-cnns5\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:48.105227 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.105091 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a-plugins-config-volume" pod "1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a" (UID: "1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:48.106916 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.106888 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a-kube-api-access-v9gt6" pod "1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a" (UID: "1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a"). InnerVolumeSpecName "kube-api-access-v9gt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:48.206524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.206488 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:48.206524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.206521 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9gt6\" (UniqueName: \"kubernetes.io/projected/1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a-kube-api-access-v9gt6\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:48.411611 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.411577 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35198b3d-3a56-45c7-8d36-d4de1780a938" path="/var/lib/kubelet/pods/35198b3d-3a56-45c7-8d36-d4de1780a938/volumes" Oct 30 17:15:48.503405 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.503283 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:48.504640 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.504452 6202 generic.go:356] "Generic (PLEG): container finished" podID="1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a" containerID="7fa91a5f3fe2fa7631292c931a2c932dfea0dff88260059ccfbccd9e9f739769" exitCode=0 Oct 30 17:15:48.504640 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.504637 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d" Oct 30 17:15:48.504916 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.504616 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d" event={"ID":"1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a","Type":"ContainerDied","Data":"7fa91a5f3fe2fa7631292c931a2c932dfea0dff88260059ccfbccd9e9f739769"} Oct 30 17:15:48.504916 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.504806 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d" event={"ID":"1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a","Type":"ContainerDied","Data":"7bf1675b34c03cd39bca0d9f1ac5eb4d62214906ee6f22eca773ed0165dbc6e4"} Oct 30 17:15:48.504916 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.504830 6202 scope.go:122] "RemoveContainer" containerID="7fa91a5f3fe2fa7631292c931a2c932dfea0dff88260059ccfbccd9e9f739769" Oct 30 17:15:48.506625 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.506565 6202 generic.go:356] "Generic (PLEG): container finished" podID="6a0d47fd-b562-4eb1-b283-23bd21abc026" containerID="2aaccceff6899cac6c28cf12aa3252ee21765ce9a8a8406a82c87e139d018ab1" exitCode=0 Oct 30 17:15:48.507415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.506621 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn" event={"ID":"6a0d47fd-b562-4eb1-b283-23bd21abc026","Type":"ContainerDied","Data":"2aaccceff6899cac6c28cf12aa3252ee21765ce9a8a8406a82c87e139d018ab1"} Oct 30 17:15:48.507415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.506630 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn" Oct 30 17:15:48.507415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.506670 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn" event={"ID":"6a0d47fd-b562-4eb1-b283-23bd21abc026","Type":"ContainerDied","Data":"5388e643ad5301321ddd2b6a627adddb23f0e7766954a3a22fb647eb11b3222e"} Oct 30 17:15:48.518972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.518817 6202 scope.go:122] "RemoveContainer" containerID="7fa91a5f3fe2fa7631292c931a2c932dfea0dff88260059ccfbccd9e9f739769" Oct 30 17:15:48.519122 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:48.519104 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa91a5f3fe2fa7631292c931a2c932dfea0dff88260059ccfbccd9e9f739769\": container with ID starting with 7fa91a5f3fe2fa7631292c931a2c932dfea0dff88260059ccfbccd9e9f739769 not found: ID does not exist" containerID="7fa91a5f3fe2fa7631292c931a2c932dfea0dff88260059ccfbccd9e9f739769" Oct 30 17:15:48.519177 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.519129 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa91a5f3fe2fa7631292c931a2c932dfea0dff88260059ccfbccd9e9f739769"} err="failed to get container status \"7fa91a5f3fe2fa7631292c931a2c932dfea0dff88260059ccfbccd9e9f739769\": rpc error: code = NotFound desc = could not find container \"7fa91a5f3fe2fa7631292c931a2c932dfea0dff88260059ccfbccd9e9f739769\": container with ID starting with 7fa91a5f3fe2fa7631292c931a2c932dfea0dff88260059ccfbccd9e9f739769 not found: ID does not exist" Oct 30 17:15:48.519177 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.519145 6202 scope.go:122] "RemoveContainer" containerID="2aaccceff6899cac6c28cf12aa3252ee21765ce9a8a8406a82c87e139d018ab1" Oct 30 17:15:48.529376 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.529361 6202 scope.go:122] "RemoveContainer" containerID="2aaccceff6899cac6c28cf12aa3252ee21765ce9a8a8406a82c87e139d018ab1" Oct 30 17:15:48.529662 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:48.529639 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aaccceff6899cac6c28cf12aa3252ee21765ce9a8a8406a82c87e139d018ab1\": container with ID starting with 2aaccceff6899cac6c28cf12aa3252ee21765ce9a8a8406a82c87e139d018ab1 not found: ID does not exist" containerID="2aaccceff6899cac6c28cf12aa3252ee21765ce9a8a8406a82c87e139d018ab1" Oct 30 17:15:48.529733 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.529671 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aaccceff6899cac6c28cf12aa3252ee21765ce9a8a8406a82c87e139d018ab1"} err="failed to get container status \"2aaccceff6899cac6c28cf12aa3252ee21765ce9a8a8406a82c87e139d018ab1\": rpc error: code = NotFound desc = could not find container \"2aaccceff6899cac6c28cf12aa3252ee21765ce9a8a8406a82c87e139d018ab1\": container with ID starting with 2aaccceff6899cac6c28cf12aa3252ee21765ce9a8a8406a82c87e139d018ab1 not found: ID does not exist" Oct 30 17:15:48.560924 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.560890 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn"] Oct 30 17:15:48.573862 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.573837 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ae3fe-694f84545b-grwrn"] Oct 30 17:15:48.602296 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.602266 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d"] Oct 30 17:15:48.606098 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.606076 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ae3fe-9748bd7f-jjh5d"] Oct 30 17:15:48.647374 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.647345 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b"] Oct 30 17:15:48.678753 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.678682 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk"] Oct 30 17:15:48.787322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.787282 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:48.910833 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:48.910799 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:49.930214 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:49.930182 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:50.411330 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:50.411295 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a" path="/var/lib/kubelet/pods/1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a/volumes" Oct 30 17:15:50.411841 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:50.411814 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a0d47fd-b562-4eb1-b283-23bd21abc026" path="/var/lib/kubelet/pods/6a0d47fd-b562-4eb1-b283-23bd21abc026/volumes" Oct 30 17:15:52.105096 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:52.105064 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:53.548266 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:53.548225 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:53.647743 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:53.647696 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" podUID="e1cedd9a-5798-4463-a6b5-a8f199716991" containerName="payload-processing" containerID="cri-o://73d775ca11fc54629d826aaf35d3659503995932e95768b116a0c82aa3c8672d" gracePeriod=25 Oct 30 17:15:53.668013 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:53.667979 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx"] Oct 30 17:15:53.668219 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:53.668181 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx" podUID="31841021-3c67-427d-9b3b-8ebfdcd657b6" containerName="payload-processing" containerID="cri-o://b1b16e1dbbe8f682366cb0f32f7cf72be41e0dc29be8ccf30cc4913b3644bf72" gracePeriod=30 Oct 30 17:15:53.679702 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:53.679651 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" podUID="be7ca116-75e7-468a-a943-4113a9fd941b" containerName="payload-pre-processing" containerID="cri-o://6c45adfa6ff6a5b1f69168048a40d02c0fa60b9e3326f8949abc798a0f0a269c" gracePeriod=25 Oct 30 17:15:53.690325 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:53.690295 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f"] Oct 30 17:15:53.690542 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:53.690518 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f" podUID="1a4abfde-fb32-4553-930a-d415efa3b941" containerName="payload-pre-processing" containerID="cri-o://56764dc14cd9ff11ba759e59bbed2f23b3fd4e162745c7684719dbf877a3485c" gracePeriod=30 Oct 30 17:15:53.755004 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:53.754975 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:53.871487 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:53.871442 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:54.115196 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.115103 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx" Oct 30 17:15:54.130238 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.130209 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f" Oct 30 17:15:54.165141 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.165105 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:54.202742 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.202696 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a4abfde-fb32-4553-930a-d415efa3b941-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a4abfde-fb32-4553-930a-d415efa3b941-plugins-config-volume\") pod \"1a4abfde-fb32-4553-930a-d415efa3b941\" (UID: \"1a4abfde-fb32-4553-930a-d415efa3b941\") " Oct 30 17:15:54.202964 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.202810 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4abfde-fb32-4553-930a-d415efa3b941-kube-api-access-ddn4z\" (UniqueName: \"kubernetes.io/projected/1a4abfde-fb32-4553-930a-d415efa3b941-kube-api-access-ddn4z\") pod \"1a4abfde-fb32-4553-930a-d415efa3b941\" (UID: \"1a4abfde-fb32-4553-930a-d415efa3b941\") " Oct 30 17:15:54.202964 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.202939 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31841021-3c67-427d-9b3b-8ebfdcd657b6-kube-api-access-jcgj9\" (UniqueName: \"kubernetes.io/projected/31841021-3c67-427d-9b3b-8ebfdcd657b6-kube-api-access-jcgj9\") pod \"31841021-3c67-427d-9b3b-8ebfdcd657b6\" (UID: \"31841021-3c67-427d-9b3b-8ebfdcd657b6\") " Oct 30 17:15:54.203374 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.203341 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4abfde-fb32-4553-930a-d415efa3b941-plugins-config-volume" pod "1a4abfde-fb32-4553-930a-d415efa3b941" (UID: "1a4abfde-fb32-4553-930a-d415efa3b941"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:54.203502 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.203396 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31841021-3c67-427d-9b3b-8ebfdcd657b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31841021-3c67-427d-9b3b-8ebfdcd657b6-plugins-config-volume\") pod \"31841021-3c67-427d-9b3b-8ebfdcd657b6\" (UID: \"31841021-3c67-427d-9b3b-8ebfdcd657b6\") " Oct 30 17:15:54.203953 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.203924 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a4abfde-fb32-4553-930a-d415efa3b941-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:54.204292 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.204258 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31841021-3c67-427d-9b3b-8ebfdcd657b6-plugins-config-volume" pod "31841021-3c67-427d-9b3b-8ebfdcd657b6" (UID: "31841021-3c67-427d-9b3b-8ebfdcd657b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:54.207181 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.206975 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4abfde-fb32-4553-930a-d415efa3b941-kube-api-access-ddn4z" pod "1a4abfde-fb32-4553-930a-d415efa3b941" (UID: "1a4abfde-fb32-4553-930a-d415efa3b941"). InnerVolumeSpecName "kube-api-access-ddn4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:54.210264 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.210234 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31841021-3c67-427d-9b3b-8ebfdcd657b6-kube-api-access-jcgj9" pod "31841021-3c67-427d-9b3b-8ebfdcd657b6" (UID: "31841021-3c67-427d-9b3b-8ebfdcd657b6"). InnerVolumeSpecName "kube-api-access-jcgj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:54.281208 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.281167 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:54.304839 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.304807 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcgj9\" (UniqueName: \"kubernetes.io/projected/31841021-3c67-427d-9b3b-8ebfdcd657b6-kube-api-access-jcgj9\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:54.304839 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.304842 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31841021-3c67-427d-9b3b-8ebfdcd657b6-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:54.305072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.304852 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddn4z\" (UniqueName: \"kubernetes.io/projected/1a4abfde-fb32-4553-930a-d415efa3b941-kube-api-access-ddn4z\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:54.391469 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.391437 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:54.556509 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.556476 6202 generic.go:356] "Generic (PLEG): container finished" podID="1a4abfde-fb32-4553-930a-d415efa3b941" containerID="56764dc14cd9ff11ba759e59bbed2f23b3fd4e162745c7684719dbf877a3485c" exitCode=0 Oct 30 17:15:54.556987 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.556546 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f" Oct 30 17:15:54.556987 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.556563 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f" event={"ID":"1a4abfde-fb32-4553-930a-d415efa3b941","Type":"ContainerDied","Data":"56764dc14cd9ff11ba759e59bbed2f23b3fd4e162745c7684719dbf877a3485c"} Oct 30 17:15:54.556987 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.556617 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f" event={"ID":"1a4abfde-fb32-4553-930a-d415efa3b941","Type":"ContainerDied","Data":"6ce02159734a0f53e6f2ff6ad9767c054470559e9d68f6e70dc15f8153fd70a7"} Oct 30 17:15:54.556987 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.556644 6202 scope.go:122] "RemoveContainer" containerID="56764dc14cd9ff11ba759e59bbed2f23b3fd4e162745c7684719dbf877a3485c" Oct 30 17:15:54.558491 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.558456 6202 generic.go:356] "Generic (PLEG): container finished" podID="31841021-3c67-427d-9b3b-8ebfdcd657b6" containerID="b1b16e1dbbe8f682366cb0f32f7cf72be41e0dc29be8ccf30cc4913b3644bf72" exitCode=0 Oct 30 17:15:54.558491 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.558491 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx" event={"ID":"31841021-3c67-427d-9b3b-8ebfdcd657b6","Type":"ContainerDied","Data":"b1b16e1dbbe8f682366cb0f32f7cf72be41e0dc29be8ccf30cc4913b3644bf72"} Oct 30 17:15:54.558715 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.558530 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx" event={"ID":"31841021-3c67-427d-9b3b-8ebfdcd657b6","Type":"ContainerDied","Data":"30dba14689feece46ebddb3fc495e9e3eb5a29b7d84ae2e0720dfa92fecbf4e9"} Oct 30 17:15:54.558802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.558725 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx" Oct 30 17:15:54.566088 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.566056 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" podUID="2dd4faa7-90ff-4815-b7c5-69081a00ed05" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:15:54.567014 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.566990 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" podUID="7bec536b-a62b-42d7-810d-fa8090168784" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:15:54.570276 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.570252 6202 scope.go:122] "RemoveContainer" containerID="56764dc14cd9ff11ba759e59bbed2f23b3fd4e162745c7684719dbf877a3485c" Oct 30 17:15:54.570641 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:54.570605 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56764dc14cd9ff11ba759e59bbed2f23b3fd4e162745c7684719dbf877a3485c\": container with ID starting with 56764dc14cd9ff11ba759e59bbed2f23b3fd4e162745c7684719dbf877a3485c not found: ID does not exist" containerID="56764dc14cd9ff11ba759e59bbed2f23b3fd4e162745c7684719dbf877a3485c" Oct 30 17:15:54.570752 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.570648 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56764dc14cd9ff11ba759e59bbed2f23b3fd4e162745c7684719dbf877a3485c"} err="failed to get container status \"56764dc14cd9ff11ba759e59bbed2f23b3fd4e162745c7684719dbf877a3485c\": rpc error: code = NotFound desc = could not find container \"56764dc14cd9ff11ba759e59bbed2f23b3fd4e162745c7684719dbf877a3485c\": container with ID starting with 56764dc14cd9ff11ba759e59bbed2f23b3fd4e162745c7684719dbf877a3485c not found: ID does not exist" Oct 30 17:15:54.570752 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.570673 6202 scope.go:122] "RemoveContainer" containerID="b1b16e1dbbe8f682366cb0f32f7cf72be41e0dc29be8ccf30cc4913b3644bf72" Oct 30 17:15:54.582714 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.582687 6202 scope.go:122] "RemoveContainer" containerID="b1b16e1dbbe8f682366cb0f32f7cf72be41e0dc29be8ccf30cc4913b3644bf72" Oct 30 17:15:54.583039 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:15:54.583016 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b16e1dbbe8f682366cb0f32f7cf72be41e0dc29be8ccf30cc4913b3644bf72\": container with ID starting with b1b16e1dbbe8f682366cb0f32f7cf72be41e0dc29be8ccf30cc4913b3644bf72 not found: ID does not exist" containerID="b1b16e1dbbe8f682366cb0f32f7cf72be41e0dc29be8ccf30cc4913b3644bf72" Oct 30 17:15:54.583155 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.583047 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b16e1dbbe8f682366cb0f32f7cf72be41e0dc29be8ccf30cc4913b3644bf72"} err="failed to get container status \"b1b16e1dbbe8f682366cb0f32f7cf72be41e0dc29be8ccf30cc4913b3644bf72\": rpc error: code = NotFound desc = could not find container \"b1b16e1dbbe8f682366cb0f32f7cf72be41e0dc29be8ccf30cc4913b3644bf72\": container with ID starting with b1b16e1dbbe8f682366cb0f32f7cf72be41e0dc29be8ccf30cc4913b3644bf72 not found: ID does not exist" Oct 30 17:15:54.611528 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.611503 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx"] Oct 30 17:15:54.615720 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.615697 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97a47d-67b9cc96dd-rvkjx"] Oct 30 17:15:54.653126 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.653050 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f"] Oct 30 17:15:54.656029 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.656003 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97a47d-56d7954947-pqs7f"] Oct 30 17:15:54.813388 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:54.813345 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:55.047215 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.047132 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:55.159399 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.159366 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:55.216401 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.216361 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" podUID="be7ca116-75e7-468a-a943-4113a9fd941b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:15:55.216547 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.216367 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" podUID="e1cedd9a-5798-4463-a6b5-a8f199716991" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:15:55.393920 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.393880 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:55.929735 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.929698 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp"] Oct 30 17:15:55.930342 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930328 6202 cpu_manager.go:414] "containerMap: removing container" podUID="35198b3d-3a56-45c7-8d36-d4de1780a938" containerName="istio-proxy" Oct 30 17:15:55.930387 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930343 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35198b3d-3a56-45c7-8d36-d4de1780a938" containerName="istio-proxy" Oct 30 17:15:55.930387 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930369 6202 cpu_manager.go:414] "containerMap: removing container" podUID="cc285144-2c80-4d79-8269-3994f8858ab3" containerName="revoke-keys" Oct 30 17:15:55.930387 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930375 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc285144-2c80-4d79-8269-3994f8858ab3" containerName="revoke-keys" Oct 30 17:15:55.930387 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930386 6202 cpu_manager.go:414] "containerMap: removing container" podUID="2d98a198-b4de-422f-a2f4-df5e059d105d" containerName="payload-processing" Oct 30 17:15:55.930516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930393 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d98a198-b4de-422f-a2f4-df5e059d105d" containerName="payload-processing" Oct 30 17:15:55.930516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930408 6202 cpu_manager.go:414] "containerMap: removing container" podUID="1a4abfde-fb32-4553-930a-d415efa3b941" containerName="payload-pre-processing" Oct 30 17:15:55.930516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930413 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4abfde-fb32-4553-930a-d415efa3b941" containerName="payload-pre-processing" Oct 30 17:15:55.930516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930422 6202 cpu_manager.go:414] "containerMap: removing container" podUID="6a0d47fd-b562-4eb1-b283-23bd21abc026" containerName="payload-processing" Oct 30 17:15:55.930516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930426 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0d47fd-b562-4eb1-b283-23bd21abc026" containerName="payload-processing" Oct 30 17:15:55.930516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930433 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5af8f435-0196-487a-b4ce-017b4adc6f50" containerName="payload-processing" Oct 30 17:15:55.930516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930438 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af8f435-0196-487a-b4ce-017b4adc6f50" containerName="payload-processing" Oct 30 17:15:55.930516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930446 6202 cpu_manager.go:414] "containerMap: removing container" podUID="31841021-3c67-427d-9b3b-8ebfdcd657b6" containerName="payload-processing" Oct 30 17:15:55.930516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930451 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31841021-3c67-427d-9b3b-8ebfdcd657b6" containerName="payload-processing" Oct 30 17:15:55.930516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930459 6202 cpu_manager.go:414] "containerMap: removing container" podUID="cfeb39d1-2907-41de-afaa-8c3b0690a828" containerName="revoke-keys" Oct 30 17:15:55.930516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930463 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfeb39d1-2907-41de-afaa-8c3b0690a828" containerName="revoke-keys" Oct 30 17:15:55.930516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930474 6202 cpu_manager.go:414] "containerMap: removing container" podUID="1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a" containerName="payload-pre-processing" Oct 30 17:15:55.930516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930479 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a" containerName="payload-pre-processing" Oct 30 17:15:55.930516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930488 6202 cpu_manager.go:414] "containerMap: removing container" podUID="122dfb4c-4008-4ee9-881f-d0e8a4525e05" containerName="payload-pre-processing" Oct 30 17:15:55.930516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930493 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122dfb4c-4008-4ee9-881f-d0e8a4525e05" containerName="payload-pre-processing" Oct 30 17:15:55.931051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930566 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="35198b3d-3a56-45c7-8d36-d4de1780a938" containerName="istio-proxy" Oct 30 17:15:55.931051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930578 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc285144-2c80-4d79-8269-3994f8858ab3" containerName="revoke-keys" Oct 30 17:15:55.931051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930584 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d98a198-b4de-422f-a2f4-df5e059d105d" containerName="payload-processing" Oct 30 17:15:55.931051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930592 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4abfde-fb32-4553-930a-d415efa3b941" containerName="payload-pre-processing" Oct 30 17:15:55.931051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930598 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0d47fd-b562-4eb1-b283-23bd21abc026" containerName="payload-processing" Oct 30 17:15:55.931051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930603 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af8f435-0196-487a-b4ce-017b4adc6f50" containerName="payload-processing" Oct 30 17:15:55.931051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930608 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="31841021-3c67-427d-9b3b-8ebfdcd657b6" containerName="payload-processing" Oct 30 17:15:55.931051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930614 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfeb39d1-2907-41de-afaa-8c3b0690a828" containerName="revoke-keys" Oct 30 17:15:55.931051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930620 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fdc1f9b-8e02-4deb-9bf3-7941d8b6047a" containerName="payload-pre-processing" Oct 30 17:15:55.931051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.930625 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="122dfb4c-4008-4ee9-881f-d0e8a4525e05" containerName="payload-pre-processing" Oct 30 17:15:55.934249 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.934233 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp" Oct 30 17:15:55.955118 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.955073 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp"] Oct 30 17:15:55.968739 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.964616 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:15:55.968739 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.964683 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:15:55.968739 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.964733 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:15:55.968739 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:55.964827 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:15:56.021990 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:56.021955 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcwwv\" (UniqueName: \"kubernetes.io/projected/f87a1115-7d38-4949-b6aa-72e5987f8bc5-kube-api-access-xcwwv\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp\" (UID: \"f87a1115-7d38-4949-b6aa-72e5987f8bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp" Oct 30 17:15:56.021990 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:56.021993 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f87a1115-7d38-4949-b6aa-72e5987f8bc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp\" (UID: \"f87a1115-7d38-4949-b6aa-72e5987f8bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp" Oct 30 17:15:56.022197 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:56.022145 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f87a1115-7d38-4949-b6aa-72e5987f8bc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp\" (UID: \"f87a1115-7d38-4949-b6aa-72e5987f8bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp" Oct 30 17:15:56.123625 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:56.123591 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcwwv\" (UniqueName: \"kubernetes.io/projected/f87a1115-7d38-4949-b6aa-72e5987f8bc5-kube-api-access-xcwwv\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp\" (UID: \"f87a1115-7d38-4949-b6aa-72e5987f8bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp" Oct 30 17:15:56.123625 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:56.123630 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f87a1115-7d38-4949-b6aa-72e5987f8bc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp\" (UID: \"f87a1115-7d38-4949-b6aa-72e5987f8bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp" Oct 30 17:15:56.123910 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:56.123761 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f87a1115-7d38-4949-b6aa-72e5987f8bc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp\" (UID: \"f87a1115-7d38-4949-b6aa-72e5987f8bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp" Oct 30 17:15:56.124353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:56.124331 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f87a1115-7d38-4949-b6aa-72e5987f8bc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp\" (UID: \"f87a1115-7d38-4949-b6aa-72e5987f8bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp" Oct 30 17:15:56.126271 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:56.126246 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f87a1115-7d38-4949-b6aa-72e5987f8bc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp\" (UID: \"f87a1115-7d38-4949-b6aa-72e5987f8bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp" Oct 30 17:15:56.149008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:56.148975 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcwwv\" (UniqueName: \"kubernetes.io/projected/f87a1115-7d38-4949-b6aa-72e5987f8bc5-kube-api-access-xcwwv\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp\" (UID: \"f87a1115-7d38-4949-b6aa-72e5987f8bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp" Oct 30 17:15:56.250059 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:56.249963 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp" Oct 30 17:15:56.411605 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:56.411571 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a4abfde-fb32-4553-930a-d415efa3b941" path="/var/lib/kubelet/pods/1a4abfde-fb32-4553-930a-d415efa3b941/volumes" Oct 30 17:15:56.412025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:56.412008 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31841021-3c67-427d-9b3b-8ebfdcd657b6" path="/var/lib/kubelet/pods/31841021-3c67-427d-9b3b-8ebfdcd657b6/volumes" Oct 30 17:15:56.416440 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:56.416417 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:56.560140 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:56.560106 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp"] Oct 30 17:15:56.560971 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:15:56.560933 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87a1115_7d38_4949_b6aa_72e5987f8bc5.slice/crio-44ad6bd6f49badc8dd2091ee4c1be2370a324f8ad2b7a1d336d137b8b778e5d4 WatchSource:0}: Error finding container 44ad6bd6f49badc8dd2091ee4c1be2370a324f8ad2b7a1d336d137b8b778e5d4: Status 404 returned error can't find the container with id 44ad6bd6f49badc8dd2091ee4c1be2370a324f8ad2b7a1d336d137b8b778e5d4 Oct 30 17:15:56.592436 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:56.592404 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp" event={"ID":"f87a1115-7d38-4949-b6aa-72e5987f8bc5","Type":"ContainerStarted","Data":"44ad6bd6f49badc8dd2091ee4c1be2370a324f8ad2b7a1d336d137b8b778e5d4"} Oct 30 17:15:57.600962 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:57.600930 6202 generic.go:356] "Generic (PLEG): container finished" podID="f87a1115-7d38-4949-b6aa-72e5987f8bc5" containerID="fb9c21c7bb078610a88e23ded98f7a6a4c5d0331c950a22b9087850c3d7dbfda" exitCode=0 Oct 30 17:15:57.601368 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:57.601027 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp" event={"ID":"f87a1115-7d38-4949-b6aa-72e5987f8bc5","Type":"ContainerDied","Data":"fb9c21c7bb078610a88e23ded98f7a6a4c5d0331c950a22b9087850c3d7dbfda"} Oct 30 17:15:58.293687 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:58.293653 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:58.780401 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:58.780370 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp" Oct 30 17:15:58.868738 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:58.868646 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f87a1115-7d38-4949-b6aa-72e5987f8bc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f87a1115-7d38-4949-b6aa-72e5987f8bc5-maas-api-openshift-ca\") pod \"f87a1115-7d38-4949-b6aa-72e5987f8bc5\" (UID: \"f87a1115-7d38-4949-b6aa-72e5987f8bc5\") " Oct 30 17:15:58.868925 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:58.868852 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f87a1115-7d38-4949-b6aa-72e5987f8bc5-kube-api-access-xcwwv\" (UniqueName: \"kubernetes.io/projected/f87a1115-7d38-4949-b6aa-72e5987f8bc5-kube-api-access-xcwwv\") pod \"f87a1115-7d38-4949-b6aa-72e5987f8bc5\" (UID: \"f87a1115-7d38-4949-b6aa-72e5987f8bc5\") " Oct 30 17:15:58.868998 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:58.868922 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f87a1115-7d38-4949-b6aa-72e5987f8bc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f87a1115-7d38-4949-b6aa-72e5987f8bc5-maas-api-xks-ca\") pod \"f87a1115-7d38-4949-b6aa-72e5987f8bc5\" (UID: \"f87a1115-7d38-4949-b6aa-72e5987f8bc5\") " Oct 30 17:15:58.869128 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:58.869096 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87a1115-7d38-4949-b6aa-72e5987f8bc5-maas-api-openshift-ca" pod "f87a1115-7d38-4949-b6aa-72e5987f8bc5" (UID: "f87a1115-7d38-4949-b6aa-72e5987f8bc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:58.869468 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:58.869442 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f87a1115-7d38-4949-b6aa-72e5987f8bc5-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:58.871682 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:58.871647 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f87a1115-7d38-4949-b6aa-72e5987f8bc5-maas-api-xks-ca" pod "f87a1115-7d38-4949-b6aa-72e5987f8bc5" (UID: "f87a1115-7d38-4949-b6aa-72e5987f8bc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:58.871867 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:58.871780 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f87a1115-7d38-4949-b6aa-72e5987f8bc5-kube-api-access-xcwwv" pod "f87a1115-7d38-4949-b6aa-72e5987f8bc5" (UID: "f87a1115-7d38-4949-b6aa-72e5987f8bc5"). InnerVolumeSpecName "kube-api-access-xcwwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:58.970584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:58.970548 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcwwv\" (UniqueName: \"kubernetes.io/projected/f87a1115-7d38-4949-b6aa-72e5987f8bc5-kube-api-access-xcwwv\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:58.970584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:58.970582 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f87a1115-7d38-4949-b6aa-72e5987f8bc5-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:15:59.619331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:59.619299 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp" Oct 30 17:15:59.619504 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:59.619323 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp" event={"ID":"f87a1115-7d38-4949-b6aa-72e5987f8bc5","Type":"ContainerDied","Data":"44ad6bd6f49badc8dd2091ee4c1be2370a324f8ad2b7a1d336d137b8b778e5d4"} Oct 30 17:15:59.619504 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:15:59.619382 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44ad6bd6f49badc8dd2091ee4c1be2370a324f8ad2b7a1d336d137b8b778e5d4" Oct 30 17:16:00.070456 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:00.070364 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:01.521760 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:01.521724 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:01.635100 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:01.635061 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:01.895705 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:01.895658 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:02.013856 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.013747 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:02.386956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.386918 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p"] Oct 30 17:16:02.387557 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.387543 6202 cpu_manager.go:414] "containerMap: removing container" podUID="f87a1115-7d38-4949-b6aa-72e5987f8bc5" containerName="revoke-keys" Oct 30 17:16:02.387600 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.387560 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f87a1115-7d38-4949-b6aa-72e5987f8bc5" containerName="revoke-keys" Oct 30 17:16:02.387681 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.387672 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="f87a1115-7d38-4949-b6aa-72e5987f8bc5" containerName="revoke-keys" Oct 30 17:16:02.391494 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.391477 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p" Oct 30 17:16:02.424813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.420452 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea89c83-ccfe-448f-83f2-afef35ef058b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p\" (UID: \"1ea89c83-ccfe-448f-83f2-afef35ef058b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p" Oct 30 17:16:02.424813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.420553 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p8pj\" (UniqueName: \"kubernetes.io/projected/1ea89c83-ccfe-448f-83f2-afef35ef058b-kube-api-access-2p8pj\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p\" (UID: \"1ea89c83-ccfe-448f-83f2-afef35ef058b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p" Oct 30 17:16:02.424813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.420690 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea89c83-ccfe-448f-83f2-afef35ef058b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p\" (UID: \"1ea89c83-ccfe-448f-83f2-afef35ef058b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p" Oct 30 17:16:02.424813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.421528 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:16:02.424813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.421631 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:16:02.424813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.421728 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:16:02.424813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.421919 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:16:02.426702 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.426674 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p"] Oct 30 17:16:02.522392 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.522359 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea89c83-ccfe-448f-83f2-afef35ef058b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p\" (UID: \"1ea89c83-ccfe-448f-83f2-afef35ef058b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p" Oct 30 17:16:02.522862 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.522418 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea89c83-ccfe-448f-83f2-afef35ef058b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p\" (UID: \"1ea89c83-ccfe-448f-83f2-afef35ef058b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p" Oct 30 17:16:02.522862 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.522492 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p8pj\" (UniqueName: \"kubernetes.io/projected/1ea89c83-ccfe-448f-83f2-afef35ef058b-kube-api-access-2p8pj\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p\" (UID: \"1ea89c83-ccfe-448f-83f2-afef35ef058b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p" Oct 30 17:16:02.523396 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.523376 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea89c83-ccfe-448f-83f2-afef35ef058b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p\" (UID: \"1ea89c83-ccfe-448f-83f2-afef35ef058b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p" Oct 30 17:16:02.525004 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.524981 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea89c83-ccfe-448f-83f2-afef35ef058b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p\" (UID: \"1ea89c83-ccfe-448f-83f2-afef35ef058b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p" Oct 30 17:16:02.547533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.547500 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p8pj\" (UniqueName: \"kubernetes.io/projected/1ea89c83-ccfe-448f-83f2-afef35ef058b-kube-api-access-2p8pj\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p\" (UID: \"1ea89c83-ccfe-448f-83f2-afef35ef058b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p" Oct 30 17:16:02.710376 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:02.710293 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p" Oct 30 17:16:03.012256 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:16:03.012216 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea89c83_ccfe_448f_83f2_afef35ef058b.slice/crio-6035f126940de5b9b3b7c2a4a0c8e31b5f26c1c340193205395273a8885686bf WatchSource:0}: Error finding container 6035f126940de5b9b3b7c2a4a0c8e31b5f26c1c340193205395273a8885686bf: Status 404 returned error can't find the container with id 6035f126940de5b9b3b7c2a4a0c8e31b5f26c1c340193205395273a8885686bf Oct 30 17:16:03.012967 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:03.012944 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p"] Oct 30 17:16:03.488405 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:03.488369 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:03.654852 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:03.654744 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p" event={"ID":"1ea89c83-ccfe-448f-83f2-afef35ef058b","Type":"ContainerStarted","Data":"8413d9e67328febb578ac9921cd19768e611308fb23476121f8e9db6b60d9c09"} Oct 30 17:16:03.655366 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:03.655339 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p" event={"ID":"1ea89c83-ccfe-448f-83f2-afef35ef058b","Type":"ContainerStarted","Data":"6035f126940de5b9b3b7c2a4a0c8e31b5f26c1c340193205395273a8885686bf"} Oct 30 17:16:03.726792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:03.726195 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p" podStartSLOduration=1.726174373 podStartE2EDuration="1.726174373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:03.725710436 +0000 UTC m=+3277.436165180" watchObservedRunningTime="2026-10-30 17:16:03.726174373 +0000 UTC m=+3277.436629119" Oct 30 17:16:04.134455 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:04.134421 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:04.241541 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:04.241507 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w"] Oct 30 17:16:04.254821 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:04.254756 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-661509-nr65w"] Oct 30 17:16:04.320009 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:04.319923 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:04.411145 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:04.411113 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a40eabb7-728d-44aa-bb47-5ec098f715e6" path="/var/lib/kubelet/pods/a40eabb7-728d-44aa-bb47-5ec098f715e6/volumes" Oct 30 17:16:04.566196 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:04.566157 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" podUID="2dd4faa7-90ff-4815-b7c5-69081a00ed05" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:04.567165 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:04.567130 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" podUID="7bec536b-a62b-42d7-810d-fa8090168784" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:04.664630 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:04.664590 6202 generic.go:356] "Generic (PLEG): container finished" podID="1ea89c83-ccfe-448f-83f2-afef35ef058b" containerID="8413d9e67328febb578ac9921cd19768e611308fb23476121f8e9db6b60d9c09" exitCode=0 Oct 30 17:16:04.665069 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:04.664675 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p" event={"ID":"1ea89c83-ccfe-448f-83f2-afef35ef058b","Type":"ContainerDied","Data":"8413d9e67328febb578ac9921cd19768e611308fb23476121f8e9db6b60d9c09"} Oct 30 17:16:04.719545 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:04.719482 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:05.216180 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:05.216142 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" podUID="e1cedd9a-5798-4463-a6b5-a8f199716991" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:05.216363 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:05.216185 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" podUID="be7ca116-75e7-468a-a943-4113a9fd941b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:05.842534 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:05.842501 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p" Oct 30 17:16:05.850879 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:05.850845 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:05.977115 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:05.977034 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ea89c83-ccfe-448f-83f2-afef35ef058b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea89c83-ccfe-448f-83f2-afef35ef058b-maas-api-xks-ca\") pod \"1ea89c83-ccfe-448f-83f2-afef35ef058b\" (UID: \"1ea89c83-ccfe-448f-83f2-afef35ef058b\") " Oct 30 17:16:05.977115 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:05.977113 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea89c83-ccfe-448f-83f2-afef35ef058b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea89c83-ccfe-448f-83f2-afef35ef058b-maas-api-openshift-ca\") pod \"1ea89c83-ccfe-448f-83f2-afef35ef058b\" (UID: \"1ea89c83-ccfe-448f-83f2-afef35ef058b\") " Oct 30 17:16:05.977357 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:05.977221 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea89c83-ccfe-448f-83f2-afef35ef058b-kube-api-access-2p8pj\" (UniqueName: \"kubernetes.io/projected/1ea89c83-ccfe-448f-83f2-afef35ef058b-kube-api-access-2p8pj\") pod \"1ea89c83-ccfe-448f-83f2-afef35ef058b\" (UID: \"1ea89c83-ccfe-448f-83f2-afef35ef058b\") " Oct 30 17:16:05.977434 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:05.977423 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea89c83-ccfe-448f-83f2-afef35ef058b-maas-api-openshift-ca" pod "1ea89c83-ccfe-448f-83f2-afef35ef058b" (UID: "1ea89c83-ccfe-448f-83f2-afef35ef058b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:05.977861 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:05.977803 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea89c83-ccfe-448f-83f2-afef35ef058b-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:05.979435 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:05.979410 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ea89c83-ccfe-448f-83f2-afef35ef058b-maas-api-xks-ca" pod "1ea89c83-ccfe-448f-83f2-afef35ef058b" (UID: "1ea89c83-ccfe-448f-83f2-afef35ef058b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:05.979564 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:05.979544 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea89c83-ccfe-448f-83f2-afef35ef058b-kube-api-access-2p8pj" pod "1ea89c83-ccfe-448f-83f2-afef35ef058b" (UID: "1ea89c83-ccfe-448f-83f2-afef35ef058b"). InnerVolumeSpecName "kube-api-access-2p8pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:05.990609 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:05.990590 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c"] Oct 30 17:16:05.990858 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:05.990833 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c" podUID="adeec1f0-813d-4ade-bab6-b1f8d49a6f28" containerName="payload-processing" containerID="cri-o://59c77a4e488404f2c9e79e9464b7575e6698d10051175a5bdebc1b9739a68110" gracePeriod=30 Oct 30 17:16:06.010249 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.010216 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b"] Oct 30 17:16:06.010458 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.010431 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b" podUID="8f2f997e-e6fc-46d2-bb0f-68c947c8607f" containerName="payload-pre-processing" containerID="cri-o://71c57426929a633bc47406761c973ffeef018f3bab52ea70bb0befa874a15ebd" gracePeriod=30 Oct 30 17:16:06.075256 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.075182 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:06.087892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.079746 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p8pj\" (UniqueName: \"kubernetes.io/projected/1ea89c83-ccfe-448f-83f2-afef35ef058b-kube-api-access-2p8pj\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:06.087892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.079798 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea89c83-ccfe-448f-83f2-afef35ef058b-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:06.192694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.188712 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:06.311802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.310691 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c" Oct 30 17:16:06.329832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.328272 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:06.411453 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.409961 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adeec1f0-813d-4ade-bab6-b1f8d49a6f28-kube-api-access-f5h8d\" (UniqueName: \"kubernetes.io/projected/adeec1f0-813d-4ade-bab6-b1f8d49a6f28-kube-api-access-f5h8d\") pod \"adeec1f0-813d-4ade-bab6-b1f8d49a6f28\" (UID: \"adeec1f0-813d-4ade-bab6-b1f8d49a6f28\") " Oct 30 17:16:06.411453 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.410011 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adeec1f0-813d-4ade-bab6-b1f8d49a6f28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adeec1f0-813d-4ade-bab6-b1f8d49a6f28-plugins-config-volume\") pod \"adeec1f0-813d-4ade-bab6-b1f8d49a6f28\" (UID: \"adeec1f0-813d-4ade-bab6-b1f8d49a6f28\") " Oct 30 17:16:06.411453 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.410613 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adeec1f0-813d-4ade-bab6-b1f8d49a6f28-plugins-config-volume" pod "adeec1f0-813d-4ade-bab6-b1f8d49a6f28" (UID: "adeec1f0-813d-4ade-bab6-b1f8d49a6f28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:06.415405 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.415375 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adeec1f0-813d-4ade-bab6-b1f8d49a6f28-kube-api-access-f5h8d" pod "adeec1f0-813d-4ade-bab6-b1f8d49a6f28" (UID: "adeec1f0-813d-4ade-bab6-b1f8d49a6f28"). InnerVolumeSpecName "kube-api-access-f5h8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:06.458713 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.458681 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b" Oct 30 17:16:06.500300 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.495944 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:06.513411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.513367 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5h8d\" (UniqueName: \"kubernetes.io/projected/adeec1f0-813d-4ade-bab6-b1f8d49a6f28-kube-api-access-f5h8d\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:06.513411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.513411 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adeec1f0-813d-4ade-bab6-b1f8d49a6f28-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:06.632789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.632653 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f2f997e-e6fc-46d2-bb0f-68c947c8607f-kube-api-access-9cmws\" (UniqueName: \"kubernetes.io/projected/8f2f997e-e6fc-46d2-bb0f-68c947c8607f-kube-api-access-9cmws\") pod \"8f2f997e-e6fc-46d2-bb0f-68c947c8607f\" (UID: \"8f2f997e-e6fc-46d2-bb0f-68c947c8607f\") " Oct 30 17:16:06.633025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.632815 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f2f997e-e6fc-46d2-bb0f-68c947c8607f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2f997e-e6fc-46d2-bb0f-68c947c8607f-plugins-config-volume\") pod \"8f2f997e-e6fc-46d2-bb0f-68c947c8607f\" (UID: \"8f2f997e-e6fc-46d2-bb0f-68c947c8607f\") " Oct 30 17:16:06.633791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.633636 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2f997e-e6fc-46d2-bb0f-68c947c8607f-plugins-config-volume" pod "8f2f997e-e6fc-46d2-bb0f-68c947c8607f" (UID: "8f2f997e-e6fc-46d2-bb0f-68c947c8607f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:06.645791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.639238 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:06.659793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.655234 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2f997e-e6fc-46d2-bb0f-68c947c8607f-kube-api-access-9cmws" pod "8f2f997e-e6fc-46d2-bb0f-68c947c8607f" (UID: "8f2f997e-e6fc-46d2-bb0f-68c947c8607f"). InnerVolumeSpecName "kube-api-access-9cmws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:06.717331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.717285 6202 generic.go:356] "Generic (PLEG): container finished" podID="adeec1f0-813d-4ade-bab6-b1f8d49a6f28" containerID="59c77a4e488404f2c9e79e9464b7575e6698d10051175a5bdebc1b9739a68110" exitCode=0 Oct 30 17:16:06.717524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.717497 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c" event={"ID":"adeec1f0-813d-4ade-bab6-b1f8d49a6f28","Type":"ContainerDied","Data":"59c77a4e488404f2c9e79e9464b7575e6698d10051175a5bdebc1b9739a68110"} Oct 30 17:16:06.717591 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.717583 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c" event={"ID":"adeec1f0-813d-4ade-bab6-b1f8d49a6f28","Type":"ContainerDied","Data":"dbe6029b3bad149515bafd84d77ed8b2bba6ab3db58671a7fd6f5b1c369fe916"} Oct 30 17:16:06.717648 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.717612 6202 scope.go:122] "RemoveContainer" containerID="59c77a4e488404f2c9e79e9464b7575e6698d10051175a5bdebc1b9739a68110" Oct 30 17:16:06.717839 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.717824 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c" Oct 30 17:16:06.721254 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.721209 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p" event={"ID":"1ea89c83-ccfe-448f-83f2-afef35ef058b","Type":"ContainerDied","Data":"6035f126940de5b9b3b7c2a4a0c8e31b5f26c1c340193205395273a8885686bf"} Oct 30 17:16:06.721389 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.721281 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6035f126940de5b9b3b7c2a4a0c8e31b5f26c1c340193205395273a8885686bf" Oct 30 17:16:06.721469 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.721389 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p" Oct 30 17:16:06.725052 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.725024 6202 generic.go:356] "Generic (PLEG): container finished" podID="8f2f997e-e6fc-46d2-bb0f-68c947c8607f" containerID="71c57426929a633bc47406761c973ffeef018f3bab52ea70bb0befa874a15ebd" exitCode=0 Oct 30 17:16:06.725202 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.725072 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b" Oct 30 17:16:06.725202 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.725133 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b" event={"ID":"8f2f997e-e6fc-46d2-bb0f-68c947c8607f","Type":"ContainerDied","Data":"71c57426929a633bc47406761c973ffeef018f3bab52ea70bb0befa874a15ebd"} Oct 30 17:16:06.725340 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.725207 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b" event={"ID":"8f2f997e-e6fc-46d2-bb0f-68c947c8607f","Type":"ContainerDied","Data":"35732afc389625bea0373ce4d6f23dcc61f24428b1f3af1aabe5acec53ffb93e"} Oct 30 17:16:06.736978 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.736950 6202 scope.go:122] "RemoveContainer" containerID="59c77a4e488404f2c9e79e9464b7575e6698d10051175a5bdebc1b9739a68110" Oct 30 17:16:06.737470 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.737443 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2f997e-e6fc-46d2-bb0f-68c947c8607f-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:06.737583 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.737475 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cmws\" (UniqueName: \"kubernetes.io/projected/8f2f997e-e6fc-46d2-bb0f-68c947c8607f-kube-api-access-9cmws\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:06.737583 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:16:06.737509 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59c77a4e488404f2c9e79e9464b7575e6698d10051175a5bdebc1b9739a68110\": container with ID starting with 59c77a4e488404f2c9e79e9464b7575e6698d10051175a5bdebc1b9739a68110 not found: ID does not exist" containerID="59c77a4e488404f2c9e79e9464b7575e6698d10051175a5bdebc1b9739a68110" Oct 30 17:16:06.737583 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.737546 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c77a4e488404f2c9e79e9464b7575e6698d10051175a5bdebc1b9739a68110"} err="failed to get container status \"59c77a4e488404f2c9e79e9464b7575e6698d10051175a5bdebc1b9739a68110\": rpc error: code = NotFound desc = could not find container \"59c77a4e488404f2c9e79e9464b7575e6698d10051175a5bdebc1b9739a68110\": container with ID starting with 59c77a4e488404f2c9e79e9464b7575e6698d10051175a5bdebc1b9739a68110 not found: ID does not exist" Oct 30 17:16:06.737583 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.737570 6202 scope.go:122] "RemoveContainer" containerID="71c57426929a633bc47406761c973ffeef018f3bab52ea70bb0befa874a15ebd" Oct 30 17:16:06.738628 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.738607 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk"] Oct 30 17:16:06.743815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.741973 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-de18d5-ndvxk"] Oct 30 17:16:06.751320 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.751302 6202 scope.go:122] "RemoveContainer" containerID="71c57426929a633bc47406761c973ffeef018f3bab52ea70bb0befa874a15ebd" Oct 30 17:16:06.751667 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:16:06.751644 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c57426929a633bc47406761c973ffeef018f3bab52ea70bb0befa874a15ebd\": container with ID starting with 71c57426929a633bc47406761c973ffeef018f3bab52ea70bb0befa874a15ebd not found: ID does not exist" containerID="71c57426929a633bc47406761c973ffeef018f3bab52ea70bb0befa874a15ebd" Oct 30 17:16:06.751741 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.751676 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c57426929a633bc47406761c973ffeef018f3bab52ea70bb0befa874a15ebd"} err="failed to get container status \"71c57426929a633bc47406761c973ffeef018f3bab52ea70bb0befa874a15ebd\": rpc error: code = NotFound desc = could not find container \"71c57426929a633bc47406761c973ffeef018f3bab52ea70bb0befa874a15ebd\": container with ID starting with 71c57426929a633bc47406761c973ffeef018f3bab52ea70bb0befa874a15ebd not found: ID does not exist" Oct 30 17:16:06.761044 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.761024 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:06.781940 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.780617 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c"] Oct 30 17:16:06.842729 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.842693 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de18d5-756bdc77fb-mxc2c"] Oct 30 17:16:06.888172 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.887696 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b"] Oct 30 17:16:06.903806 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:06.900018 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-de18d5-d8984d9d7-txz5b"] Oct 30 17:16:07.265222 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.265180 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:07.385149 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.385100 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:07.571972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.571946 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" Oct 30 17:16:07.580354 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.580331 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" Oct 30 17:16:07.734481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.734448 6202 generic.go:356] "Generic (PLEG): container finished" podID="2dd4faa7-90ff-4815-b7c5-69081a00ed05" containerID="7802d6433d34e774bed4f1d0b9a263d7a00db6863d2314768684bd51f103469a" exitCode=0 Oct 30 17:16:07.734697 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.734509 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" Oct 30 17:16:07.734697 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.734541 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" event={"ID":"2dd4faa7-90ff-4815-b7c5-69081a00ed05","Type":"ContainerDied","Data":"7802d6433d34e774bed4f1d0b9a263d7a00db6863d2314768684bd51f103469a"} Oct 30 17:16:07.734697 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.734614 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml" event={"ID":"2dd4faa7-90ff-4815-b7c5-69081a00ed05","Type":"ContainerDied","Data":"5828f598c33d34c03a019fb6aa13083052fc985a0f38f7e45522ae8fcf4e1990"} Oct 30 17:16:07.734697 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.734650 6202 scope.go:122] "RemoveContainer" containerID="7802d6433d34e774bed4f1d0b9a263d7a00db6863d2314768684bd51f103469a" Oct 30 17:16:07.736057 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.736035 6202 generic.go:356] "Generic (PLEG): container finished" podID="7bec536b-a62b-42d7-810d-fa8090168784" containerID="9e9ea2e407ca284194fde9dd557d968c2c2b224c6018ac0ff8e3815fa1b680fd" exitCode=0 Oct 30 17:16:07.736166 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.736075 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" Oct 30 17:16:07.736166 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.736120 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" event={"ID":"7bec536b-a62b-42d7-810d-fa8090168784","Type":"ContainerDied","Data":"9e9ea2e407ca284194fde9dd557d968c2c2b224c6018ac0ff8e3815fa1b680fd"} Oct 30 17:16:07.736166 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.736160 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg" event={"ID":"7bec536b-a62b-42d7-810d-fa8090168784","Type":"ContainerDied","Data":"18ebe20c5f6e289a8392d85c3552ba88ae34de3ec326c6fe4981806449e4f229"} Oct 30 17:16:07.745367 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.745351 6202 scope.go:122] "RemoveContainer" containerID="7802d6433d34e774bed4f1d0b9a263d7a00db6863d2314768684bd51f103469a" Oct 30 17:16:07.745648 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:16:07.745627 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7802d6433d34e774bed4f1d0b9a263d7a00db6863d2314768684bd51f103469a\": container with ID starting with 7802d6433d34e774bed4f1d0b9a263d7a00db6863d2314768684bd51f103469a not found: ID does not exist" containerID="7802d6433d34e774bed4f1d0b9a263d7a00db6863d2314768684bd51f103469a" Oct 30 17:16:07.745727 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.745658 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7802d6433d34e774bed4f1d0b9a263d7a00db6863d2314768684bd51f103469a"} err="failed to get container status \"7802d6433d34e774bed4f1d0b9a263d7a00db6863d2314768684bd51f103469a\": rpc error: code = NotFound desc = could not find container \"7802d6433d34e774bed4f1d0b9a263d7a00db6863d2314768684bd51f103469a\": container with ID starting with 7802d6433d34e774bed4f1d0b9a263d7a00db6863d2314768684bd51f103469a not found: ID does not exist" Oct 30 17:16:07.745727 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.745676 6202 scope.go:122] "RemoveContainer" containerID="9e9ea2e407ca284194fde9dd557d968c2c2b224c6018ac0ff8e3815fa1b680fd" Oct 30 17:16:07.761374 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.761344 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bec536b-a62b-42d7-810d-fa8090168784-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bec536b-a62b-42d7-810d-fa8090168784-plugins-config-volume\") pod \"7bec536b-a62b-42d7-810d-fa8090168784\" (UID: \"7bec536b-a62b-42d7-810d-fa8090168784\") " Oct 30 17:16:07.761470 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.761426 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bec536b-a62b-42d7-810d-fa8090168784-kube-api-access-pbz9w\" (UniqueName: \"kubernetes.io/projected/7bec536b-a62b-42d7-810d-fa8090168784-kube-api-access-pbz9w\") pod \"7bec536b-a62b-42d7-810d-fa8090168784\" (UID: \"7bec536b-a62b-42d7-810d-fa8090168784\") " Oct 30 17:16:07.761522 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.761475 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd4faa7-90ff-4815-b7c5-69081a00ed05-kube-api-access-8q5xc\" (UniqueName: \"kubernetes.io/projected/2dd4faa7-90ff-4815-b7c5-69081a00ed05-kube-api-access-8q5xc\") pod \"2dd4faa7-90ff-4815-b7c5-69081a00ed05\" (UID: \"2dd4faa7-90ff-4815-b7c5-69081a00ed05\") " Oct 30 17:16:07.761622 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.761602 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd4faa7-90ff-4815-b7c5-69081a00ed05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd4faa7-90ff-4815-b7c5-69081a00ed05-plugins-config-volume\") pod \"2dd4faa7-90ff-4815-b7c5-69081a00ed05\" (UID: \"2dd4faa7-90ff-4815-b7c5-69081a00ed05\") " Oct 30 17:16:07.762654 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.762617 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bec536b-a62b-42d7-810d-fa8090168784-plugins-config-volume" pod "7bec536b-a62b-42d7-810d-fa8090168784" (UID: "7bec536b-a62b-42d7-810d-fa8090168784"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:07.762788 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.762668 6202 scope.go:122] "RemoveContainer" containerID="9e9ea2e407ca284194fde9dd557d968c2c2b224c6018ac0ff8e3815fa1b680fd" Oct 30 17:16:07.762880 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.762848 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd4faa7-90ff-4815-b7c5-69081a00ed05-plugins-config-volume" pod "2dd4faa7-90ff-4815-b7c5-69081a00ed05" (UID: "2dd4faa7-90ff-4815-b7c5-69081a00ed05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:07.763063 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:16:07.763040 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e9ea2e407ca284194fde9dd557d968c2c2b224c6018ac0ff8e3815fa1b680fd\": container with ID starting with 9e9ea2e407ca284194fde9dd557d968c2c2b224c6018ac0ff8e3815fa1b680fd not found: ID does not exist" containerID="9e9ea2e407ca284194fde9dd557d968c2c2b224c6018ac0ff8e3815fa1b680fd" Oct 30 17:16:07.763186 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.763072 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9ea2e407ca284194fde9dd557d968c2c2b224c6018ac0ff8e3815fa1b680fd"} err="failed to get container status \"9e9ea2e407ca284194fde9dd557d968c2c2b224c6018ac0ff8e3815fa1b680fd\": rpc error: code = NotFound desc = could not find container \"9e9ea2e407ca284194fde9dd557d968c2c2b224c6018ac0ff8e3815fa1b680fd\": container with ID starting with 9e9ea2e407ca284194fde9dd557d968c2c2b224c6018ac0ff8e3815fa1b680fd not found: ID does not exist" Oct 30 17:16:07.764316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.764288 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bec536b-a62b-42d7-810d-fa8090168784-kube-api-access-pbz9w" pod "7bec536b-a62b-42d7-810d-fa8090168784" (UID: "7bec536b-a62b-42d7-810d-fa8090168784"). InnerVolumeSpecName "kube-api-access-pbz9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:07.765082 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.765062 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd4faa7-90ff-4815-b7c5-69081a00ed05-kube-api-access-8q5xc" pod "2dd4faa7-90ff-4815-b7c5-69081a00ed05" (UID: "2dd4faa7-90ff-4815-b7c5-69081a00ed05"). InnerVolumeSpecName "kube-api-access-8q5xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:07.863476 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.863436 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd4faa7-90ff-4815-b7c5-69081a00ed05-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:07.863476 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.863472 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bec536b-a62b-42d7-810d-fa8090168784-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:07.864028 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.863487 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbz9w\" (UniqueName: \"kubernetes.io/projected/7bec536b-a62b-42d7-810d-fa8090168784-kube-api-access-pbz9w\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:07.864028 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:07.863500 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q5xc\" (UniqueName: \"kubernetes.io/projected/2dd4faa7-90ff-4815-b7c5-69081a00ed05-kube-api-access-8q5xc\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:08.088255 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:08.088224 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml"] Oct 30 17:16:08.092086 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:08.092059 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-661509-89c676c8b-drzml"] Oct 30 17:16:08.126717 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:08.126685 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg"] Oct 30 17:16:08.132746 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:08.132714 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-661509-7f7bc4c5cf-rvmgg"] Oct 30 17:16:08.411196 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:08.411108 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd4faa7-90ff-4815-b7c5-69081a00ed05" path="/var/lib/kubelet/pods/2dd4faa7-90ff-4815-b7c5-69081a00ed05/volumes" Oct 30 17:16:08.411626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:08.411603 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bec536b-a62b-42d7-810d-fa8090168784" path="/var/lib/kubelet/pods/7bec536b-a62b-42d7-810d-fa8090168784/volumes" Oct 30 17:16:08.412097 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:08.412075 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f2f997e-e6fc-46d2-bb0f-68c947c8607f" path="/var/lib/kubelet/pods/8f2f997e-e6fc-46d2-bb0f-68c947c8607f/volumes" Oct 30 17:16:08.412505 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:08.412484 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adeec1f0-813d-4ade-bab6-b1f8d49a6f28" path="/var/lib/kubelet/pods/adeec1f0-813d-4ade-bab6-b1f8d49a6f28/volumes" Oct 30 17:16:08.413263 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:08.413240 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc285144-2c80-4d79-8269-3994f8858ab3" path="/var/lib/kubelet/pods/cc285144-2c80-4d79-8269-3994f8858ab3/volumes" Oct 30 17:16:08.602542 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:08.602511 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:10.445643 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:10.445609 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:11.296301 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:11.296260 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:11.410699 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:11.410658 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:11.631709 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:11.631678 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4"] Oct 30 17:16:11.632278 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:11.631937 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" podUID="6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" containerName="istio-proxy" containerID="cri-o://485ba0b4fc23776c5ad4f1dbb1f9f6d0e8fb041f8fe211f74c3c4ab9ecb44190" gracePeriod=30 Oct 30 17:16:11.660017 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:11.659986 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:11.956066 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:11.955984 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:12.176863 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:12.176819 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:12.458026 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:12.457977 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:12.570785 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:12.570736 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:12.592951 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:12.592913 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb"] Oct 30 17:16:12.595465 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:12.595443 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c4b84-ngnwb"] Oct 30 17:16:13.037148 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:13.037110 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn"] Oct 30 17:16:13.113102 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:13.113061 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:13.224317 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:13.224288 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:13.525367 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:13.525332 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:16:13.525522 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:13.525395 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" podUID="6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:16:13.817634 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:13.817596 6202 generic.go:356] "Generic (PLEG): container finished" podID="be7ca116-75e7-468a-a943-4113a9fd941b" containerID="6c45adfa6ff6a5b1f69168048a40d02c0fa60b9e3326f8949abc798a0f0a269c" exitCode=0 Oct 30 17:16:13.817958 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:13.817903 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" event={"ID":"be7ca116-75e7-468a-a943-4113a9fd941b","Type":"ContainerDied","Data":"6c45adfa6ff6a5b1f69168048a40d02c0fa60b9e3326f8949abc798a0f0a269c"} Oct 30 17:16:13.819530 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:13.819505 6202 generic.go:356] "Generic (PLEG): container finished" podID="e1cedd9a-5798-4463-a6b5-a8f199716991" containerID="73d775ca11fc54629d826aaf35d3659503995932e95768b116a0c82aa3c8672d" exitCode=0 Oct 30 17:16:13.819645 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:13.819582 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" event={"ID":"e1cedd9a-5798-4463-a6b5-a8f199716991","Type":"ContainerDied","Data":"73d775ca11fc54629d826aaf35d3659503995932e95768b116a0c82aa3c8672d"} Oct 30 17:16:13.997959 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:13.997937 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" Oct 30 17:16:14.004573 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.004551 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" Oct 30 17:16:14.056823 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.056692 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be7ca116-75e7-468a-a943-4113a9fd941b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be7ca116-75e7-468a-a943-4113a9fd941b-plugins-config-volume\") pod \"be7ca116-75e7-468a-a943-4113a9fd941b\" (UID: \"be7ca116-75e7-468a-a943-4113a9fd941b\") " Oct 30 17:16:14.057339 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.056926 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be7ca116-75e7-468a-a943-4113a9fd941b-kube-api-access-xkhpz\" (UniqueName: \"kubernetes.io/projected/be7ca116-75e7-468a-a943-4113a9fd941b-kube-api-access-xkhpz\") pod \"be7ca116-75e7-468a-a943-4113a9fd941b\" (UID: \"be7ca116-75e7-468a-a943-4113a9fd941b\") " Oct 30 17:16:14.057339 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.056978 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1cedd9a-5798-4463-a6b5-a8f199716991-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1cedd9a-5798-4463-a6b5-a8f199716991-plugins-config-volume\") pod \"e1cedd9a-5798-4463-a6b5-a8f199716991\" (UID: \"e1cedd9a-5798-4463-a6b5-a8f199716991\") " Oct 30 17:16:14.057339 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.057106 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1cedd9a-5798-4463-a6b5-a8f199716991-kube-api-access-rwhlh\" (UniqueName: \"kubernetes.io/projected/e1cedd9a-5798-4463-a6b5-a8f199716991-kube-api-access-rwhlh\") pod \"e1cedd9a-5798-4463-a6b5-a8f199716991\" (UID: \"e1cedd9a-5798-4463-a6b5-a8f199716991\") " Oct 30 17:16:14.057339 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.057142 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7ca116-75e7-468a-a943-4113a9fd941b-plugins-config-volume" pod "be7ca116-75e7-468a-a943-4113a9fd941b" (UID: "be7ca116-75e7-468a-a943-4113a9fd941b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:14.057708 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.057655 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be7ca116-75e7-468a-a943-4113a9fd941b-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:14.058695 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.058403 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1cedd9a-5798-4463-a6b5-a8f199716991-plugins-config-volume" pod "e1cedd9a-5798-4463-a6b5-a8f199716991" (UID: "e1cedd9a-5798-4463-a6b5-a8f199716991"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:14.060096 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.060060 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7ca116-75e7-468a-a943-4113a9fd941b-kube-api-access-xkhpz" pod "be7ca116-75e7-468a-a943-4113a9fd941b" (UID: "be7ca116-75e7-468a-a943-4113a9fd941b"). InnerVolumeSpecName "kube-api-access-xkhpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:14.060667 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.060640 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1cedd9a-5798-4463-a6b5-a8f199716991-kube-api-access-rwhlh" pod "e1cedd9a-5798-4463-a6b5-a8f199716991" (UID: "e1cedd9a-5798-4463-a6b5-a8f199716991"). InnerVolumeSpecName "kube-api-access-rwhlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:14.076035 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:16:14.075997 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:16:14.158640 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.158599 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwhlh\" (UniqueName: \"kubernetes.io/projected/e1cedd9a-5798-4463-a6b5-a8f199716991-kube-api-access-rwhlh\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:14.158640 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.158638 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkhpz\" (UniqueName: \"kubernetes.io/projected/be7ca116-75e7-468a-a943-4113a9fd941b-kube-api-access-xkhpz\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:14.158893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.158653 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1cedd9a-5798-4463-a6b5-a8f199716991-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:14.410840 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.410805 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfeb39d1-2907-41de-afaa-8c3b0690a828" path="/var/lib/kubelet/pods/cfeb39d1-2907-41de-afaa-8c3b0690a828/volumes" Oct 30 17:16:14.458805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.458759 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:14.583228 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.583196 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:14.829011 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.828973 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" Oct 30 17:16:14.829011 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.828995 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b" event={"ID":"e1cedd9a-5798-4463-a6b5-a8f199716991","Type":"ContainerDied","Data":"0629e1cf14918eaf36a0e58b492f95fcd5e5d1d6f32ff55b8be996a4b52251f7"} Oct 30 17:16:14.829262 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.829065 6202 scope.go:122] "RemoveContainer" containerID="73d775ca11fc54629d826aaf35d3659503995932e95768b116a0c82aa3c8672d" Oct 30 17:16:14.839055 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.839018 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" event={"ID":"be7ca116-75e7-468a-a943-4113a9fd941b","Type":"ContainerDied","Data":"05f146434887754822783f02bcae8489b9cf40325363a96bba7e76a978e1cbfb"} Oct 30 17:16:14.839283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.839245 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk" Oct 30 17:16:14.850008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.849955 6202 scope.go:122] "RemoveContainer" containerID="6c45adfa6ff6a5b1f69168048a40d02c0fa60b9e3326f8949abc798a0f0a269c" Oct 30 17:16:14.850008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.849965 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:14.903941 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.892653 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b"] Oct 30 17:16:14.912792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.911962 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7c4b84-5b54fc57c8-2hs5b"] Oct 30 17:16:14.956340 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.948221 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk"] Oct 30 17:16:14.956340 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:14.948278 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7c4b84-57dbc4b9c9-sb6fk"] Oct 30 17:16:15.337490 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.337454 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7"] Oct 30 17:16:15.338227 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338207 6202 cpu_manager.go:414] "containerMap: removing container" podUID="2dd4faa7-90ff-4815-b7c5-69081a00ed05" containerName="payload-processing" Oct 30 17:16:15.338275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338229 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd4faa7-90ff-4815-b7c5-69081a00ed05" containerName="payload-processing" Oct 30 17:16:15.338275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338249 6202 cpu_manager.go:414] "containerMap: removing container" podUID="be7ca116-75e7-468a-a943-4113a9fd941b" containerName="payload-pre-processing" Oct 30 17:16:15.338275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338258 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7ca116-75e7-468a-a943-4113a9fd941b" containerName="payload-pre-processing" Oct 30 17:16:15.338390 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338286 6202 cpu_manager.go:414] "containerMap: removing container" podUID="adeec1f0-813d-4ade-bab6-b1f8d49a6f28" containerName="payload-processing" Oct 30 17:16:15.338390 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338295 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adeec1f0-813d-4ade-bab6-b1f8d49a6f28" containerName="payload-processing" Oct 30 17:16:15.338390 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338321 6202 cpu_manager.go:414] "containerMap: removing container" podUID="7bec536b-a62b-42d7-810d-fa8090168784" containerName="payload-pre-processing" Oct 30 17:16:15.338390 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338328 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bec536b-a62b-42d7-810d-fa8090168784" containerName="payload-pre-processing" Oct 30 17:16:15.338390 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338341 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8f2f997e-e6fc-46d2-bb0f-68c947c8607f" containerName="payload-pre-processing" Oct 30 17:16:15.338390 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338348 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f2f997e-e6fc-46d2-bb0f-68c947c8607f" containerName="payload-pre-processing" Oct 30 17:16:15.338390 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338373 6202 cpu_manager.go:414] "containerMap: removing container" podUID="e1cedd9a-5798-4463-a6b5-a8f199716991" containerName="payload-processing" Oct 30 17:16:15.338390 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338380 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1cedd9a-5798-4463-a6b5-a8f199716991" containerName="payload-processing" Oct 30 17:16:15.338390 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338390 6202 cpu_manager.go:414] "containerMap: removing container" podUID="1ea89c83-ccfe-448f-83f2-afef35ef058b" containerName="revoke-keys" Oct 30 17:16:15.338668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338397 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea89c83-ccfe-448f-83f2-afef35ef058b" containerName="revoke-keys" Oct 30 17:16:15.338668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338512 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="adeec1f0-813d-4ade-bab6-b1f8d49a6f28" containerName="payload-processing" Oct 30 17:16:15.338668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338527 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bec536b-a62b-42d7-810d-fa8090168784" containerName="payload-pre-processing" Oct 30 17:16:15.338668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338539 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f2f997e-e6fc-46d2-bb0f-68c947c8607f" containerName="payload-pre-processing" Oct 30 17:16:15.338668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338550 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1cedd9a-5798-4463-a6b5-a8f199716991" containerName="payload-processing" Oct 30 17:16:15.338668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338559 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea89c83-ccfe-448f-83f2-afef35ef058b" containerName="revoke-keys" Oct 30 17:16:15.338668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338573 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd4faa7-90ff-4815-b7c5-69081a00ed05" containerName="payload-processing" Oct 30 17:16:15.338668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.338581 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7ca116-75e7-468a-a943-4113a9fd941b" containerName="payload-pre-processing" Oct 30 17:16:15.343785 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.343750 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7" Oct 30 17:16:15.360750 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.360727 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:16:15.360992 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.360969 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:16:15.362100 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.362076 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7"] Oct 30 17:16:15.379796 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.379754 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5668b79a-d001-423a-a0bb-6ddb3ae4036f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7\" (UID: \"5668b79a-d001-423a-a0bb-6ddb3ae4036f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7" Oct 30 17:16:15.379949 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.379825 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5668b79a-d001-423a-a0bb-6ddb3ae4036f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7\" (UID: \"5668b79a-d001-423a-a0bb-6ddb3ae4036f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7" Oct 30 17:16:15.379949 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.379931 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk4fd\" (UniqueName: \"kubernetes.io/projected/5668b79a-d001-423a-a0bb-6ddb3ae4036f-kube-api-access-rk4fd\") pod \"maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7\" (UID: \"5668b79a-d001-423a-a0bb-6ddb3ae4036f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7" Oct 30 17:16:15.440911 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.440880 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:16:15.441069 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.440920 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:16:15.481619 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.481590 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5668b79a-d001-423a-a0bb-6ddb3ae4036f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7\" (UID: \"5668b79a-d001-423a-a0bb-6ddb3ae4036f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7" Oct 30 17:16:15.481849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.481634 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5668b79a-d001-423a-a0bb-6ddb3ae4036f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7\" (UID: \"5668b79a-d001-423a-a0bb-6ddb3ae4036f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7" Oct 30 17:16:15.481849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.481658 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk4fd\" (UniqueName: \"kubernetes.io/projected/5668b79a-d001-423a-a0bb-6ddb3ae4036f-kube-api-access-rk4fd\") pod \"maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7\" (UID: \"5668b79a-d001-423a-a0bb-6ddb3ae4036f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7" Oct 30 17:16:15.482367 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.482346 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5668b79a-d001-423a-a0bb-6ddb3ae4036f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7\" (UID: \"5668b79a-d001-423a-a0bb-6ddb3ae4036f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7" Oct 30 17:16:15.484084 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.484064 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5668b79a-d001-423a-a0bb-6ddb3ae4036f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7\" (UID: \"5668b79a-d001-423a-a0bb-6ddb3ae4036f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7" Oct 30 17:16:15.506461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.506430 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk4fd\" (UniqueName: \"kubernetes.io/projected/5668b79a-d001-423a-a0bb-6ddb3ae4036f-kube-api-access-rk4fd\") pod \"maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7\" (UID: \"5668b79a-d001-423a-a0bb-6ddb3ae4036f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7" Oct 30 17:16:15.657878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.657795 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7" Oct 30 17:16:15.963254 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:15.963218 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7"] Oct 30 17:16:15.963780 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:16:15.963740 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5668b79a_d001_423a_a0bb_6ddb3ae4036f.slice/crio-9e818de799149c63b6bfeecbd37ecb70b6c884d077c98001b035ec6872ecb27e WatchSource:0}: Error finding container 9e818de799149c63b6bfeecbd37ecb70b6c884d077c98001b035ec6872ecb27e: Status 404 returned error can't find the container with id 9e818de799149c63b6bfeecbd37ecb70b6c884d077c98001b035ec6872ecb27e Oct 30 17:16:16.411208 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:16.411176 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be7ca116-75e7-468a-a943-4113a9fd941b" path="/var/lib/kubelet/pods/be7ca116-75e7-468a-a943-4113a9fd941b/volumes" Oct 30 17:16:16.411607 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:16.411549 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1cedd9a-5798-4463-a6b5-a8f199716991" path="/var/lib/kubelet/pods/e1cedd9a-5798-4463-a6b5-a8f199716991/volumes" Oct 30 17:16:16.510598 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:16.510527 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:16.880119 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:16.865066 6202 generic.go:356] "Generic (PLEG): container finished" podID="6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" containerID="485ba0b4fc23776c5ad4f1dbb1f9f6d0e8fb041f8fe211f74c3c4ab9ecb44190" exitCode=0 Oct 30 17:16:16.880119 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:16.865142 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" event={"ID":"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd","Type":"ContainerDied","Data":"485ba0b4fc23776c5ad4f1dbb1f9f6d0e8fb041f8fe211f74c3c4ab9ecb44190"} Oct 30 17:16:16.880119 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:16.865219 6202 scope.go:122] "RemoveContainer" containerID="7586261b741c10e7484cf3219c709e5b65080b977caf1ca27e51b0f2cdf1e0c9" Oct 30 17:16:16.884523 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:16.884478 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7" event={"ID":"5668b79a-d001-423a-a0bb-6ddb3ae4036f","Type":"ContainerStarted","Data":"f4f34b03deb10988c817c6c71da7528283c7f238c91fd666915298ba4850aea7"} Oct 30 17:16:16.884707 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:16.884542 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7" event={"ID":"5668b79a-d001-423a-a0bb-6ddb3ae4036f","Type":"ContainerStarted","Data":"9e818de799149c63b6bfeecbd37ecb70b6c884d077c98001b035ec6872ecb27e"} Oct 30 17:16:16.928488 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:16.928440 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7" podStartSLOduration=1.928425372 podStartE2EDuration="1.928425372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:16.927653065 +0000 UTC m=+3290.638107810" watchObservedRunningTime="2026-10-30 17:16:16.928425372 +0000 UTC m=+3290.638880115" Oct 30 17:16:16.961975 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:16.961943 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 17:16:16.999906 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:16.999874 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-workload-socket\") pod \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " Oct 30 17:16:17.000073 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:16.999937 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-workload-certs\") pod \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " Oct 30 17:16:17.000073 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:16.999965 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-credential-socket\") pod \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " Oct 30 17:16:17.000073 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:16.999995 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-token\" (UniqueName: \"kubernetes.io/projected/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-token\") pod \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " Oct 30 17:16:17.000073 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.000032 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-kube-api-access-vrwww\" (UniqueName: \"kubernetes.io/projected/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-kube-api-access-vrwww\") pod \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " Oct 30 17:16:17.000306 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.000082 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-podinfo\") pod \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " Oct 30 17:16:17.000306 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.000183 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-envoy\") pod \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " Oct 30 17:16:17.000436 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.000178 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-workload-socket" pod "6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" (UID: "6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:17.000436 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.000316 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istiod-ca-cert\") pod \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " Oct 30 17:16:17.000436 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.000295 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-workload-certs" pod "6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" (UID: "6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:17.000436 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.000319 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-credential-socket" pod "6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" (UID: "6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:17.000436 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.000390 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-data\") pod \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\" (UID: \"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd\") " Oct 30 17:16:17.000723 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.000641 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-data" pod "6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" (UID: "6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:17.000723 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.000665 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istiod-ca-cert" pod "6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" (UID: "6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:17.001363 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.001206 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:17.001363 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.001241 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:17.001363 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.001255 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:17.001363 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.001264 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:17.001363 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.001276 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:17.002598 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.002574 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-podinfo" pod "6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" (UID: "6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:16:17.002719 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.002604 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-envoy" pod "6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" (UID: "6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:17.002819 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.002745 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-kube-api-access-vrwww" pod "6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" (UID: "6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd"). InnerVolumeSpecName "kube-api-access-vrwww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:17.003140 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.003123 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-token" pod "6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" (UID: "6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:17.102358 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.102320 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:17.102358 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.102357 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrwww\" (UniqueName: \"kubernetes.io/projected/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-kube-api-access-vrwww\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:17.102596 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.102371 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:17.102596 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.102386 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:17.131257 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.131226 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq"] Oct 30 17:16:17.135831 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.135798 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97"] Oct 30 17:16:17.142797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.141814 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94"] Oct 30 17:16:17.266038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.265623 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:17.894054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.894020 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" Oct 30 17:16:17.894054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.894042 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4" event={"ID":"6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd","Type":"ContainerDied","Data":"2cd62faf605496ce2e18b6972aaa9107f90d716eeadff36f5434fa3abde3a7ef"} Oct 30 17:16:17.894524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.894105 6202 scope.go:122] "RemoveContainer" containerID="485ba0b4fc23776c5ad4f1dbb1f9f6d0e8fb041f8fe211f74c3c4ab9ecb44190" Oct 30 17:16:17.896192 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.896161 6202 generic.go:356] "Generic (PLEG): container finished" podID="5668b79a-d001-423a-a0bb-6ddb3ae4036f" containerID="f4f34b03deb10988c817c6c71da7528283c7f238c91fd666915298ba4850aea7" exitCode=0 Oct 30 17:16:17.896303 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.896209 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7" event={"ID":"5668b79a-d001-423a-a0bb-6ddb3ae4036f","Type":"ContainerDied","Data":"f4f34b03deb10988c817c6c71da7528283c7f238c91fd666915298ba4850aea7"} Oct 30 17:16:17.994776 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:17.994719 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4"] Oct 30 17:16:18.009159 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:18.009125 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-661509-openshift-default-b5d9994f-z2kd4"] Oct 30 17:16:18.037951 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:18.037909 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" podUID="71a9b980-5908-4440-b609-20cddc8c2327" containerName="payload-pre-processing" containerID="cri-o://f41e2acabaaef5dc30f711a57be593767e8cc5992c6323b04a73deca2f41274c" gracePeriod=25 Oct 30 17:16:18.411116 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:18.411082 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" path="/var/lib/kubelet/pods/6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd/volumes" Oct 30 17:16:18.620618 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:18.620577 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" podUID="71a9b980-5908-4440-b609-20cddc8c2327" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:18.627350 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:18.627322 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:18.750883 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:18.750791 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:19.139719 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:19.139689 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:19.149370 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:19.149340 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7" Oct 30 17:16:19.233601 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:19.233571 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5668b79a-d001-423a-a0bb-6ddb3ae4036f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5668b79a-d001-423a-a0bb-6ddb3ae4036f-maas-api-openshift-ca\") pod \"5668b79a-d001-423a-a0bb-6ddb3ae4036f\" (UID: \"5668b79a-d001-423a-a0bb-6ddb3ae4036f\") " Oct 30 17:16:19.233787 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:19.233734 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5668b79a-d001-423a-a0bb-6ddb3ae4036f-kube-api-access-rk4fd\" (UniqueName: \"kubernetes.io/projected/5668b79a-d001-423a-a0bb-6ddb3ae4036f-kube-api-access-rk4fd\") pod \"5668b79a-d001-423a-a0bb-6ddb3ae4036f\" (UID: \"5668b79a-d001-423a-a0bb-6ddb3ae4036f\") " Oct 30 17:16:19.233837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:19.233817 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5668b79a-d001-423a-a0bb-6ddb3ae4036f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5668b79a-d001-423a-a0bb-6ddb3ae4036f-maas-api-xks-ca\") pod \"5668b79a-d001-423a-a0bb-6ddb3ae4036f\" (UID: \"5668b79a-d001-423a-a0bb-6ddb3ae4036f\") " Oct 30 17:16:19.233975 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:19.233949 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5668b79a-d001-423a-a0bb-6ddb3ae4036f-maas-api-openshift-ca" pod "5668b79a-d001-423a-a0bb-6ddb3ae4036f" (UID: "5668b79a-d001-423a-a0bb-6ddb3ae4036f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:19.234176 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:19.234162 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5668b79a-d001-423a-a0bb-6ddb3ae4036f-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:19.236010 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:19.235985 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5668b79a-d001-423a-a0bb-6ddb3ae4036f-kube-api-access-rk4fd" pod "5668b79a-d001-423a-a0bb-6ddb3ae4036f" (UID: "5668b79a-d001-423a-a0bb-6ddb3ae4036f"). InnerVolumeSpecName "kube-api-access-rk4fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:19.236010 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:19.236001 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5668b79a-d001-423a-a0bb-6ddb3ae4036f-maas-api-xks-ca" pod "5668b79a-d001-423a-a0bb-6ddb3ae4036f" (UID: "5668b79a-d001-423a-a0bb-6ddb3ae4036f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:19.254610 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:19.254583 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:19.301244 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:19.301205 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p"] Oct 30 17:16:19.305415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:19.305378 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05810b-nkh2p"] Oct 30 17:16:19.335396 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:19.335352 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk4fd\" (UniqueName: \"kubernetes.io/projected/5668b79a-d001-423a-a0bb-6ddb3ae4036f-kube-api-access-rk4fd\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:19.335396 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:19.335392 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5668b79a-d001-423a-a0bb-6ddb3ae4036f-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:19.913104 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:19.913070 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7" Oct 30 17:16:19.913104 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:19.913090 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7" event={"ID":"5668b79a-d001-423a-a0bb-6ddb3ae4036f","Type":"ContainerDied","Data":"9e818de799149c63b6bfeecbd37ecb70b6c884d077c98001b035ec6872ecb27e"} Oct 30 17:16:19.913380 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:19.913136 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e818de799149c63b6bfeecbd37ecb70b6c884d077c98001b035ec6872ecb27e" Oct 30 17:16:20.410492 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:20.410457 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea89c83-ccfe-448f-83f2-afef35ef058b" path="/var/lib/kubelet/pods/1ea89c83-ccfe-448f-83f2-afef35ef058b/volumes" Oct 30 17:16:20.601186 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:20.601149 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:20.720377 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:20.720296 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:20.912488 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:20.912439 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g"] Oct 30 17:16:20.913020 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:20.912822 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" podUID="9720d030-7806-479f-840c-38401180d116" containerName="istio-proxy" containerID="cri-o://2f0baa4e01f05c6df1e9d110478272b8f49779d7f8d84d7741db5676101d1014" gracePeriod=30 Oct 30 17:16:20.970189 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:20.970150 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:21.118789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.110359 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:21.477878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.477753 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:21.600494 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.600458 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd"] Oct 30 17:16:21.601092 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.601077 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5668b79a-d001-423a-a0bb-6ddb3ae4036f" containerName="revoke-keys" Oct 30 17:16:21.601151 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.601094 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5668b79a-d001-423a-a0bb-6ddb3ae4036f" containerName="revoke-keys" Oct 30 17:16:21.601151 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.601109 6202 cpu_manager.go:414] "containerMap: removing container" podUID="6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" containerName="istio-proxy" Oct 30 17:16:21.601151 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.601115 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" containerName="istio-proxy" Oct 30 17:16:21.601151 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.601126 6202 cpu_manager.go:414] "containerMap: removing container" podUID="6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" containerName="istio-proxy" Oct 30 17:16:21.601151 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.601130 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" containerName="istio-proxy" Oct 30 17:16:21.601322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.601219 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5668b79a-d001-423a-a0bb-6ddb3ae4036f" containerName="revoke-keys" Oct 30 17:16:21.601322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.601231 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" containerName="istio-proxy" Oct 30 17:16:21.601322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.601237 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb9aafb-a8f2-4a76-acc6-1769c0b2acdd" containerName="istio-proxy" Oct 30 17:16:21.605111 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.605094 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd" Oct 30 17:16:21.622039 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.622013 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:16:21.622193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.622155 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:16:21.626880 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.626852 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd"] Oct 30 17:16:21.659565 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.659528 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:16:21.665896 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.665862 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:16:21.688810 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.688759 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ef8a93-8965-43f9-a86e-f2101beef171-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd\" (UID: \"75ef8a93-8965-43f9-a86e-f2101beef171\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd" Oct 30 17:16:21.688935 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.688880 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ef8a93-8965-43f9-a86e-f2101beef171-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd\" (UID: \"75ef8a93-8965-43f9-a86e-f2101beef171\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd" Oct 30 17:16:21.689008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.688947 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvg7b\" (UniqueName: \"kubernetes.io/projected/75ef8a93-8965-43f9-a86e-f2101beef171-kube-api-access-nvg7b\") pod \"maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd\" (UID: \"75ef8a93-8965-43f9-a86e-f2101beef171\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd" Oct 30 17:16:21.790181 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.790092 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ef8a93-8965-43f9-a86e-f2101beef171-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd\" (UID: \"75ef8a93-8965-43f9-a86e-f2101beef171\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd" Oct 30 17:16:21.790181 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.790161 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ef8a93-8965-43f9-a86e-f2101beef171-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd\" (UID: \"75ef8a93-8965-43f9-a86e-f2101beef171\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd" Oct 30 17:16:21.790392 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.790231 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvg7b\" (UniqueName: \"kubernetes.io/projected/75ef8a93-8965-43f9-a86e-f2101beef171-kube-api-access-nvg7b\") pod \"maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd\" (UID: \"75ef8a93-8965-43f9-a86e-f2101beef171\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd" Oct 30 17:16:21.790849 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.790820 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ef8a93-8965-43f9-a86e-f2101beef171-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd\" (UID: \"75ef8a93-8965-43f9-a86e-f2101beef171\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd" Oct 30 17:16:21.792962 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.792934 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ef8a93-8965-43f9-a86e-f2101beef171-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd\" (UID: \"75ef8a93-8965-43f9-a86e-f2101beef171\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd" Oct 30 17:16:21.815966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.815934 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvg7b\" (UniqueName: \"kubernetes.io/projected/75ef8a93-8965-43f9-a86e-f2101beef171-kube-api-access-nvg7b\") pod \"maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd\" (UID: \"75ef8a93-8965-43f9-a86e-f2101beef171\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd" Oct 30 17:16:21.923788 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:21.922982 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd" Oct 30 17:16:22.267533 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:16:22.267501 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ef8a93_8965_43f9_a86e_f2101beef171.slice/crio-f3673d4ce91226ab1547d51f21574a360788582d2f14d782bb4ba0f0e323118f WatchSource:0}: Error finding container f3673d4ce91226ab1547d51f21574a360788582d2f14d782bb4ba0f0e323118f: Status 404 returned error can't find the container with id f3673d4ce91226ab1547d51f21574a360788582d2f14d782bb4ba0f0e323118f Oct 30 17:16:22.268211 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:22.268185 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd"] Oct 30 17:16:22.360062 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:22.360028 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g"] Oct 30 17:16:22.366581 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:22.366555 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g" Oct 30 17:16:22.384905 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:22.384878 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g"] Oct 30 17:16:22.498201 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:22.498172 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b79b467-33a3-45f3-8523-a2eda2ef2bca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g\" (UID: \"3b79b467-33a3-45f3-8523-a2eda2ef2bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g" Oct 30 17:16:22.498595 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:22.498219 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbc9g\" (UniqueName: \"kubernetes.io/projected/3b79b467-33a3-45f3-8523-a2eda2ef2bca-kube-api-access-xbc9g\") pod \"maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g\" (UID: \"3b79b467-33a3-45f3-8523-a2eda2ef2bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g" Oct 30 17:16:22.498595 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:22.498422 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b79b467-33a3-45f3-8523-a2eda2ef2bca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g\" (UID: \"3b79b467-33a3-45f3-8523-a2eda2ef2bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g" Oct 30 17:16:22.599778 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:22.599743 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b79b467-33a3-45f3-8523-a2eda2ef2bca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g\" (UID: \"3b79b467-33a3-45f3-8523-a2eda2ef2bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g" Oct 30 17:16:22.599946 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:22.599927 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b79b467-33a3-45f3-8523-a2eda2ef2bca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g\" (UID: \"3b79b467-33a3-45f3-8523-a2eda2ef2bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g" Oct 30 17:16:22.600017 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:22.599963 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbc9g\" (UniqueName: \"kubernetes.io/projected/3b79b467-33a3-45f3-8523-a2eda2ef2bca-kube-api-access-xbc9g\") pod \"maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g\" (UID: \"3b79b467-33a3-45f3-8523-a2eda2ef2bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g" Oct 30 17:16:22.600531 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:22.600501 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b79b467-33a3-45f3-8523-a2eda2ef2bca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g\" (UID: \"3b79b467-33a3-45f3-8523-a2eda2ef2bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g" Oct 30 17:16:22.602352 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:22.602332 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b79b467-33a3-45f3-8523-a2eda2ef2bca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g\" (UID: \"3b79b467-33a3-45f3-8523-a2eda2ef2bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g" Oct 30 17:16:22.624216 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:22.624195 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbc9g\" (UniqueName: \"kubernetes.io/projected/3b79b467-33a3-45f3-8523-a2eda2ef2bca-kube-api-access-xbc9g\") pod \"maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g\" (UID: \"3b79b467-33a3-45f3-8523-a2eda2ef2bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g" Oct 30 17:16:22.681779 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:22.681748 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g" Oct 30 17:16:22.955238 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:22.955195 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd" event={"ID":"75ef8a93-8965-43f9-a86e-f2101beef171","Type":"ContainerStarted","Data":"73ba203fb90fcf1efff3f89accc1127be6470fd992675bc5e1f838a45e2c9532"} Oct 30 17:16:22.955447 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:22.955295 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd" event={"ID":"75ef8a93-8965-43f9-a86e-f2101beef171","Type":"ContainerStarted","Data":"f3673d4ce91226ab1547d51f21574a360788582d2f14d782bb4ba0f0e323118f"} Oct 30 17:16:22.991151 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:16:22.991116 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b79b467_33a3_45f3_8523_a2eda2ef2bca.slice/crio-9930a79844c4370810a4d573d7732b09eb5a4722cc580130ed5fac322f2f6ee6 WatchSource:0}: Error finding container 9930a79844c4370810a4d573d7732b09eb5a4722cc580130ed5fac322f2f6ee6: Status 404 returned error can't find the container with id 9930a79844c4370810a4d573d7732b09eb5a4722cc580130ed5fac322f2f6ee6 Oct 30 17:16:22.993196 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:22.993170 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g"] Oct 30 17:16:23.006253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:23.006199 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd" podStartSLOduration=2.006181809 podStartE2EDuration="2.006181809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:22.996373951 +0000 UTC m=+3296.706828695" watchObservedRunningTime="2026-10-30 17:16:23.006181809 +0000 UTC m=+3296.716636554" Oct 30 17:16:23.173337 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:23.173301 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:23.966074 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:23.966025 6202 generic.go:356] "Generic (PLEG): container finished" podID="75ef8a93-8965-43f9-a86e-f2101beef171" containerID="73ba203fb90fcf1efff3f89accc1127be6470fd992675bc5e1f838a45e2c9532" exitCode=0 Oct 30 17:16:23.966566 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:23.966156 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd" event={"ID":"75ef8a93-8965-43f9-a86e-f2101beef171","Type":"ContainerDied","Data":"73ba203fb90fcf1efff3f89accc1127be6470fd992675bc5e1f838a45e2c9532"} Oct 30 17:16:23.968392 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:23.968368 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g" event={"ID":"3b79b467-33a3-45f3-8523-a2eda2ef2bca","Type":"ContainerStarted","Data":"b8b5416a466ab3c2f9f260e36f6baf1b36d05da1a679f87a60c518513e9be9f6"} Oct 30 17:16:23.968475 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:23.968417 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g" event={"ID":"3b79b467-33a3-45f3-8523-a2eda2ef2bca","Type":"ContainerStarted","Data":"9930a79844c4370810a4d573d7732b09eb5a4722cc580130ed5fac322f2f6ee6"} Oct 30 17:16:24.058391 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:24.058332 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g" podStartSLOduration=2.058313401 podStartE2EDuration="2.058313401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:24.053935707 +0000 UTC m=+3297.764390452" watchObservedRunningTime="2026-10-30 17:16:24.058313401 +0000 UTC m=+3297.768768143" Oct 30 17:16:24.743113 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:24.743069 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:25.024642 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:25.022204 6202 generic.go:356] "Generic (PLEG): container finished" podID="3b79b467-33a3-45f3-8523-a2eda2ef2bca" containerID="b8b5416a466ab3c2f9f260e36f6baf1b36d05da1a679f87a60c518513e9be9f6" exitCode=0 Oct 30 17:16:25.024642 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:25.022799 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g" event={"ID":"3b79b467-33a3-45f3-8523-a2eda2ef2bca","Type":"ContainerDied","Data":"b8b5416a466ab3c2f9f260e36f6baf1b36d05da1a679f87a60c518513e9be9f6"} Oct 30 17:16:25.273178 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:25.273153 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd" Oct 30 17:16:25.309187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:25.309122 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ef8a93-8965-43f9-a86e-f2101beef171-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ef8a93-8965-43f9-a86e-f2101beef171-maas-api-openshift-ca\") pod \"75ef8a93-8965-43f9-a86e-f2101beef171\" (UID: \"75ef8a93-8965-43f9-a86e-f2101beef171\") " Oct 30 17:16:25.309338 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:25.309184 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ef8a93-8965-43f9-a86e-f2101beef171-kube-api-access-nvg7b\" (UniqueName: \"kubernetes.io/projected/75ef8a93-8965-43f9-a86e-f2101beef171-kube-api-access-nvg7b\") pod \"75ef8a93-8965-43f9-a86e-f2101beef171\" (UID: \"75ef8a93-8965-43f9-a86e-f2101beef171\") " Oct 30 17:16:25.309338 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:25.309328 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75ef8a93-8965-43f9-a86e-f2101beef171-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ef8a93-8965-43f9-a86e-f2101beef171-maas-api-xks-ca\") pod \"75ef8a93-8965-43f9-a86e-f2101beef171\" (UID: \"75ef8a93-8965-43f9-a86e-f2101beef171\") " Oct 30 17:16:25.309506 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:25.309479 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ef8a93-8965-43f9-a86e-f2101beef171-maas-api-openshift-ca" pod "75ef8a93-8965-43f9-a86e-f2101beef171" (UID: "75ef8a93-8965-43f9-a86e-f2101beef171"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:25.309733 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:25.309716 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ef8a93-8965-43f9-a86e-f2101beef171-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:25.311615 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:25.311590 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ef8a93-8965-43f9-a86e-f2101beef171-kube-api-access-nvg7b" pod "75ef8a93-8965-43f9-a86e-f2101beef171" (UID: "75ef8a93-8965-43f9-a86e-f2101beef171"). InnerVolumeSpecName "kube-api-access-nvg7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:25.311720 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:25.311629 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ef8a93-8965-43f9-a86e-f2101beef171-maas-api-xks-ca" pod "75ef8a93-8965-43f9-a86e-f2101beef171" (UID: "75ef8a93-8965-43f9-a86e-f2101beef171"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:25.324948 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:25.324666 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:25.410715 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:25.410680 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ef8a93-8965-43f9-a86e-f2101beef171-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:25.410715 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:25.410715 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvg7b\" (UniqueName: \"kubernetes.io/projected/75ef8a93-8965-43f9-a86e-f2101beef171-kube-api-access-nvg7b\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:25.435760 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:25.435724 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:25.460809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:25.458666 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp"] Oct 30 17:16:25.461408 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:25.461384 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b4214a-5vrfp"] Oct 30 17:16:25.551663 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:25.551624 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:25.971029 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:25.971000 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:26.031157 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.031128 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd" Oct 30 17:16:26.031624 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.031148 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd" event={"ID":"75ef8a93-8965-43f9-a86e-f2101beef171","Type":"ContainerDied","Data":"f3673d4ce91226ab1547d51f21574a360788582d2f14d782bb4ba0f0e323118f"} Oct 30 17:16:26.031624 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.031208 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3673d4ce91226ab1547d51f21574a360788582d2f14d782bb4ba0f0e323118f" Oct 30 17:16:26.208803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.208752 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g" Oct 30 17:16:26.220955 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.220929 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b79b467-33a3-45f3-8523-a2eda2ef2bca-kube-api-access-xbc9g\" (UniqueName: \"kubernetes.io/projected/3b79b467-33a3-45f3-8523-a2eda2ef2bca-kube-api-access-xbc9g\") pod \"3b79b467-33a3-45f3-8523-a2eda2ef2bca\" (UID: \"3b79b467-33a3-45f3-8523-a2eda2ef2bca\") " Oct 30 17:16:26.221184 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.221166 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b79b467-33a3-45f3-8523-a2eda2ef2bca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b79b467-33a3-45f3-8523-a2eda2ef2bca-maas-api-openshift-ca\") pod \"3b79b467-33a3-45f3-8523-a2eda2ef2bca\" (UID: \"3b79b467-33a3-45f3-8523-a2eda2ef2bca\") " Oct 30 17:16:26.221282 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.221266 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b79b467-33a3-45f3-8523-a2eda2ef2bca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b79b467-33a3-45f3-8523-a2eda2ef2bca-maas-api-xks-ca\") pod \"3b79b467-33a3-45f3-8523-a2eda2ef2bca\" (UID: \"3b79b467-33a3-45f3-8523-a2eda2ef2bca\") " Oct 30 17:16:26.221609 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.221584 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b79b467-33a3-45f3-8523-a2eda2ef2bca-maas-api-openshift-ca" pod "3b79b467-33a3-45f3-8523-a2eda2ef2bca" (UID: "3b79b467-33a3-45f3-8523-a2eda2ef2bca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:26.221745 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.221718 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b79b467-33a3-45f3-8523-a2eda2ef2bca-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.223842 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.223813 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b79b467-33a3-45f3-8523-a2eda2ef2bca-maas-api-xks-ca" pod "3b79b467-33a3-45f3-8523-a2eda2ef2bca" (UID: "3b79b467-33a3-45f3-8523-a2eda2ef2bca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:26.224035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.224014 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b79b467-33a3-45f3-8523-a2eda2ef2bca-kube-api-access-xbc9g" pod "3b79b467-33a3-45f3-8523-a2eda2ef2bca" (UID: "3b79b467-33a3-45f3-8523-a2eda2ef2bca"). InnerVolumeSpecName "kube-api-access-xbc9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:26.234249 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.234229 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 17:16:26.322401 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.322357 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-credential-socket\") pod \"9720d030-7806-479f-840c-38401180d116\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " Oct 30 17:16:26.322599 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.322441 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-istio-data\") pod \"9720d030-7806-479f-840c-38401180d116\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " Oct 30 17:16:26.322599 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.322500 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-istio-envoy\") pod \"9720d030-7806-479f-840c-38401180d116\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " Oct 30 17:16:26.322599 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.322543 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9720d030-7806-479f-840c-38401180d116-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9720d030-7806-479f-840c-38401180d116-istiod-ca-cert\") pod \"9720d030-7806-479f-840c-38401180d116\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " Oct 30 17:16:26.322803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.322625 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9720d030-7806-479f-840c-38401180d116-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9720d030-7806-479f-840c-38401180d116-istio-podinfo\") pod \"9720d030-7806-479f-840c-38401180d116\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " Oct 30 17:16:26.322803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.322636 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-credential-socket" pod "9720d030-7806-479f-840c-38401180d116" (UID: "9720d030-7806-479f-840c-38401180d116"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:26.322803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.322665 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-workload-socket\") pod \"9720d030-7806-479f-840c-38401180d116\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " Oct 30 17:16:26.322803 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.322751 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-istio-data" pod "9720d030-7806-479f-840c-38401180d116" (UID: "9720d030-7806-479f-840c-38401180d116"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:26.323104 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.322953 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9720d030-7806-479f-840c-38401180d116-istio-token\" (UniqueName: \"kubernetes.io/projected/9720d030-7806-479f-840c-38401180d116-istio-token\") pod \"9720d030-7806-479f-840c-38401180d116\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " Oct 30 17:16:26.323104 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.323036 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9720d030-7806-479f-840c-38401180d116-istiod-ca-cert" pod "9720d030-7806-479f-840c-38401180d116" (UID: "9720d030-7806-479f-840c-38401180d116"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:26.323104 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.323037 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9720d030-7806-479f-840c-38401180d116-kube-api-access-h66db\" (UniqueName: \"kubernetes.io/projected/9720d030-7806-479f-840c-38401180d116-kube-api-access-h66db\") pod \"9720d030-7806-479f-840c-38401180d116\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " Oct 30 17:16:26.323104 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.323072 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-workload-socket" pod "9720d030-7806-479f-840c-38401180d116" (UID: "9720d030-7806-479f-840c-38401180d116"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:26.323280 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.323130 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-workload-certs\") pod \"9720d030-7806-479f-840c-38401180d116\" (UID: \"9720d030-7806-479f-840c-38401180d116\") " Oct 30 17:16:26.323789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.323708 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-workload-certs" pod "9720d030-7806-479f-840c-38401180d116" (UID: "9720d030-7806-479f-840c-38401180d116"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:26.324018 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.323993 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.324123 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.324023 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.324123 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.324039 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.324123 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.324067 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b79b467-33a3-45f3-8523-a2eda2ef2bca-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.324123 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.324081 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9720d030-7806-479f-840c-38401180d116-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.324123 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.324095 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbc9g\" (UniqueName: \"kubernetes.io/projected/3b79b467-33a3-45f3-8523-a2eda2ef2bca-kube-api-access-xbc9g\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.324123 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.324109 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.325561 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.325532 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9720d030-7806-479f-840c-38401180d116-istio-token" pod "9720d030-7806-479f-840c-38401180d116" (UID: "9720d030-7806-479f-840c-38401180d116"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:26.325876 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.325848 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9720d030-7806-479f-840c-38401180d116-istio-podinfo" pod "9720d030-7806-479f-840c-38401180d116" (UID: "9720d030-7806-479f-840c-38401180d116"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:16:26.326262 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.326242 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-istio-envoy" pod "9720d030-7806-479f-840c-38401180d116" (UID: "9720d030-7806-479f-840c-38401180d116"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:26.326502 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.326483 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9720d030-7806-479f-840c-38401180d116-kube-api-access-h66db" pod "9720d030-7806-479f-840c-38401180d116" (UID: "9720d030-7806-479f-840c-38401180d116"). InnerVolumeSpecName "kube-api-access-h66db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:26.393922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.393888 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk"] Oct 30 17:16:26.412354 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.412312 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f87a1115-7d38-4949-b6aa-72e5987f8bc5" path="/var/lib/kubelet/pods/f87a1115-7d38-4949-b6aa-72e5987f8bc5/volumes" Oct 30 17:16:26.425161 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.425128 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9720d030-7806-479f-840c-38401180d116-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.425161 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.425153 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9720d030-7806-479f-840c-38401180d116-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.425296 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.425166 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9720d030-7806-479f-840c-38401180d116-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.425296 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.425179 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h66db\" (UniqueName: \"kubernetes.io/projected/9720d030-7806-479f-840c-38401180d116-kube-api-access-h66db\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:26.523429 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:26.523348 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:27.040670 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:27.040636 6202 generic.go:356] "Generic (PLEG): container finished" podID="9720d030-7806-479f-840c-38401180d116" containerID="2f0baa4e01f05c6df1e9d110478272b8f49779d7f8d84d7741db5676101d1014" exitCode=0 Oct 30 17:16:27.041376 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:27.040937 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" Oct 30 17:16:27.041459 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:27.041432 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" event={"ID":"9720d030-7806-479f-840c-38401180d116","Type":"ContainerDied","Data":"2f0baa4e01f05c6df1e9d110478272b8f49779d7f8d84d7741db5676101d1014"} Oct 30 17:16:27.041521 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:27.041476 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g" event={"ID":"9720d030-7806-479f-840c-38401180d116","Type":"ContainerDied","Data":"8dbe6c25b2a4a30141489e50375bcb4f6a768f5b92ef5f7cd1b336a377c71178"} Oct 30 17:16:27.041521 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:27.041499 6202 scope.go:122] "RemoveContainer" containerID="2f0baa4e01f05c6df1e9d110478272b8f49779d7f8d84d7741db5676101d1014" Oct 30 17:16:27.044183 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:27.044156 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g" event={"ID":"3b79b467-33a3-45f3-8523-a2eda2ef2bca","Type":"ContainerDied","Data":"9930a79844c4370810a4d573d7732b09eb5a4722cc580130ed5fac322f2f6ee6"} Oct 30 17:16:27.044183 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:27.044175 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g" Oct 30 17:16:27.044389 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:27.044205 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9930a79844c4370810a4d573d7732b09eb5a4722cc580130ed5fac322f2f6ee6" Oct 30 17:16:27.053283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:27.053262 6202 scope.go:122] "RemoveContainer" containerID="cf0bdbadc7890ddb7d870142156a3ced9ac0fbd159a375d3a80d5ff0a2e8529e" Oct 30 17:16:27.064815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:27.064754 6202 scope.go:122] "RemoveContainer" containerID="2f0baa4e01f05c6df1e9d110478272b8f49779d7f8d84d7741db5676101d1014" Oct 30 17:16:27.065126 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:16:27.065105 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0baa4e01f05c6df1e9d110478272b8f49779d7f8d84d7741db5676101d1014\": container with ID starting with 2f0baa4e01f05c6df1e9d110478272b8f49779d7f8d84d7741db5676101d1014 not found: ID does not exist" containerID="2f0baa4e01f05c6df1e9d110478272b8f49779d7f8d84d7741db5676101d1014" Oct 30 17:16:27.065235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:27.065132 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0baa4e01f05c6df1e9d110478272b8f49779d7f8d84d7741db5676101d1014"} err="failed to get container status \"2f0baa4e01f05c6df1e9d110478272b8f49779d7f8d84d7741db5676101d1014\": rpc error: code = NotFound desc = could not find container \"2f0baa4e01f05c6df1e9d110478272b8f49779d7f8d84d7741db5676101d1014\": container with ID starting with 2f0baa4e01f05c6df1e9d110478272b8f49779d7f8d84d7741db5676101d1014 not found: ID does not exist" Oct 30 17:16:27.065235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:27.065150 6202 scope.go:122] "RemoveContainer" containerID="cf0bdbadc7890ddb7d870142156a3ced9ac0fbd159a375d3a80d5ff0a2e8529e" Oct 30 17:16:27.065439 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:16:27.065420 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0bdbadc7890ddb7d870142156a3ced9ac0fbd159a375d3a80d5ff0a2e8529e\": container with ID starting with cf0bdbadc7890ddb7d870142156a3ced9ac0fbd159a375d3a80d5ff0a2e8529e not found: ID does not exist" containerID="cf0bdbadc7890ddb7d870142156a3ced9ac0fbd159a375d3a80d5ff0a2e8529e" Oct 30 17:16:27.065481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:27.065448 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0bdbadc7890ddb7d870142156a3ced9ac0fbd159a375d3a80d5ff0a2e8529e"} err="failed to get container status \"cf0bdbadc7890ddb7d870142156a3ced9ac0fbd159a375d3a80d5ff0a2e8529e\": rpc error: code = NotFound desc = could not find container \"cf0bdbadc7890ddb7d870142156a3ced9ac0fbd159a375d3a80d5ff0a2e8529e\": container with ID starting with cf0bdbadc7890ddb7d870142156a3ced9ac0fbd159a375d3a80d5ff0a2e8529e not found: ID does not exist" Oct 30 17:16:27.135709 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:27.135679 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g"] Oct 30 17:16:27.144293 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:27.144262 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-zds2g"] Oct 30 17:16:27.392035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:27.391984 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:28.410209 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:28.410173 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9720d030-7806-479f-840c-38401180d116" path="/var/lib/kubelet/pods/9720d030-7806-479f-840c-38401180d116/volumes" Oct 30 17:16:28.620971 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:28.620928 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" podUID="71a9b980-5908-4440-b609-20cddc8c2327" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:29.206562 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:29.206527 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:29.512359 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:29.512274 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:29.810143 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:29.810053 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:30.895546 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:30.895504 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:32.020794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.020747 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj"] Oct 30 17:16:32.036471 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.036428 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:32.150797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.150733 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:32.161947 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.161909 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" containerID="cri-o://d4547bf9d19c6e8d191af019bcc3e3aac0c0ff2280ee7b5a432f5886f396f170" gracePeriod=45 Oct 30 17:16:32.164016 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.163986 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" containerID="cri-o://863ae4cffd7037cecc08e523b35e721f2ce6126e1eae6d51adaa000171928dbc" gracePeriod=45 Oct 30 17:16:32.169107 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.169076 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" containerID="cri-o://ce91b8ee3cf5aabf314dff53a5b7fd5fbe1d2d31f675085584b64a0fe59d4db6" gracePeriod=45 Oct 30 17:16:32.264967 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.264934 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:32.330038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.324167 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": dial tcp 10.133.0.47:8000: connect: connection refused" Oct 30 17:16:32.346819 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.341863 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7"] Oct 30 17:16:32.346819 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.342591 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" podUID="e7bcb077-0d1d-4046-9a5f-75fc61dd2886" containerName="istio-proxy" containerID="cri-o://16dfe3a2dd90e82eb79cac11fc402a181a1a6fe8fe6eca025395f9414e8cfd81" gracePeriod=30 Oct 30 17:16:32.379626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.379577 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:32.524521 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.524367 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 17:16:32.526553 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.526532 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 17:16:32.538629 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.538606 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 17:16:32.559479 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.559420 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:32.613739 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.613633 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-home\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-home\") pod \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " Oct 30 17:16:32.613739 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.613700 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-model-cache\") pod \"afe32415-d7f2-480d-9035-45509a784874\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " Oct 30 17:16:32.614040 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.613786 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-tmp-dir\") pod \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " Oct 30 17:16:32.614040 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.613827 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-tmp-dir\") pod \"afe32415-d7f2-480d-9035-45509a784874\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " Oct 30 17:16:32.614040 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.613863 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-tls-certs\" (UniqueName: \"kubernetes.io/secret/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-tls-certs\") pod \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " Oct 30 17:16:32.614040 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.613912 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-model-cache\") pod \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " Oct 30 17:16:32.614040 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.613956 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-kube-api-access-s9b2m\" (UniqueName: \"kubernetes.io/projected/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-kube-api-access-s9b2m\") pod \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " Oct 30 17:16:32.614040 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.613998 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-model-cache\") pod \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " Oct 30 17:16:32.614368 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.614059 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-dshm\") pod \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " Oct 30 17:16:32.614427 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.614354 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-model-cache" pod "fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" (UID: "fdb161b5-f964-4fc7-95cd-cc17db3b0fbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:32.614427 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.614367 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-model-cache" pod "afe32415-d7f2-480d-9035-45509a784874" (UID: "afe32415-d7f2-480d-9035-45509a784874"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:32.614427 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.614084 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-home\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-home\") pod \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " Oct 30 17:16:32.614588 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.614461 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-dshm\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-dshm\") pod \"afe32415-d7f2-480d-9035-45509a784874\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " Oct 30 17:16:32.614588 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.614510 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-home\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-home\") pod \"afe32415-d7f2-480d-9035-45509a784874\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " Oct 30 17:16:32.614588 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.614542 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-tls-certs\") pod \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " Oct 30 17:16:32.614759 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.614628 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-tmp-dir" pod "afe32415-d7f2-480d-9035-45509a784874" (UID: "afe32415-d7f2-480d-9035-45509a784874"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:32.614759 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.614680 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-kube-api-access-cbpkn\" (UniqueName: \"kubernetes.io/projected/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-kube-api-access-cbpkn\") pod \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " Oct 30 17:16:32.614759 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.614702 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-home" pod "fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" (UID: "fdb161b5-f964-4fc7-95cd-cc17db3b0fbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:32.614759 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.614727 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-dshm\") pod \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\" (UID: \"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9\") " Oct 30 17:16:32.615022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.614819 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe32415-d7f2-480d-9035-45509a784874-kube-api-access-lfj9l\" (UniqueName: \"kubernetes.io/projected/afe32415-d7f2-480d-9035-45509a784874-kube-api-access-lfj9l\") pod \"afe32415-d7f2-480d-9035-45509a784874\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " Oct 30 17:16:32.615022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.614849 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-home" pod "afe32415-d7f2-480d-9035-45509a784874" (UID: "afe32415-d7f2-480d-9035-45509a784874"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:32.615022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.614866 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afe32415-d7f2-480d-9035-45509a784874-tls-certs\" (UniqueName: \"kubernetes.io/secret/afe32415-d7f2-480d-9035-45509a784874-tls-certs\") pod \"afe32415-d7f2-480d-9035-45509a784874\" (UID: \"afe32415-d7f2-480d-9035-45509a784874\") " Oct 30 17:16:32.615022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.614908 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-tmp-dir\") pod \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\" (UID: \"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb\") " Oct 30 17:16:32.615352 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.615333 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-model-cache" pod "7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" (UID: "7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:32.615484 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.615456 6202 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-model-cache\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.615573 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.615505 6202 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-tmp-dir\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.615573 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.615521 6202 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-model-cache\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.615573 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.615533 6202 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-model-cache\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.615573 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.615535 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-tmp-dir" pod "fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" (UID: "fdb161b5-f964-4fc7-95cd-cc17db3b0fbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:32.615573 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.615547 6202 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-home\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.616008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.615577 6202 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-home\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.616008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.615743 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-tmp-dir" pod "7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" (UID: "7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:32.616809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.616742 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-home" pod "7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" (UID: "7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:32.619042 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.619014 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe32415-d7f2-480d-9035-45509a784874-kube-api-access-lfj9l" pod "afe32415-d7f2-480d-9035-45509a784874" (UID: "afe32415-d7f2-480d-9035-45509a784874"). InnerVolumeSpecName "kube-api-access-lfj9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:32.622850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.622759 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-tls-certs" pod "7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" (UID: "7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:32.622850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.622798 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe32415-d7f2-480d-9035-45509a784874-tls-certs" pod "afe32415-d7f2-480d-9035-45509a784874" (UID: "afe32415-d7f2-480d-9035-45509a784874"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:32.623005 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.622859 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-tls-certs" pod "fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" (UID: "fdb161b5-f964-4fc7-95cd-cc17db3b0fbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:32.623108 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.623082 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-kube-api-access-s9b2m" pod "fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" (UID: "fdb161b5-f964-4fc7-95cd-cc17db3b0fbb"). InnerVolumeSpecName "kube-api-access-s9b2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:32.623160 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.623108 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-dshm" pod "fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" (UID: "fdb161b5-f964-4fc7-95cd-cc17db3b0fbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:32.623160 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.623118 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-dshm" pod "afe32415-d7f2-480d-9035-45509a784874" (UID: "afe32415-d7f2-480d-9035-45509a784874"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:32.624491 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.624463 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-dshm" pod "7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" (UID: "7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:32.624825 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.624799 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-kube-api-access-cbpkn" pod "7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" (UID: "7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9"). InnerVolumeSpecName "kube-api-access-cbpkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:32.718709 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.718663 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn"] Oct 30 17:16:32.725411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.721116 6202 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-tmp-dir\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.725411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.721144 6202 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-home\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.725411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.721157 6202 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-tmp-dir\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.725411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.721175 6202 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-tls-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.725411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.721188 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9b2m\" (UniqueName: \"kubernetes.io/projected/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-kube-api-access-s9b2m\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.725411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.721200 6202 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb-dshm\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.725411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.721214 6202 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afe32415-d7f2-480d-9035-45509a784874-dshm\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.725411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.721229 6202 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-tls-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.725411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.721243 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbpkn\" (UniqueName: \"kubernetes.io/projected/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-kube-api-access-cbpkn\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.725411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.721255 6202 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9-dshm\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.725411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.721267 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfj9l\" (UniqueName: \"kubernetes.io/projected/afe32415-d7f2-480d-9035-45509a784874-kube-api-access-lfj9l\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.725411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.721283 6202 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe32415-d7f2-480d-9035-45509a784874-tls-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.755548 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.755513 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf"] Oct 30 17:16:32.858274 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.858242 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:32.874847 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.874814 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq"] Oct 30 17:16:32.875369 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875356 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9720d030-7806-479f-840c-38401180d116" containerName="istio-proxy" Oct 30 17:16:32.875418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875371 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9720d030-7806-479f-840c-38401180d116" containerName="istio-proxy" Oct 30 17:16:32.875418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875394 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9720d030-7806-479f-840c-38401180d116" containerName="istio-proxy" Oct 30 17:16:32.875418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875399 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9720d030-7806-479f-840c-38401180d116" containerName="istio-proxy" Oct 30 17:16:32.875418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875412 6202 cpu_manager.go:414] "containerMap: removing container" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" Oct 30 17:16:32.875418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875417 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" Oct 30 17:16:32.875580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875433 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3b79b467-33a3-45f3-8523-a2eda2ef2bca" containerName="revoke-keys" Oct 30 17:16:32.875580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875439 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b79b467-33a3-45f3-8523-a2eda2ef2bca" containerName="revoke-keys" Oct 30 17:16:32.875580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875446 6202 cpu_manager.go:414] "containerMap: removing container" podUID="75ef8a93-8965-43f9-a86e-f2101beef171" containerName="revoke-keys" Oct 30 17:16:32.875580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875450 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ef8a93-8965-43f9-a86e-f2101beef171" containerName="revoke-keys" Oct 30 17:16:32.875580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875460 6202 cpu_manager.go:414] "containerMap: removing container" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" Oct 30 17:16:32.875580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875464 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" Oct 30 17:16:32.875580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875475 6202 cpu_manager.go:414] "containerMap: removing container" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" Oct 30 17:16:32.875580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875481 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" Oct 30 17:16:32.875580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875552 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9720d030-7806-479f-840c-38401180d116" containerName="istio-proxy" Oct 30 17:16:32.875580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875559 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerName="main" Oct 30 17:16:32.875580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875569 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b79b467-33a3-45f3-8523-a2eda2ef2bca" containerName="revoke-keys" Oct 30 17:16:32.875580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875574 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ef8a93-8965-43f9-a86e-f2101beef171" containerName="revoke-keys" Oct 30 17:16:32.875580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875582 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerName="main" Oct 30 17:16:32.876016 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.875589 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe32415-d7f2-480d-9035-45509a784874" containerName="main" Oct 30 17:16:32.879120 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.879091 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq" Oct 30 17:16:32.895110 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.895074 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:16:32.895284 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.895262 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:16:32.895377 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.895294 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:16:32.895377 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.895373 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:16:32.898740 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.898715 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq"] Oct 30 17:16:32.984802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:32.980549 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:33.025636 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.025595 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fc04ec-c880-421c-9776-eb326ec1c557-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq\" (UID: \"94fc04ec-c880-421c-9776-eb326ec1c557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq" Oct 30 17:16:33.026209 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.025721 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-476hz\" (UniqueName: \"kubernetes.io/projected/94fc04ec-c880-421c-9776-eb326ec1c557-kube-api-access-476hz\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq\" (UID: \"94fc04ec-c880-421c-9776-eb326ec1c557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq" Oct 30 17:16:33.026209 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.025839 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fc04ec-c880-421c-9776-eb326ec1c557-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq\" (UID: \"94fc04ec-c880-421c-9776-eb326ec1c557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq" Oct 30 17:16:33.108967 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.108886 6202 generic.go:356] "Generic (PLEG): container finished" podID="afe32415-d7f2-480d-9035-45509a784874" containerID="d4547bf9d19c6e8d191af019bcc3e3aac0c0ff2280ee7b5a432f5886f396f170" exitCode=0 Oct 30 17:16:33.108967 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.108954 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" Oct 30 17:16:33.109212 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.108976 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" event={"ID":"afe32415-d7f2-480d-9035-45509a784874","Type":"ContainerDied","Data":"d4547bf9d19c6e8d191af019bcc3e3aac0c0ff2280ee7b5a432f5886f396f170"} Oct 30 17:16:33.109212 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.109018 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq" event={"ID":"afe32415-d7f2-480d-9035-45509a784874","Type":"ContainerDied","Data":"7058cad8da1f4880cbac4b29ea21c271e44f3d078330adcdc08e4c31cbb8d6f8"} Oct 30 17:16:33.109212 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.109040 6202 scope.go:122] "RemoveContainer" containerID="d4547bf9d19c6e8d191af019bcc3e3aac0c0ff2280ee7b5a432f5886f396f170" Oct 30 17:16:33.110636 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.110613 6202 generic.go:356] "Generic (PLEG): container finished" podID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" containerID="ce91b8ee3cf5aabf314dff53a5b7fd5fbe1d2d31f675085584b64a0fe59d4db6" exitCode=0 Oct 30 17:16:33.110829 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.110812 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" Oct 30 17:16:33.110912 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.110813 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" event={"ID":"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb","Type":"ContainerDied","Data":"ce91b8ee3cf5aabf314dff53a5b7fd5fbe1d2d31f675085584b64a0fe59d4db6"} Oct 30 17:16:33.110978 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.110935 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94" event={"ID":"fdb161b5-f964-4fc7-95cd-cc17db3b0fbb","Type":"ContainerDied","Data":"6d3794c900c6e903deba68570ec573f214e4d25fcb2348f9a4837c03741c5af5"} Oct 30 17:16:33.112791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.112743 6202 generic.go:356] "Generic (PLEG): container finished" podID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" containerID="863ae4cffd7037cecc08e523b35e721f2ce6126e1eae6d51adaa000171928dbc" exitCode=0 Oct 30 17:16:33.112943 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.112843 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" event={"ID":"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9","Type":"ContainerDied","Data":"863ae4cffd7037cecc08e523b35e721f2ce6126e1eae6d51adaa000171928dbc"} Oct 30 17:16:33.112943 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.112876 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" event={"ID":"7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9","Type":"ContainerDied","Data":"1a506d9a9bfbd4a15486846931af29c13f4ebca5ca83b4ee5d2b3329276d32fa"} Oct 30 17:16:33.113049 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.112948 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97" Oct 30 17:16:33.125138 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.125116 6202 scope.go:122] "RemoveContainer" containerID="d4547bf9d19c6e8d191af019bcc3e3aac0c0ff2280ee7b5a432f5886f396f170" Oct 30 17:16:33.125565 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:16:33.125543 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4547bf9d19c6e8d191af019bcc3e3aac0c0ff2280ee7b5a432f5886f396f170\": container with ID starting with d4547bf9d19c6e8d191af019bcc3e3aac0c0ff2280ee7b5a432f5886f396f170 not found: ID does not exist" containerID="d4547bf9d19c6e8d191af019bcc3e3aac0c0ff2280ee7b5a432f5886f396f170" Oct 30 17:16:33.125652 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.125576 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4547bf9d19c6e8d191af019bcc3e3aac0c0ff2280ee7b5a432f5886f396f170"} err="failed to get container status \"d4547bf9d19c6e8d191af019bcc3e3aac0c0ff2280ee7b5a432f5886f396f170\": rpc error: code = NotFound desc = could not find container \"d4547bf9d19c6e8d191af019bcc3e3aac0c0ff2280ee7b5a432f5886f396f170\": container with ID starting with d4547bf9d19c6e8d191af019bcc3e3aac0c0ff2280ee7b5a432f5886f396f170 not found: ID does not exist" Oct 30 17:16:33.125652 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.125595 6202 scope.go:122] "RemoveContainer" containerID="ce91b8ee3cf5aabf314dff53a5b7fd5fbe1d2d31f675085584b64a0fe59d4db6" Oct 30 17:16:33.126662 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.126636 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fc04ec-c880-421c-9776-eb326ec1c557-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq\" (UID: \"94fc04ec-c880-421c-9776-eb326ec1c557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq" Oct 30 17:16:33.126815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.126685 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-476hz\" (UniqueName: \"kubernetes.io/projected/94fc04ec-c880-421c-9776-eb326ec1c557-kube-api-access-476hz\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq\" (UID: \"94fc04ec-c880-421c-9776-eb326ec1c557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq" Oct 30 17:16:33.126815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.126720 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fc04ec-c880-421c-9776-eb326ec1c557-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq\" (UID: \"94fc04ec-c880-421c-9776-eb326ec1c557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq" Oct 30 17:16:33.127497 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.127478 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fc04ec-c880-421c-9776-eb326ec1c557-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq\" (UID: \"94fc04ec-c880-421c-9776-eb326ec1c557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq" Oct 30 17:16:33.129272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.129247 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fc04ec-c880-421c-9776-eb326ec1c557-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq\" (UID: \"94fc04ec-c880-421c-9776-eb326ec1c557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq" Oct 30 17:16:33.135530 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.135513 6202 scope.go:122] "RemoveContainer" containerID="ce91b8ee3cf5aabf314dff53a5b7fd5fbe1d2d31f675085584b64a0fe59d4db6" Oct 30 17:16:33.135843 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:16:33.135823 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce91b8ee3cf5aabf314dff53a5b7fd5fbe1d2d31f675085584b64a0fe59d4db6\": container with ID starting with ce91b8ee3cf5aabf314dff53a5b7fd5fbe1d2d31f675085584b64a0fe59d4db6 not found: ID does not exist" containerID="ce91b8ee3cf5aabf314dff53a5b7fd5fbe1d2d31f675085584b64a0fe59d4db6" Oct 30 17:16:33.135911 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.135851 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce91b8ee3cf5aabf314dff53a5b7fd5fbe1d2d31f675085584b64a0fe59d4db6"} err="failed to get container status \"ce91b8ee3cf5aabf314dff53a5b7fd5fbe1d2d31f675085584b64a0fe59d4db6\": rpc error: code = NotFound desc = could not find container \"ce91b8ee3cf5aabf314dff53a5b7fd5fbe1d2d31f675085584b64a0fe59d4db6\": container with ID starting with ce91b8ee3cf5aabf314dff53a5b7fd5fbe1d2d31f675085584b64a0fe59d4db6 not found: ID does not exist" Oct 30 17:16:33.135911 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.135867 6202 scope.go:122] "RemoveContainer" containerID="863ae4cffd7037cecc08e523b35e721f2ce6126e1eae6d51adaa000171928dbc" Oct 30 17:16:33.144813 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.144791 6202 scope.go:122] "RemoveContainer" containerID="863ae4cffd7037cecc08e523b35e721f2ce6126e1eae6d51adaa000171928dbc" Oct 30 17:16:33.145107 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:16:33.145092 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"863ae4cffd7037cecc08e523b35e721f2ce6126e1eae6d51adaa000171928dbc\": container with ID starting with 863ae4cffd7037cecc08e523b35e721f2ce6126e1eae6d51adaa000171928dbc not found: ID does not exist" containerID="863ae4cffd7037cecc08e523b35e721f2ce6126e1eae6d51adaa000171928dbc" Oct 30 17:16:33.145175 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.145113 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863ae4cffd7037cecc08e523b35e721f2ce6126e1eae6d51adaa000171928dbc"} err="failed to get container status \"863ae4cffd7037cecc08e523b35e721f2ce6126e1eae6d51adaa000171928dbc\": rpc error: code = NotFound desc = could not find container \"863ae4cffd7037cecc08e523b35e721f2ce6126e1eae6d51adaa000171928dbc\": container with ID starting with 863ae4cffd7037cecc08e523b35e721f2ce6126e1eae6d51adaa000171928dbc not found: ID does not exist" Oct 30 17:16:33.151111 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.151091 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-476hz\" (UniqueName: \"kubernetes.io/projected/94fc04ec-c880-421c-9776-eb326ec1c557-kube-api-access-476hz\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq\" (UID: \"94fc04ec-c880-421c-9776-eb326ec1c557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq" Oct 30 17:16:33.180592 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.180565 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq"] Oct 30 17:16:33.185424 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.185402 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/distinct-2-w3-0b2348-kserve-6f4d9f4944-dh7kq"] Oct 30 17:16:33.195511 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.195492 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq" Oct 30 17:16:33.246281 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.246239 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97"] Oct 30 17:16:33.260555 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.260527 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/distinct-w3-0b2348-kserve-7d8cd756c9-rlp97"] Oct 30 17:16:33.307836 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.300686 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94"] Oct 30 17:16:33.307836 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.305534 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/premium-w3-0b2348-kserve-7c87b8c748-qnd94"] Oct 30 17:16:33.497020 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:16:33.496987 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fc04ec_c880_421c_9776_eb326ec1c557.slice/crio-51367b9ed8913a372579854758707504f244abf4d79345e581b3da947647de6d WatchSource:0}: Error finding container 51367b9ed8913a372579854758707504f244abf4d79345e581b3da947647de6d: Status 404 returned error can't find the container with id 51367b9ed8913a372579854758707504f244abf4d79345e581b3da947647de6d Oct 30 17:16:33.498049 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:33.498021 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq"] Oct 30 17:16:34.126533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:34.126495 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq" event={"ID":"94fc04ec-c880-421c-9776-eb326ec1c557","Type":"ContainerStarted","Data":"c3f6e942e1d519eff4860ebe3fd8959b4f53c98db3cafdaa1930239a0a33a660"} Oct 30 17:16:34.127058 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:34.126557 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq" event={"ID":"94fc04ec-c880-421c-9776-eb326ec1c557","Type":"ContainerStarted","Data":"51367b9ed8913a372579854758707504f244abf4d79345e581b3da947647de6d"} Oct 30 17:16:34.168848 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:34.168750 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq" podStartSLOduration=2.168731747 podStartE2EDuration="2.168731747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:34.168430733 +0000 UTC m=+3307.878885477" watchObservedRunningTime="2026-10-30 17:16:34.168731747 +0000 UTC m=+3307.879186494" Oct 30 17:16:34.410691 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:34.410616 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9" path="/var/lib/kubelet/pods/7a8a1983-6f7b-4a9c-b9d5-8beaf2a490f9/volumes" Oct 30 17:16:34.411125 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:34.411108 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe32415-d7f2-480d-9035-45509a784874" path="/var/lib/kubelet/pods/afe32415-d7f2-480d-9035-45509a784874/volumes" Oct 30 17:16:34.411475 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:34.411463 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb161b5-f964-4fc7-95cd-cc17db3b0fbb" path="/var/lib/kubelet/pods/fdb161b5-f964-4fc7-95cd-cc17db3b0fbb/volumes" Oct 30 17:16:34.510853 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:34.510822 6202 scope.go:122] "RemoveContainer" containerID="72bf8fbf06dc3f6563476b1e148ee169eb7940e98faebe3e75ac197ce35abae7" Oct 30 17:16:34.628030 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:34.628000 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:35.137253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:35.137211 6202 generic.go:356] "Generic (PLEG): container finished" podID="94fc04ec-c880-421c-9776-eb326ec1c557" containerID="c3f6e942e1d519eff4860ebe3fd8959b4f53c98db3cafdaa1930239a0a33a660" exitCode=0 Oct 30 17:16:35.137718 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:35.137305 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq" event={"ID":"94fc04ec-c880-421c-9776-eb326ec1c557","Type":"ContainerDied","Data":"c3f6e942e1d519eff4860ebe3fd8959b4f53c98db3cafdaa1930239a0a33a660"} Oct 30 17:16:36.070748 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.070710 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:36.078893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.078857 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5"] Oct 30 17:16:36.079537 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.079522 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9720d030-7806-479f-840c-38401180d116" containerName="istio-proxy" Oct 30 17:16:36.084385 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.084366 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5" Oct 30 17:16:36.099725 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.099701 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx"] Oct 30 17:16:36.104197 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.103902 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5"] Oct 30 17:16:36.104920 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.104879 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx" Oct 30 17:16:36.199797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.173796 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60ed6312-nd7s5\" (UID: \"a1687a6a-5aa8-4770-85c4-bfd703bb96b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5" Oct 30 17:16:36.199797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.173917 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh5wf\" (UniqueName: \"kubernetes.io/projected/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-kube-api-access-fh5wf\") pod \"maas-api-revoke-sub-60ed6312-nd7s5\" (UID: \"a1687a6a-5aa8-4770-85c4-bfd703bb96b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5" Oct 30 17:16:36.199797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.173957 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60ed6312-nd7s5\" (UID: \"a1687a6a-5aa8-4770-85c4-bfd703bb96b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5" Oct 30 17:16:36.214399 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.202057 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:36.220947 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.220911 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx"] Oct 30 17:16:36.275051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.275016 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57982247-907d-43dd-9274-a28b925bcf5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1de1d73b-8z7fx\" (UID: \"57982247-907d-43dd-9274-a28b925bcf5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx" Oct 30 17:16:36.275247 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.275071 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57982247-907d-43dd-9274-a28b925bcf5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1de1d73b-8z7fx\" (UID: \"57982247-907d-43dd-9274-a28b925bcf5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx" Oct 30 17:16:36.275247 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.275108 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm7xp\" (UniqueName: \"kubernetes.io/projected/57982247-907d-43dd-9274-a28b925bcf5b-kube-api-access-xm7xp\") pod \"maas-api-revoke-sub-1de1d73b-8z7fx\" (UID: \"57982247-907d-43dd-9274-a28b925bcf5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx" Oct 30 17:16:36.275247 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.275187 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60ed6312-nd7s5\" (UID: \"a1687a6a-5aa8-4770-85c4-bfd703bb96b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5" Oct 30 17:16:36.275415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.275259 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh5wf\" (UniqueName: \"kubernetes.io/projected/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-kube-api-access-fh5wf\") pod \"maas-api-revoke-sub-60ed6312-nd7s5\" (UID: \"a1687a6a-5aa8-4770-85c4-bfd703bb96b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5" Oct 30 17:16:36.275415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.275279 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60ed6312-nd7s5\" (UID: \"a1687a6a-5aa8-4770-85c4-bfd703bb96b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5" Oct 30 17:16:36.277798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.276388 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60ed6312-nd7s5\" (UID: \"a1687a6a-5aa8-4770-85c4-bfd703bb96b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5" Oct 30 17:16:36.278658 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.278632 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60ed6312-nd7s5\" (UID: \"a1687a6a-5aa8-4770-85c4-bfd703bb96b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5" Oct 30 17:16:36.300157 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.300129 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh5wf\" (UniqueName: \"kubernetes.io/projected/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-kube-api-access-fh5wf\") pod \"maas-api-revoke-sub-60ed6312-nd7s5\" (UID: \"a1687a6a-5aa8-4770-85c4-bfd703bb96b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5" Oct 30 17:16:36.328277 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.328232 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:36.376709 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.376667 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57982247-907d-43dd-9274-a28b925bcf5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1de1d73b-8z7fx\" (UID: \"57982247-907d-43dd-9274-a28b925bcf5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx" Oct 30 17:16:36.377001 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.376739 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57982247-907d-43dd-9274-a28b925bcf5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1de1d73b-8z7fx\" (UID: \"57982247-907d-43dd-9274-a28b925bcf5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx" Oct 30 17:16:36.377001 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.376799 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm7xp\" (UniqueName: \"kubernetes.io/projected/57982247-907d-43dd-9274-a28b925bcf5b-kube-api-access-xm7xp\") pod \"maas-api-revoke-sub-1de1d73b-8z7fx\" (UID: \"57982247-907d-43dd-9274-a28b925bcf5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx" Oct 30 17:16:36.378427 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.378378 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57982247-907d-43dd-9274-a28b925bcf5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1de1d73b-8z7fx\" (UID: \"57982247-907d-43dd-9274-a28b925bcf5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx" Oct 30 17:16:36.380599 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.380567 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57982247-907d-43dd-9274-a28b925bcf5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1de1d73b-8z7fx\" (UID: \"57982247-907d-43dd-9274-a28b925bcf5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx" Oct 30 17:16:36.399747 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.399717 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5" Oct 30 17:16:36.402472 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.402451 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm7xp\" (UniqueName: \"kubernetes.io/projected/57982247-907d-43dd-9274-a28b925bcf5b-kube-api-access-xm7xp\") pod \"maas-api-revoke-sub-1de1d73b-8z7fx\" (UID: \"57982247-907d-43dd-9274-a28b925bcf5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx" Oct 30 17:16:36.420915 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.420884 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx" Oct 30 17:16:36.434532 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.434505 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq" Oct 30 17:16:36.441436 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.441406 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:36.580985 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.580819 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94fc04ec-c880-421c-9776-eb326ec1c557-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fc04ec-c880-421c-9776-eb326ec1c557-maas-api-xks-ca\") pod \"94fc04ec-c880-421c-9776-eb326ec1c557\" (UID: \"94fc04ec-c880-421c-9776-eb326ec1c557\") " Oct 30 17:16:36.580985 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.580898 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94fc04ec-c880-421c-9776-eb326ec1c557-kube-api-access-476hz\" (UniqueName: \"kubernetes.io/projected/94fc04ec-c880-421c-9776-eb326ec1c557-kube-api-access-476hz\") pod \"94fc04ec-c880-421c-9776-eb326ec1c557\" (UID: \"94fc04ec-c880-421c-9776-eb326ec1c557\") " Oct 30 17:16:36.581206 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.581036 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94fc04ec-c880-421c-9776-eb326ec1c557-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fc04ec-c880-421c-9776-eb326ec1c557-maas-api-openshift-ca\") pod \"94fc04ec-c880-421c-9776-eb326ec1c557\" (UID: \"94fc04ec-c880-421c-9776-eb326ec1c557\") " Oct 30 17:16:36.581812 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.581760 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94fc04ec-c880-421c-9776-eb326ec1c557-maas-api-openshift-ca" pod "94fc04ec-c880-421c-9776-eb326ec1c557" (UID: "94fc04ec-c880-421c-9776-eb326ec1c557"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:36.583930 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.583904 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94fc04ec-c880-421c-9776-eb326ec1c557-kube-api-access-476hz" pod "94fc04ec-c880-421c-9776-eb326ec1c557" (UID: "94fc04ec-c880-421c-9776-eb326ec1c557"). InnerVolumeSpecName "kube-api-access-476hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:36.584097 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.584071 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94fc04ec-c880-421c-9776-eb326ec1c557-maas-api-xks-ca" pod "94fc04ec-c880-421c-9776-eb326ec1c557" (UID: "94fc04ec-c880-421c-9776-eb326ec1c557"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:36.682721 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.682628 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fc04ec-c880-421c-9776-eb326ec1c557-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:36.682721 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.682667 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-476hz\" (UniqueName: \"kubernetes.io/projected/94fc04ec-c880-421c-9776-eb326ec1c557-kube-api-access-476hz\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:36.682721 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.682684 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fc04ec-c880-421c-9776-eb326ec1c557-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:36.754336 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.754299 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx"] Oct 30 17:16:36.756075 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:16:36.756037 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57982247_907d_43dd_9274_a28b925bcf5b.slice/crio-d44ad42202d92e937d96f9d89066ac3b36ad391ed5c5ad695be6d830e93711cc WatchSource:0}: Error finding container d44ad42202d92e937d96f9d89066ac3b36ad391ed5c5ad695be6d830e93711cc: Status 404 returned error can't find the container with id d44ad42202d92e937d96f9d89066ac3b36ad391ed5c5ad695be6d830e93711cc Oct 30 17:16:36.765139 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.763404 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5"] Oct 30 17:16:36.765139 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:16:36.764633 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1687a6a_5aa8_4770_85c4_bfd703bb96b8.slice/crio-b00239030d3a6e953a0d8b74719022314a6f836cf3c541cba72a8e0c5463b5d7 WatchSource:0}: Error finding container b00239030d3a6e953a0d8b74719022314a6f836cf3c541cba72a8e0c5463b5d7: Status 404 returned error can't find the container with id b00239030d3a6e953a0d8b74719022314a6f836cf3c541cba72a8e0c5463b5d7 Oct 30 17:16:36.915980 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:36.915893 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:37.021808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.021744 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj" podUID="ce1d5787-a563-4b1e-822f-b34c2603346b" containerName="payload-pre-processing" containerID="cri-o://de870726e83e7d7e42e1e4ae895588ac0d0891c80d52efe10b7560387fd533f3" gracePeriod=25 Oct 30 17:16:37.033203 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.033175 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:37.060551 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.060514 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7"] Oct 30 17:16:37.063205 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.063177 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc10d-vvbq7"] Oct 30 17:16:37.155967 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.155897 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:37.269845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.268828 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq" event={"ID":"94fc04ec-c880-421c-9776-eb326ec1c557","Type":"ContainerDied","Data":"51367b9ed8913a372579854758707504f244abf4d79345e581b3da947647de6d"} Oct 30 17:16:37.269845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.268888 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51367b9ed8913a372579854758707504f244abf4d79345e581b3da947647de6d" Oct 30 17:16:37.269845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.268949 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq" Oct 30 17:16:37.272821 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.272760 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx" event={"ID":"57982247-907d-43dd-9274-a28b925bcf5b","Type":"ContainerStarted","Data":"bd99352b1c8ebee123bd4ecd3d69e825debfc360d7b0e19b07154166dfa1c25b"} Oct 30 17:16:37.273073 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.272851 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx" event={"ID":"57982247-907d-43dd-9274-a28b925bcf5b","Type":"ContainerStarted","Data":"d44ad42202d92e937d96f9d89066ac3b36ad391ed5c5ad695be6d830e93711cc"} Oct 30 17:16:37.275797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.275703 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5" event={"ID":"a1687a6a-5aa8-4770-85c4-bfd703bb96b8","Type":"ContainerStarted","Data":"b00239030d3a6e953a0d8b74719022314a6f836cf3c541cba72a8e0c5463b5d7"} Oct 30 17:16:37.277687 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.277660 6202 generic.go:356] "Generic (PLEG): container finished" podID="ce1d5787-a563-4b1e-822f-b34c2603346b" containerID="de870726e83e7d7e42e1e4ae895588ac0d0891c80d52efe10b7560387fd533f3" exitCode=0 Oct 30 17:16:37.277865 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.277826 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj" event={"ID":"ce1d5787-a563-4b1e-822f-b34c2603346b","Type":"ContainerDied","Data":"de870726e83e7d7e42e1e4ae895588ac0d0891c80d52efe10b7560387fd533f3"} Oct 30 17:16:37.315022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.314990 6202 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:16:37.315136 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.315036 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" podUID="e7bcb077-0d1d-4046-9a5f-75fc61dd2886" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:16:37.316081 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.316039 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx" podStartSLOduration=1.316027422 podStartE2EDuration="1.316027422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:37.315339966 +0000 UTC m=+3311.025794710" watchObservedRunningTime="2026-10-30 17:16:37.316027422 +0000 UTC m=+3311.026482166" Oct 30 17:16:37.319642 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.319622 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj" Oct 30 17:16:37.375608 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.375544 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5" podStartSLOduration=1.375525528 podStartE2EDuration="1.375525528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:37.374541448 +0000 UTC m=+3311.084996192" watchObservedRunningTime="2026-10-30 17:16:37.375525528 +0000 UTC m=+3311.085980273" Oct 30 17:16:37.389749 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.389722 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce1d5787-a563-4b1e-822f-b34c2603346b-kube-api-access-9sngw\" (UniqueName: \"kubernetes.io/projected/ce1d5787-a563-4b1e-822f-b34c2603346b-kube-api-access-9sngw\") pod \"ce1d5787-a563-4b1e-822f-b34c2603346b\" (UID: \"ce1d5787-a563-4b1e-822f-b34c2603346b\") " Oct 30 17:16:37.389914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.389889 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce1d5787-a563-4b1e-822f-b34c2603346b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce1d5787-a563-4b1e-822f-b34c2603346b-plugins-config-volume\") pod \"ce1d5787-a563-4b1e-822f-b34c2603346b\" (UID: \"ce1d5787-a563-4b1e-822f-b34c2603346b\") " Oct 30 17:16:37.390388 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.390350 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1d5787-a563-4b1e-822f-b34c2603346b-plugins-config-volume" pod "ce1d5787-a563-4b1e-822f-b34c2603346b" (UID: "ce1d5787-a563-4b1e-822f-b34c2603346b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:37.390624 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.390586 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce1d5787-a563-4b1e-822f-b34c2603346b-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:37.392644 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.392622 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1d5787-a563-4b1e-822f-b34c2603346b-kube-api-access-9sngw" pod "ce1d5787-a563-4b1e-822f-b34c2603346b" (UID: "ce1d5787-a563-4b1e-822f-b34c2603346b"). InnerVolumeSpecName "kube-api-access-9sngw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:37.491867 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.491809 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sngw\" (UniqueName: \"kubernetes.io/projected/ce1d5787-a563-4b1e-822f-b34c2603346b-kube-api-access-9sngw\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:37.579642 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.579596 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:37.604907 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.604884 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:16:37.694649 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.694552 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-workload-certs\") pod \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " Oct 30 17:16:37.694649 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.694599 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-credential-socket\") pod \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " Oct 30 17:16:37.694649 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.694650 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-workload-socket\") pod \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " Oct 30 17:16:37.695022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.694671 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istiod-ca-cert\") pod \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " Oct 30 17:16:37.695022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.694708 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-token\" (UniqueName: \"kubernetes.io/projected/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-token\") pod \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " Oct 30 17:16:37.695022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.694849 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-kube-api-access-nzw97\" (UniqueName: \"kubernetes.io/projected/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-kube-api-access-nzw97\") pod \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " Oct 30 17:16:37.695022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.694890 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-envoy\") pod \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " Oct 30 17:16:37.695022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.694948 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-podinfo\") pod \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " Oct 30 17:16:37.695022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.694986 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-data\") pod \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\" (UID: \"e7bcb077-0d1d-4046-9a5f-75fc61dd2886\") " Oct 30 17:16:37.695367 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.695043 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-workload-socket" pod "e7bcb077-0d1d-4046-9a5f-75fc61dd2886" (UID: "e7bcb077-0d1d-4046-9a5f-75fc61dd2886"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:37.695367 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.695244 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-data" pod "e7bcb077-0d1d-4046-9a5f-75fc61dd2886" (UID: "e7bcb077-0d1d-4046-9a5f-75fc61dd2886"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:37.695367 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.695281 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istiod-ca-cert" pod "e7bcb077-0d1d-4046-9a5f-75fc61dd2886" (UID: "e7bcb077-0d1d-4046-9a5f-75fc61dd2886"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:37.695551 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.695483 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:37.695551 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.695506 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:37.695551 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.695520 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:37.695868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.695844 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-workload-certs" pod "e7bcb077-0d1d-4046-9a5f-75fc61dd2886" (UID: "e7bcb077-0d1d-4046-9a5f-75fc61dd2886"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:37.696042 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.696017 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-credential-socket" pod "e7bcb077-0d1d-4046-9a5f-75fc61dd2886" (UID: "e7bcb077-0d1d-4046-9a5f-75fc61dd2886"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:37.697510 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.697486 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-envoy" pod "e7bcb077-0d1d-4046-9a5f-75fc61dd2886" (UID: "e7bcb077-0d1d-4046-9a5f-75fc61dd2886"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:37.698154 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.698133 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-token" pod "e7bcb077-0d1d-4046-9a5f-75fc61dd2886" (UID: "e7bcb077-0d1d-4046-9a5f-75fc61dd2886"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:37.698260 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.698230 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-kube-api-access-nzw97" pod "e7bcb077-0d1d-4046-9a5f-75fc61dd2886" (UID: "e7bcb077-0d1d-4046-9a5f-75fc61dd2886"). InnerVolumeSpecName "kube-api-access-nzw97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:37.698963 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.698938 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-podinfo" pod "e7bcb077-0d1d-4046-9a5f-75fc61dd2886" (UID: "e7bcb077-0d1d-4046-9a5f-75fc61dd2886"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:16:37.720087 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.720054 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" podUID="f750fe64-42e5-4d6f-91f7-071c1493c881" containerName="payload-processing" containerID="cri-o://704f712bbb2dbfd40ae22408a6c69ba1e475c9ff4d79039bced0825f04ae1a49" gracePeriod=25 Oct 30 17:16:37.755910 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.755838 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" podUID="1b94338f-f1d5-4e7c-880d-905f46fd9c7e" containerName="payload-pre-processing" containerID="cri-o://f778a7ac0ea6b96d017c00add1efc0dbb8b7fa2135dae671c99b669e524868aa" gracePeriod=25 Oct 30 17:16:37.796572 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.796540 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzw97\" (UniqueName: \"kubernetes.io/projected/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-kube-api-access-nzw97\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:37.796572 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.796570 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:37.796732 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.796580 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:37.796732 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.796588 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:37.796732 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.796596 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:37.796732 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:37.796603 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7bcb077-0d1d-4046-9a5f-75fc61dd2886-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:38.288786 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.288728 6202 generic.go:356] "Generic (PLEG): container finished" podID="57982247-907d-43dd-9274-a28b925bcf5b" containerID="bd99352b1c8ebee123bd4ecd3d69e825debfc360d7b0e19b07154166dfa1c25b" exitCode=0 Oct 30 17:16:38.289296 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.288827 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx" event={"ID":"57982247-907d-43dd-9274-a28b925bcf5b","Type":"ContainerDied","Data":"bd99352b1c8ebee123bd4ecd3d69e825debfc360d7b0e19b07154166dfa1c25b"} Oct 30 17:16:38.290838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.290807 6202 generic.go:356] "Generic (PLEG): container finished" podID="a1687a6a-5aa8-4770-85c4-bfd703bb96b8" containerID="09f70a0b42f9f400409753b7c3f5643f9e5494fc0fd82337fc1e9a1497d213aa" exitCode=0 Oct 30 17:16:38.290944 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.290902 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5" event={"ID":"a1687a6a-5aa8-4770-85c4-bfd703bb96b8","Type":"ContainerDied","Data":"09f70a0b42f9f400409753b7c3f5643f9e5494fc0fd82337fc1e9a1497d213aa"} Oct 30 17:16:38.292702 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.292679 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj" Oct 30 17:16:38.292945 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.292699 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj" event={"ID":"ce1d5787-a563-4b1e-822f-b34c2603346b","Type":"ContainerDied","Data":"c23e2ff0fda83be872e72ccc4dc96877eee0b82298279f779864e78444ecac74"} Oct 30 17:16:38.292945 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.292755 6202 scope.go:122] "RemoveContainer" containerID="de870726e83e7d7e42e1e4ae895588ac0d0891c80d52efe10b7560387fd533f3" Oct 30 17:16:38.295058 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.295033 6202 generic.go:356] "Generic (PLEG): container finished" podID="e7bcb077-0d1d-4046-9a5f-75fc61dd2886" containerID="16dfe3a2dd90e82eb79cac11fc402a181a1a6fe8fe6eca025395f9414e8cfd81" exitCode=0 Oct 30 17:16:38.295277 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.295093 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" Oct 30 17:16:38.295277 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.295132 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" event={"ID":"e7bcb077-0d1d-4046-9a5f-75fc61dd2886","Type":"ContainerDied","Data":"16dfe3a2dd90e82eb79cac11fc402a181a1a6fe8fe6eca025395f9414e8cfd81"} Oct 30 17:16:38.295406 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.295289 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7" event={"ID":"e7bcb077-0d1d-4046-9a5f-75fc61dd2886","Type":"ContainerDied","Data":"82924796ddb011f300b8dcc1f72c87727263cb29446f5b39319138c08391d929"} Oct 30 17:16:38.297709 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.297683 6202 generic.go:356] "Generic (PLEG): container finished" podID="71a9b980-5908-4440-b609-20cddc8c2327" containerID="f41e2acabaaef5dc30f711a57be593767e8cc5992c6323b04a73deca2f41274c" exitCode=0 Oct 30 17:16:38.297853 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.297718 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" event={"ID":"71a9b980-5908-4440-b609-20cddc8c2327","Type":"ContainerDied","Data":"f41e2acabaaef5dc30f711a57be593767e8cc5992c6323b04a73deca2f41274c"} Oct 30 17:16:38.307131 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.307113 6202 scope.go:122] "RemoveContainer" containerID="16dfe3a2dd90e82eb79cac11fc402a181a1a6fe8fe6eca025395f9414e8cfd81" Oct 30 17:16:38.328175 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.328158 6202 scope.go:122] "RemoveContainer" containerID="ce8b40adbd789f536bf361e2b3ad0e0616e05d724036925874c9ffa350ed2c5d" Oct 30 17:16:38.328542 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.328524 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" Oct 30 17:16:38.341149 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.341062 6202 scope.go:122] "RemoveContainer" containerID="16dfe3a2dd90e82eb79cac11fc402a181a1a6fe8fe6eca025395f9414e8cfd81" Oct 30 17:16:38.341437 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:16:38.341406 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16dfe3a2dd90e82eb79cac11fc402a181a1a6fe8fe6eca025395f9414e8cfd81\": container with ID starting with 16dfe3a2dd90e82eb79cac11fc402a181a1a6fe8fe6eca025395f9414e8cfd81 not found: ID does not exist" containerID="16dfe3a2dd90e82eb79cac11fc402a181a1a6fe8fe6eca025395f9414e8cfd81" Oct 30 17:16:38.341515 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.341445 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16dfe3a2dd90e82eb79cac11fc402a181a1a6fe8fe6eca025395f9414e8cfd81"} err="failed to get container status \"16dfe3a2dd90e82eb79cac11fc402a181a1a6fe8fe6eca025395f9414e8cfd81\": rpc error: code = NotFound desc = could not find container \"16dfe3a2dd90e82eb79cac11fc402a181a1a6fe8fe6eca025395f9414e8cfd81\": container with ID starting with 16dfe3a2dd90e82eb79cac11fc402a181a1a6fe8fe6eca025395f9414e8cfd81 not found: ID does not exist" Oct 30 17:16:38.341515 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.341465 6202 scope.go:122] "RemoveContainer" containerID="ce8b40adbd789f536bf361e2b3ad0e0616e05d724036925874c9ffa350ed2c5d" Oct 30 17:16:38.341739 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:16:38.341718 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8b40adbd789f536bf361e2b3ad0e0616e05d724036925874c9ffa350ed2c5d\": container with ID starting with ce8b40adbd789f536bf361e2b3ad0e0616e05d724036925874c9ffa350ed2c5d not found: ID does not exist" containerID="ce8b40adbd789f536bf361e2b3ad0e0616e05d724036925874c9ffa350ed2c5d" Oct 30 17:16:38.341842 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.341746 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8b40adbd789f536bf361e2b3ad0e0616e05d724036925874c9ffa350ed2c5d"} err="failed to get container status \"ce8b40adbd789f536bf361e2b3ad0e0616e05d724036925874c9ffa350ed2c5d\": rpc error: code = NotFound desc = could not find container \"ce8b40adbd789f536bf361e2b3ad0e0616e05d724036925874c9ffa350ed2c5d\": container with ID starting with ce8b40adbd789f536bf361e2b3ad0e0616e05d724036925874c9ffa350ed2c5d not found: ID does not exist" Oct 30 17:16:38.372786 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.372738 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7"] Oct 30 17:16:38.385587 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.385546 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b4214a-openshift-default-584cc56697-zbss7"] Oct 30 17:16:38.402822 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.402750 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a9b980-5908-4440-b609-20cddc8c2327-kube-api-access-j7z92\" (UniqueName: \"kubernetes.io/projected/71a9b980-5908-4440-b609-20cddc8c2327-kube-api-access-j7z92\") pod \"71a9b980-5908-4440-b609-20cddc8c2327\" (UID: \"71a9b980-5908-4440-b609-20cddc8c2327\") " Oct 30 17:16:38.402976 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.402933 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a9b980-5908-4440-b609-20cddc8c2327-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a9b980-5908-4440-b609-20cddc8c2327-plugins-config-volume\") pod \"71a9b980-5908-4440-b609-20cddc8c2327\" (UID: \"71a9b980-5908-4440-b609-20cddc8c2327\") " Oct 30 17:16:38.403319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.403282 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a9b980-5908-4440-b609-20cddc8c2327-plugins-config-volume" pod "71a9b980-5908-4440-b609-20cddc8c2327" (UID: "71a9b980-5908-4440-b609-20cddc8c2327"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:38.405077 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.405056 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a9b980-5908-4440-b609-20cddc8c2327-kube-api-access-j7z92" pod "71a9b980-5908-4440-b609-20cddc8c2327" (UID: "71a9b980-5908-4440-b609-20cddc8c2327"). InnerVolumeSpecName "kube-api-access-j7z92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:38.411577 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.411558 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5668b79a-d001-423a-a0bb-6ddb3ae4036f" path="/var/lib/kubelet/pods/5668b79a-d001-423a-a0bb-6ddb3ae4036f/volumes" Oct 30 17:16:38.412119 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.412104 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7bcb077-0d1d-4046-9a5f-75fc61dd2886" path="/var/lib/kubelet/pods/e7bcb077-0d1d-4046-9a5f-75fc61dd2886/volumes" Oct 30 17:16:38.457033 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.456998 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj"] Oct 30 17:16:38.461250 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.461217 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-263f00-5ffc44b54c-rznrj"] Oct 30 17:16:38.504639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.504606 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7z92\" (UniqueName: \"kubernetes.io/projected/71a9b980-5908-4440-b609-20cddc8c2327-kube-api-access-j7z92\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:38.504639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:38.504634 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a9b980-5908-4440-b609-20cddc8c2327-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:39.221488 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.221449 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:39.307322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.307291 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" Oct 30 17:16:39.307836 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.307327 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn" event={"ID":"71a9b980-5908-4440-b609-20cddc8c2327","Type":"ContainerDied","Data":"cb1424763aef090c1ed0546b9fe936712c980f608a8ad47e8af67b8517cc03a4"} Oct 30 17:16:39.307836 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.307374 6202 scope.go:122] "RemoveContainer" containerID="f41e2acabaaef5dc30f711a57be593767e8cc5992c6323b04a73deca2f41274c" Oct 30 17:16:39.365828 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.365794 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn"] Oct 30 17:16:39.380868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.379931 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-05810b-5f5d8bcbc9-rs8jn"] Oct 30 17:16:39.473555 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.473167 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5" Oct 30 17:16:39.503077 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.503053 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx" Oct 30 17:16:39.517822 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.517792 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-maas-api-xks-ca\") pod \"a1687a6a-5aa8-4770-85c4-bfd703bb96b8\" (UID: \"a1687a6a-5aa8-4770-85c4-bfd703bb96b8\") " Oct 30 17:16:39.517961 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.517904 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-maas-api-openshift-ca\") pod \"a1687a6a-5aa8-4770-85c4-bfd703bb96b8\" (UID: \"a1687a6a-5aa8-4770-85c4-bfd703bb96b8\") " Oct 30 17:16:39.517961 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.517955 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-kube-api-access-fh5wf\" (UniqueName: \"kubernetes.io/projected/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-kube-api-access-fh5wf\") pod \"a1687a6a-5aa8-4770-85c4-bfd703bb96b8\" (UID: \"a1687a6a-5aa8-4770-85c4-bfd703bb96b8\") " Oct 30 17:16:39.520419 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.520384 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-maas-api-openshift-ca" pod "a1687a6a-5aa8-4770-85c4-bfd703bb96b8" (UID: "a1687a6a-5aa8-4770-85c4-bfd703bb96b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:39.535270 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.535233 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-maas-api-xks-ca" pod "a1687a6a-5aa8-4770-85c4-bfd703bb96b8" (UID: "a1687a6a-5aa8-4770-85c4-bfd703bb96b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:39.535396 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.535256 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-kube-api-access-fh5wf" pod "a1687a6a-5aa8-4770-85c4-bfd703bb96b8" (UID: "a1687a6a-5aa8-4770-85c4-bfd703bb96b8"). InnerVolumeSpecName "kube-api-access-fh5wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:39.620705 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.620670 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57982247-907d-43dd-9274-a28b925bcf5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57982247-907d-43dd-9274-a28b925bcf5b-maas-api-openshift-ca\") pod \"57982247-907d-43dd-9274-a28b925bcf5b\" (UID: \"57982247-907d-43dd-9274-a28b925bcf5b\") " Oct 30 17:16:39.620892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.620850 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57982247-907d-43dd-9274-a28b925bcf5b-kube-api-access-xm7xp\" (UniqueName: \"kubernetes.io/projected/57982247-907d-43dd-9274-a28b925bcf5b-kube-api-access-xm7xp\") pod \"57982247-907d-43dd-9274-a28b925bcf5b\" (UID: \"57982247-907d-43dd-9274-a28b925bcf5b\") " Oct 30 17:16:39.620990 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.620960 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57982247-907d-43dd-9274-a28b925bcf5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57982247-907d-43dd-9274-a28b925bcf5b-maas-api-xks-ca\") pod \"57982247-907d-43dd-9274-a28b925bcf5b\" (UID: \"57982247-907d-43dd-9274-a28b925bcf5b\") " Oct 30 17:16:39.621058 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.621023 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57982247-907d-43dd-9274-a28b925bcf5b-maas-api-openshift-ca" pod "57982247-907d-43dd-9274-a28b925bcf5b" (UID: "57982247-907d-43dd-9274-a28b925bcf5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:39.621333 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.621314 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57982247-907d-43dd-9274-a28b925bcf5b-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:39.621407 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.621338 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:39.621407 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.621355 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:39.621407 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.621367 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh5wf\" (UniqueName: \"kubernetes.io/projected/a1687a6a-5aa8-4770-85c4-bfd703bb96b8-kube-api-access-fh5wf\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:39.622973 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.622949 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57982247-907d-43dd-9274-a28b925bcf5b-maas-api-xks-ca" pod "57982247-907d-43dd-9274-a28b925bcf5b" (UID: "57982247-907d-43dd-9274-a28b925bcf5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:39.623138 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.623116 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57982247-907d-43dd-9274-a28b925bcf5b-kube-api-access-xm7xp" pod "57982247-907d-43dd-9274-a28b925bcf5b" (UID: "57982247-907d-43dd-9274-a28b925bcf5b"). InnerVolumeSpecName "kube-api-access-xm7xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:39.722272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.722237 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm7xp\" (UniqueName: \"kubernetes.io/projected/57982247-907d-43dd-9274-a28b925bcf5b-kube-api-access-xm7xp\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:39.722272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:39.722269 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57982247-907d-43dd-9274-a28b925bcf5b-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:40.315908 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:40.315863 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx" Oct 30 17:16:40.315908 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:40.315894 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx" event={"ID":"57982247-907d-43dd-9274-a28b925bcf5b","Type":"ContainerDied","Data":"d44ad42202d92e937d96f9d89066ac3b36ad391ed5c5ad695be6d830e93711cc"} Oct 30 17:16:40.316411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:40.315978 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d44ad42202d92e937d96f9d89066ac3b36ad391ed5c5ad695be6d830e93711cc" Oct 30 17:16:40.317350 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:40.317327 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5" event={"ID":"a1687a6a-5aa8-4770-85c4-bfd703bb96b8","Type":"ContainerDied","Data":"b00239030d3a6e953a0d8b74719022314a6f836cf3c541cba72a8e0c5463b5d7"} Oct 30 17:16:40.317472 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:40.317367 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b00239030d3a6e953a0d8b74719022314a6f836cf3c541cba72a8e0c5463b5d7" Oct 30 17:16:40.317472 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:40.317331 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5" Oct 30 17:16:40.411079 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:40.411048 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a9b980-5908-4440-b609-20cddc8c2327" path="/var/lib/kubelet/pods/71a9b980-5908-4440-b609-20cddc8c2327/volumes" Oct 30 17:16:40.411456 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:40.411443 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce1d5787-a563-4b1e-822f-b34c2603346b" path="/var/lib/kubelet/pods/ce1d5787-a563-4b1e-822f-b34c2603346b/volumes" Oct 30 17:16:40.912411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:40.912380 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:41.420264 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.420187 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" containerID="cri-o://34ef19b24a07f4654236190370b147b1a28f4fb8904156664a0ebe669ecee82d" gracePeriod=45 Oct 30 17:16:41.713667 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.713622 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 17:16:41.846047 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.846009 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-tmp-dir\") pod \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " Oct 30 17:16:41.846313 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.846163 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-tls-certs\") pod \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " Oct 30 17:16:41.846313 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.846213 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-home\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-home\") pod \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " Oct 30 17:16:41.846313 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.846298 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-tmp-dir" pod "940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" (UID: "940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.846544 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.846522 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-home" pod "940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" (UID: "940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.846618 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.846530 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-model-cache\") pod \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " Oct 30 17:16:41.846618 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.846595 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-dshm\") pod \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " Oct 30 17:16:41.846797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.846743 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-kube-api-access-gx29h\" (UniqueName: \"kubernetes.io/projected/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-kube-api-access-gx29h\") pod \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\" (UID: \"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9\") " Oct 30 17:16:41.846890 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.846741 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-model-cache" pod "940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" (UID: "940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.847338 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.847312 6202 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-tmp-dir\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.847338 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.847339 6202 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-home\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.847526 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.847353 6202 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-model-cache\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.854805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.849988 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-tls-certs" pod "940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" (UID: "940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:41.854805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.850228 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-dshm" pod "940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" (UID: "940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.854805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.850683 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-kube-api-access-gx29h" pod "940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" (UID: "940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9"). InnerVolumeSpecName "kube-api-access-gx29h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:41.948915 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.948827 6202 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-tls-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.948915 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.948865 6202 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-dshm\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.948915 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:41.948877 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx29h\" (UniqueName: \"kubernetes.io/projected/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9-kube-api-access-gx29h\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:42.334221 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.334188 6202 generic.go:356] "Generic (PLEG): container finished" podID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerID="34ef19b24a07f4654236190370b147b1a28f4fb8904156664a0ebe669ecee82d" exitCode=0 Oct 30 17:16:42.334405 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.334234 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" Oct 30 17:16:42.334405 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.334277 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" event={"ID":"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9","Type":"ContainerDied","Data":"34ef19b24a07f4654236190370b147b1a28f4fb8904156664a0ebe669ecee82d"} Oct 30 17:16:42.334405 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.334323 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk" event={"ID":"940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9","Type":"ContainerDied","Data":"feb44cc899530f952331e1ddd500f73cddc22a678904bfff2eeb6b56e6140035"} Oct 30 17:16:42.334405 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.334343 6202 scope.go:122] "RemoveContainer" containerID="34ef19b24a07f4654236190370b147b1a28f4fb8904156664a0ebe669ecee82d" Oct 30 17:16:42.345304 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.345280 6202 scope.go:122] "RemoveContainer" containerID="34ef19b24a07f4654236190370b147b1a28f4fb8904156664a0ebe669ecee82d" Oct 30 17:16:42.345588 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:16:42.345571 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34ef19b24a07f4654236190370b147b1a28f4fb8904156664a0ebe669ecee82d\": container with ID starting with 34ef19b24a07f4654236190370b147b1a28f4fb8904156664a0ebe669ecee82d not found: ID does not exist" containerID="34ef19b24a07f4654236190370b147b1a28f4fb8904156664a0ebe669ecee82d" Oct 30 17:16:42.345649 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.345597 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ef19b24a07f4654236190370b147b1a28f4fb8904156664a0ebe669ecee82d"} err="failed to get container status \"34ef19b24a07f4654236190370b147b1a28f4fb8904156664a0ebe669ecee82d\": rpc error: code = NotFound desc = could not find container \"34ef19b24a07f4654236190370b147b1a28f4fb8904156664a0ebe669ecee82d\": container with ID starting with 34ef19b24a07f4654236190370b147b1a28f4fb8904156664a0ebe669ecee82d not found: ID does not exist" Oct 30 17:16:42.379183 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379145 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn"] Oct 30 17:16:42.379685 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379666 6202 cpu_manager.go:414] "containerMap: removing container" podUID="e7bcb077-0d1d-4046-9a5f-75fc61dd2886" containerName="istio-proxy" Oct 30 17:16:42.379685 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379683 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7bcb077-0d1d-4046-9a5f-75fc61dd2886" containerName="istio-proxy" Oct 30 17:16:42.379893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379693 6202 cpu_manager.go:414] "containerMap: removing container" podUID="57982247-907d-43dd-9274-a28b925bcf5b" containerName="revoke-keys" Oct 30 17:16:42.379893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379698 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57982247-907d-43dd-9274-a28b925bcf5b" containerName="revoke-keys" Oct 30 17:16:42.379893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379712 6202 cpu_manager.go:414] "containerMap: removing container" podUID="ce1d5787-a563-4b1e-822f-b34c2603346b" containerName="payload-pre-processing" Oct 30 17:16:42.379893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379720 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce1d5787-a563-4b1e-822f-b34c2603346b" containerName="payload-pre-processing" Oct 30 17:16:42.379893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379733 6202 cpu_manager.go:414] "containerMap: removing container" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" Oct 30 17:16:42.379893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379738 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" Oct 30 17:16:42.379893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379761 6202 cpu_manager.go:414] "containerMap: removing container" podUID="e7bcb077-0d1d-4046-9a5f-75fc61dd2886" containerName="istio-proxy" Oct 30 17:16:42.379893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379781 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7bcb077-0d1d-4046-9a5f-75fc61dd2886" containerName="istio-proxy" Oct 30 17:16:42.379893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379791 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a1687a6a-5aa8-4770-85c4-bfd703bb96b8" containerName="revoke-keys" Oct 30 17:16:42.379893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379799 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1687a6a-5aa8-4770-85c4-bfd703bb96b8" containerName="revoke-keys" Oct 30 17:16:42.379893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379809 6202 cpu_manager.go:414] "containerMap: removing container" podUID="94fc04ec-c880-421c-9776-eb326ec1c557" containerName="revoke-keys" Oct 30 17:16:42.379893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379814 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94fc04ec-c880-421c-9776-eb326ec1c557" containerName="revoke-keys" Oct 30 17:16:42.379893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379826 6202 cpu_manager.go:414] "containerMap: removing container" podUID="71a9b980-5908-4440-b609-20cddc8c2327" containerName="payload-pre-processing" Oct 30 17:16:42.379893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379830 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a9b980-5908-4440-b609-20cddc8c2327" containerName="payload-pre-processing" Oct 30 17:16:42.380353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379930 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7bcb077-0d1d-4046-9a5f-75fc61dd2886" containerName="istio-proxy" Oct 30 17:16:42.380353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379940 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1687a6a-5aa8-4770-85c4-bfd703bb96b8" containerName="revoke-keys" Oct 30 17:16:42.380353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379946 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="94fc04ec-c880-421c-9776-eb326ec1c557" containerName="revoke-keys" Oct 30 17:16:42.380353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379956 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a9b980-5908-4440-b609-20cddc8c2327" containerName="payload-pre-processing" Oct 30 17:16:42.380353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379970 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="57982247-907d-43dd-9274-a28b925bcf5b" containerName="revoke-keys" Oct 30 17:16:42.380353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379975 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce1d5787-a563-4b1e-822f-b34c2603346b" containerName="payload-pre-processing" Oct 30 17:16:42.380353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.379981 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" containerName="main" Oct 30 17:16:42.382889 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.382871 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn" Oct 30 17:16:42.391078 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.391057 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk"] Oct 30 17:16:42.397196 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.397175 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/distinct-w4-263f00-kserve-78c874f856-nbqfk"] Oct 30 17:16:42.401425 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.401374 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc"] Oct 30 17:16:42.413445 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.402337 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7bcb077-0d1d-4046-9a5f-75fc61dd2886" containerName="istio-proxy" Oct 30 17:16:42.413445 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.406750 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:16:42.413445 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.406883 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:16:42.413445 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.407077 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:16:42.413445 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.407710 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:16:42.426495 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.426458 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn"] Oct 30 17:16:42.430000 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.429970 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc" Oct 30 17:16:42.486175 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.472237 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjtzg\" (UniqueName: \"kubernetes.io/projected/a9f595f1-0e66-4066-af92-0ded0ada9ac1-kube-api-access-jjtzg\") pod \"maas-api-revoke-sub-8de92707-885gn\" (UID: \"a9f595f1-0e66-4066-af92-0ded0ada9ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn" Oct 30 17:16:42.486175 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.472396 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f595f1-0e66-4066-af92-0ded0ada9ac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8de92707-885gn\" (UID: \"a9f595f1-0e66-4066-af92-0ded0ada9ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn" Oct 30 17:16:42.486175 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.472488 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f595f1-0e66-4066-af92-0ded0ada9ac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8de92707-885gn\" (UID: \"a9f595f1-0e66-4066-af92-0ded0ada9ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn" Oct 30 17:16:42.486175 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.485174 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9" path="/var/lib/kubelet/pods/940f627d-3d5f-4c5d-9b6b-4ffd5b5502d9/volumes" Oct 30 17:16:42.491812 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.491040 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc"] Oct 30 17:16:42.573438 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.573400 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66ckx\" (UniqueName: \"kubernetes.io/projected/635a2ae3-b676-441a-81f8-8037caefe261-kube-api-access-66ckx\") pod \"maas-api-revoke-sub-6ed70e1c-k5wdc\" (UID: \"635a2ae3-b676-441a-81f8-8037caefe261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc" Oct 30 17:16:42.573598 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.573445 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f595f1-0e66-4066-af92-0ded0ada9ac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8de92707-885gn\" (UID: \"a9f595f1-0e66-4066-af92-0ded0ada9ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn" Oct 30 17:16:42.573598 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.573540 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtzg\" (UniqueName: \"kubernetes.io/projected/a9f595f1-0e66-4066-af92-0ded0ada9ac1-kube-api-access-jjtzg\") pod \"maas-api-revoke-sub-8de92707-885gn\" (UID: \"a9f595f1-0e66-4066-af92-0ded0ada9ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn" Oct 30 17:16:42.573598 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.573594 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635a2ae3-b676-441a-81f8-8037caefe261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ed70e1c-k5wdc\" (UID: \"635a2ae3-b676-441a-81f8-8037caefe261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc" Oct 30 17:16:42.573731 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.573683 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f595f1-0e66-4066-af92-0ded0ada9ac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8de92707-885gn\" (UID: \"a9f595f1-0e66-4066-af92-0ded0ada9ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn" Oct 30 17:16:42.573731 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.573721 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635a2ae3-b676-441a-81f8-8037caefe261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ed70e1c-k5wdc\" (UID: \"635a2ae3-b676-441a-81f8-8037caefe261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc" Oct 30 17:16:42.574347 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.574275 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f595f1-0e66-4066-af92-0ded0ada9ac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8de92707-885gn\" (UID: \"a9f595f1-0e66-4066-af92-0ded0ada9ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn" Oct 30 17:16:42.576345 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.576322 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f595f1-0e66-4066-af92-0ded0ada9ac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8de92707-885gn\" (UID: \"a9f595f1-0e66-4066-af92-0ded0ada9ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn" Oct 30 17:16:42.598076 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.598020 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjtzg\" (UniqueName: \"kubernetes.io/projected/a9f595f1-0e66-4066-af92-0ded0ada9ac1-kube-api-access-jjtzg\") pod \"maas-api-revoke-sub-8de92707-885gn\" (UID: \"a9f595f1-0e66-4066-af92-0ded0ada9ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn" Oct 30 17:16:42.675027 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.674995 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635a2ae3-b676-441a-81f8-8037caefe261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ed70e1c-k5wdc\" (UID: \"635a2ae3-b676-441a-81f8-8037caefe261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc" Oct 30 17:16:42.675181 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.675056 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66ckx\" (UniqueName: \"kubernetes.io/projected/635a2ae3-b676-441a-81f8-8037caefe261-kube-api-access-66ckx\") pod \"maas-api-revoke-sub-6ed70e1c-k5wdc\" (UID: \"635a2ae3-b676-441a-81f8-8037caefe261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc" Oct 30 17:16:42.675181 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.675142 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635a2ae3-b676-441a-81f8-8037caefe261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ed70e1c-k5wdc\" (UID: \"635a2ae3-b676-441a-81f8-8037caefe261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc" Oct 30 17:16:42.675826 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.675807 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635a2ae3-b676-441a-81f8-8037caefe261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ed70e1c-k5wdc\" (UID: \"635a2ae3-b676-441a-81f8-8037caefe261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc" Oct 30 17:16:42.677512 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.677492 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635a2ae3-b676-441a-81f8-8037caefe261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ed70e1c-k5wdc\" (UID: \"635a2ae3-b676-441a-81f8-8037caefe261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc" Oct 30 17:16:42.696507 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.696481 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn" Oct 30 17:16:42.699189 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.699171 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66ckx\" (UniqueName: \"kubernetes.io/projected/635a2ae3-b676-441a-81f8-8037caefe261-kube-api-access-66ckx\") pod \"maas-api-revoke-sub-6ed70e1c-k5wdc\" (UID: \"635a2ae3-b676-441a-81f8-8037caefe261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc" Oct 30 17:16:42.781749 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.781712 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc" Oct 30 17:16:42.928473 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.928377 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:42.996610 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:16:42.996573 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f595f1_0e66_4066_af92_0ded0ada9ac1.slice/crio-6de7188d9d7d1d4a2f3fcaf45e9d47e30fd25ecb0da7203fa573ef0d98d4731a WatchSource:0}: Error finding container 6de7188d9d7d1d4a2f3fcaf45e9d47e30fd25ecb0da7203fa573ef0d98d4731a: Status 404 returned error can't find the container with id 6de7188d9d7d1d4a2f3fcaf45e9d47e30fd25ecb0da7203fa573ef0d98d4731a Oct 30 17:16:42.996797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:42.996683 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn"] Oct 30 17:16:43.054078 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:43.054038 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:43.088233 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:43.088200 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc"] Oct 30 17:16:43.088690 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:16:43.088667 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635a2ae3_b676_441a_81f8_8037caefe261.slice/crio-54c68a828190201d2fd94b5bd77ecd9de5a40b0b132b0b241d3e10c4a24bef72 WatchSource:0}: Error finding container 54c68a828190201d2fd94b5bd77ecd9de5a40b0b132b0b241d3e10c4a24bef72: Status 404 returned error can't find the container with id 54c68a828190201d2fd94b5bd77ecd9de5a40b0b132b0b241d3e10c4a24bef72 Oct 30 17:16:43.237644 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:43.237555 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:43.270522 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:43.270491 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28"] Oct 30 17:16:43.299654 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:43.299621 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx"] Oct 30 17:16:43.345044 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:43.345007 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn" event={"ID":"a9f595f1-0e66-4066-af92-0ded0ada9ac1","Type":"ContainerStarted","Data":"1554465d4e37aca41f321d597cfaf0eaf746142ea85fc4dfca6e848bb228a0e5"} Oct 30 17:16:43.345212 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:43.345077 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn" event={"ID":"a9f595f1-0e66-4066-af92-0ded0ada9ac1","Type":"ContainerStarted","Data":"6de7188d9d7d1d4a2f3fcaf45e9d47e30fd25ecb0da7203fa573ef0d98d4731a"} Oct 30 17:16:43.347685 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:43.347647 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc" event={"ID":"635a2ae3-b676-441a-81f8-8037caefe261","Type":"ContainerStarted","Data":"54c68a828190201d2fd94b5bd77ecd9de5a40b0b132b0b241d3e10c4a24bef72"} Oct 30 17:16:43.400118 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:43.400068 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn" podStartSLOduration=1.4000532159999999 podStartE2EDuration="1.400053216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:43.397632901 +0000 UTC m=+3317.108087646" watchObservedRunningTime="2026-10-30 17:16:43.400053216 +0000 UTC m=+3317.110507989" Oct 30 17:16:43.407149 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:43.407118 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:43.531489 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:43.531400 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:44.068879 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:16:44.068839 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:16:44.269972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:44.269876 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:44.382161 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:44.382122 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:44.384920 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:44.384887 6202 generic.go:356] "Generic (PLEG): container finished" podID="a9f595f1-0e66-4066-af92-0ded0ada9ac1" containerID="1554465d4e37aca41f321d597cfaf0eaf746142ea85fc4dfca6e848bb228a0e5" exitCode=0 Oct 30 17:16:44.385094 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:44.384981 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn" event={"ID":"a9f595f1-0e66-4066-af92-0ded0ada9ac1","Type":"ContainerDied","Data":"1554465d4e37aca41f321d597cfaf0eaf746142ea85fc4dfca6e848bb228a0e5"} Oct 30 17:16:44.387967 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:44.387929 6202 generic.go:356] "Generic (PLEG): container finished" podID="635a2ae3-b676-441a-81f8-8037caefe261" containerID="8d9bac77c7e666b0903b8aeb85684c1afb33df28bc320ddc530c9ca98cc3aeac" exitCode=0 Oct 30 17:16:44.388135 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:44.388091 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc" event={"ID":"635a2ae3-b676-441a-81f8-8037caefe261","Type":"ContainerDied","Data":"8d9bac77c7e666b0903b8aeb85684c1afb33df28bc320ddc530c9ca98cc3aeac"} Oct 30 17:16:44.496788 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:44.496736 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:44.614580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:44.614547 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:44.993920 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:44.993827 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:45.112939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.112910 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:45.125540 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.125511 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq"] Oct 30 17:16:45.149792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.146893 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ae5afd-s6zkq"] Oct 30 17:16:45.494972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.494656 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" podUID="1b94338f-f1d5-4e7c-880d-905f46fd9c7e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:45.495450 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.495382 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" podUID="f750fe64-42e5-4d6f-91f7-071c1493c881" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:45.572596 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.572569 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn" Oct 30 17:16:45.575871 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.575849 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc" Oct 30 17:16:45.596881 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.596854 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:45.722790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.722694 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/635a2ae3-b676-441a-81f8-8037caefe261-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635a2ae3-b676-441a-81f8-8037caefe261-maas-api-openshift-ca\") pod \"635a2ae3-b676-441a-81f8-8037caefe261\" (UID: \"635a2ae3-b676-441a-81f8-8037caefe261\") " Oct 30 17:16:45.723224 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.722889 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635a2ae3-b676-441a-81f8-8037caefe261-kube-api-access-66ckx\" (UniqueName: \"kubernetes.io/projected/635a2ae3-b676-441a-81f8-8037caefe261-kube-api-access-66ckx\") pod \"635a2ae3-b676-441a-81f8-8037caefe261\" (UID: \"635a2ae3-b676-441a-81f8-8037caefe261\") " Oct 30 17:16:45.723224 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.722942 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9f595f1-0e66-4066-af92-0ded0ada9ac1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f595f1-0e66-4066-af92-0ded0ada9ac1-maas-api-openshift-ca\") pod \"a9f595f1-0e66-4066-af92-0ded0ada9ac1\" (UID: \"a9f595f1-0e66-4066-af92-0ded0ada9ac1\") " Oct 30 17:16:45.723224 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.723046 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9f595f1-0e66-4066-af92-0ded0ada9ac1-kube-api-access-jjtzg\" (UniqueName: \"kubernetes.io/projected/a9f595f1-0e66-4066-af92-0ded0ada9ac1-kube-api-access-jjtzg\") pod \"a9f595f1-0e66-4066-af92-0ded0ada9ac1\" (UID: \"a9f595f1-0e66-4066-af92-0ded0ada9ac1\") " Oct 30 17:16:45.723224 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.723071 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635a2ae3-b676-441a-81f8-8037caefe261-maas-api-openshift-ca" pod "635a2ae3-b676-441a-81f8-8037caefe261" (UID: "635a2ae3-b676-441a-81f8-8037caefe261"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:45.723224 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.723144 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9f595f1-0e66-4066-af92-0ded0ada9ac1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f595f1-0e66-4066-af92-0ded0ada9ac1-maas-api-xks-ca\") pod \"a9f595f1-0e66-4066-af92-0ded0ada9ac1\" (UID: \"a9f595f1-0e66-4066-af92-0ded0ada9ac1\") " Oct 30 17:16:45.723224 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.723201 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/635a2ae3-b676-441a-81f8-8037caefe261-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635a2ae3-b676-441a-81f8-8037caefe261-maas-api-xks-ca\") pod \"635a2ae3-b676-441a-81f8-8037caefe261\" (UID: \"635a2ae3-b676-441a-81f8-8037caefe261\") " Oct 30 17:16:45.723558 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.723334 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f595f1-0e66-4066-af92-0ded0ada9ac1-maas-api-openshift-ca" pod "a9f595f1-0e66-4066-af92-0ded0ada9ac1" (UID: "a9f595f1-0e66-4066-af92-0ded0ada9ac1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:45.723890 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.723862 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f595f1-0e66-4066-af92-0ded0ada9ac1-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:45.723890 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.723892 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635a2ae3-b676-441a-81f8-8037caefe261-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:45.725603 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.725573 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9f595f1-0e66-4066-af92-0ded0ada9ac1-maas-api-xks-ca" pod "a9f595f1-0e66-4066-af92-0ded0ada9ac1" (UID: "a9f595f1-0e66-4066-af92-0ded0ada9ac1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:45.725747 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.725712 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635a2ae3-b676-441a-81f8-8037caefe261-kube-api-access-66ckx" pod "635a2ae3-b676-441a-81f8-8037caefe261" (UID: "635a2ae3-b676-441a-81f8-8037caefe261"). InnerVolumeSpecName "kube-api-access-66ckx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:45.725981 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.725958 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f595f1-0e66-4066-af92-0ded0ada9ac1-kube-api-access-jjtzg" pod "a9f595f1-0e66-4066-af92-0ded0ada9ac1" (UID: "a9f595f1-0e66-4066-af92-0ded0ada9ac1"). InnerVolumeSpecName "kube-api-access-jjtzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:45.726086 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.725958 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/635a2ae3-b676-441a-81f8-8037caefe261-maas-api-xks-ca" pod "635a2ae3-b676-441a-81f8-8037caefe261" (UID: "635a2ae3-b676-441a-81f8-8037caefe261"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:45.824955 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.824917 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjtzg\" (UniqueName: \"kubernetes.io/projected/a9f595f1-0e66-4066-af92-0ded0ada9ac1-kube-api-access-jjtzg\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:45.824955 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.824949 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f595f1-0e66-4066-af92-0ded0ada9ac1-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:45.824955 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.824958 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635a2ae3-b676-441a-81f8-8037caefe261-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:45.824955 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:45.824967 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66ckx\" (UniqueName: \"kubernetes.io/projected/635a2ae3-b676-441a-81f8-8037caefe261-kube-api-access-66ckx\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:46.408758 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.408725 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn" Oct 30 17:16:46.412413 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.412384 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc" Oct 30 17:16:46.413452 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.413421 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94fc04ec-c880-421c-9776-eb326ec1c557" path="/var/lib/kubelet/pods/94fc04ec-c880-421c-9776-eb326ec1c557/volumes" Oct 30 17:16:46.415177 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.415150 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn" event={"ID":"a9f595f1-0e66-4066-af92-0ded0ada9ac1","Type":"ContainerDied","Data":"6de7188d9d7d1d4a2f3fcaf45e9d47e30fd25ecb0da7203fa573ef0d98d4731a"} Oct 30 17:16:46.415319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.415200 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6de7188d9d7d1d4a2f3fcaf45e9d47e30fd25ecb0da7203fa573ef0d98d4731a" Oct 30 17:16:46.415319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.415214 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc" event={"ID":"635a2ae3-b676-441a-81f8-8037caefe261","Type":"ContainerDied","Data":"54c68a828190201d2fd94b5bd77ecd9de5a40b0b132b0b241d3e10c4a24bef72"} Oct 30 17:16:46.415319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.415237 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54c68a828190201d2fd94b5bd77ecd9de5a40b0b132b0b241d3e10c4a24bef72" Oct 30 17:16:46.449797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.448921 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k"] Oct 30 17:16:46.449797 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.449225 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" podUID="e38c4872-9809-48e2-a8f5-9a0076deddd5" containerName="istio-proxy" containerID="cri-o://e4a7280f35037c8160667af6852578229aa6fa3eda75cb06cf4c8189696e6c32" gracePeriod=30 Oct 30 17:16:46.474878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.474847 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:46.926461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.926425 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6"] Oct 30 17:16:46.927237 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.927211 6202 cpu_manager.go:414] "containerMap: removing container" podUID="635a2ae3-b676-441a-81f8-8037caefe261" containerName="revoke-keys" Oct 30 17:16:46.927299 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.927245 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635a2ae3-b676-441a-81f8-8037caefe261" containerName="revoke-keys" Oct 30 17:16:46.927336 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.927312 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a9f595f1-0e66-4066-af92-0ded0ada9ac1" containerName="revoke-keys" Oct 30 17:16:46.927336 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.927320 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f595f1-0e66-4066-af92-0ded0ada9ac1" containerName="revoke-keys" Oct 30 17:16:46.927468 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.927456 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f595f1-0e66-4066-af92-0ded0ada9ac1" containerName="revoke-keys" Oct 30 17:16:46.927505 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.927475 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="635a2ae3-b676-441a-81f8-8037caefe261" containerName="revoke-keys" Oct 30 17:16:46.930726 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.930703 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6" Oct 30 17:16:46.947156 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.947128 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:16:46.947325 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.947226 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:16:46.947435 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.947177 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:16:46.947435 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.947319 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:16:46.950245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:46.950043 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6"] Oct 30 17:16:47.040666 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:47.040628 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pwp8\" (UniqueName: \"kubernetes.io/projected/51806a0b-50a9-4d05-91db-3ba5340f3e07-kube-api-access-9pwp8\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6\" (UID: \"51806a0b-50a9-4d05-91db-3ba5340f3e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6" Oct 30 17:16:47.040877 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:47.040776 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51806a0b-50a9-4d05-91db-3ba5340f3e07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6\" (UID: \"51806a0b-50a9-4d05-91db-3ba5340f3e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6" Oct 30 17:16:47.040877 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:47.040834 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51806a0b-50a9-4d05-91db-3ba5340f3e07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6\" (UID: \"51806a0b-50a9-4d05-91db-3ba5340f3e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6" Oct 30 17:16:47.093333 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:47.093293 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:47.145217 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:47.145182 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51806a0b-50a9-4d05-91db-3ba5340f3e07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6\" (UID: \"51806a0b-50a9-4d05-91db-3ba5340f3e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6" Oct 30 17:16:47.145403 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:47.145256 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51806a0b-50a9-4d05-91db-3ba5340f3e07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6\" (UID: \"51806a0b-50a9-4d05-91db-3ba5340f3e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6" Oct 30 17:16:47.145403 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:47.145313 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pwp8\" (UniqueName: \"kubernetes.io/projected/51806a0b-50a9-4d05-91db-3ba5340f3e07-kube-api-access-9pwp8\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6\" (UID: \"51806a0b-50a9-4d05-91db-3ba5340f3e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6" Oct 30 17:16:47.148033 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:47.148004 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51806a0b-50a9-4d05-91db-3ba5340f3e07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6\" (UID: \"51806a0b-50a9-4d05-91db-3ba5340f3e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6" Oct 30 17:16:47.150272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:47.150248 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51806a0b-50a9-4d05-91db-3ba5340f3e07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6\" (UID: \"51806a0b-50a9-4d05-91db-3ba5340f3e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6" Oct 30 17:16:47.170865 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:47.170824 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pwp8\" (UniqueName: \"kubernetes.io/projected/51806a0b-50a9-4d05-91db-3ba5340f3e07-kube-api-access-9pwp8\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6\" (UID: \"51806a0b-50a9-4d05-91db-3ba5340f3e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6" Oct 30 17:16:47.246637 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:47.246557 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6" Oct 30 17:16:47.558034 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:16:47.558003 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51806a0b_50a9_4d05_91db_3ba5340f3e07.slice/crio-819138d1dbe7db6751845f2fda5cbd899dae8063b07e972227ac7059be8bc9ad WatchSource:0}: Error finding container 819138d1dbe7db6751845f2fda5cbd899dae8063b07e972227ac7059be8bc9ad: Status 404 returned error can't find the container with id 819138d1dbe7db6751845f2fda5cbd899dae8063b07e972227ac7059be8bc9ad Oct 30 17:16:47.559130 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:47.559097 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6"] Oct 30 17:16:48.271760 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:48.271652 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" podUID="3807a7b3-8948-416d-97fe-e94528e239d2" containerName="payload-processing" containerID="cri-o://771bb3e055ce6ba9ef32a1134ff9a5894652e6924b717421b4cb5e7ea4242003" gracePeriod=25 Oct 30 17:16:48.300659 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:48.300602 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" podUID="b123429e-9af8-402a-b91a-1a0c2fdc7697" containerName="payload-pre-processing" containerID="cri-o://267285a07aaec76c93ac695ab23734819cd5035e61889d38d0c8a760d990e767" gracePeriod=25 Oct 30 17:16:48.429348 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:48.429307 6202 generic.go:356] "Generic (PLEG): container finished" podID="51806a0b-50a9-4d05-91db-3ba5340f3e07" containerID="6f6dd964ac58f7a59e62f43c4ebb86cbe33edbe86a8ffe034239757da41885bd" exitCode=0 Oct 30 17:16:48.429530 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:48.429388 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6" event={"ID":"51806a0b-50a9-4d05-91db-3ba5340f3e07","Type":"ContainerDied","Data":"6f6dd964ac58f7a59e62f43c4ebb86cbe33edbe86a8ffe034239757da41885bd"} Oct 30 17:16:48.429530 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:48.429462 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6" event={"ID":"51806a0b-50a9-4d05-91db-3ba5340f3e07","Type":"ContainerStarted","Data":"819138d1dbe7db6751845f2fda5cbd899dae8063b07e972227ac7059be8bc9ad"} Oct 30 17:16:48.904027 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:48.903990 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:49.590360 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:49.590326 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6" Oct 30 17:16:49.674027 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:49.673989 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51806a0b-50a9-4d05-91db-3ba5340f3e07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51806a0b-50a9-4d05-91db-3ba5340f3e07-maas-api-openshift-ca\") pod \"51806a0b-50a9-4d05-91db-3ba5340f3e07\" (UID: \"51806a0b-50a9-4d05-91db-3ba5340f3e07\") " Oct 30 17:16:49.674238 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:49.674172 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51806a0b-50a9-4d05-91db-3ba5340f3e07-kube-api-access-9pwp8\" (UniqueName: \"kubernetes.io/projected/51806a0b-50a9-4d05-91db-3ba5340f3e07-kube-api-access-9pwp8\") pod \"51806a0b-50a9-4d05-91db-3ba5340f3e07\" (UID: \"51806a0b-50a9-4d05-91db-3ba5340f3e07\") " Oct 30 17:16:49.674314 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:49.674269 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51806a0b-50a9-4d05-91db-3ba5340f3e07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51806a0b-50a9-4d05-91db-3ba5340f3e07-maas-api-xks-ca\") pod \"51806a0b-50a9-4d05-91db-3ba5340f3e07\" (UID: \"51806a0b-50a9-4d05-91db-3ba5340f3e07\") " Oct 30 17:16:49.674480 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:49.674447 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51806a0b-50a9-4d05-91db-3ba5340f3e07-maas-api-openshift-ca" pod "51806a0b-50a9-4d05-91db-3ba5340f3e07" (UID: "51806a0b-50a9-4d05-91db-3ba5340f3e07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:49.676652 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:49.676620 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51806a0b-50a9-4d05-91db-3ba5340f3e07-maas-api-xks-ca" pod "51806a0b-50a9-4d05-91db-3ba5340f3e07" (UID: "51806a0b-50a9-4d05-91db-3ba5340f3e07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:49.676936 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:49.676912 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51806a0b-50a9-4d05-91db-3ba5340f3e07-kube-api-access-9pwp8" pod "51806a0b-50a9-4d05-91db-3ba5340f3e07" (UID: "51806a0b-50a9-4d05-91db-3ba5340f3e07"). InnerVolumeSpecName "kube-api-access-9pwp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:49.775556 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:49.775448 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pwp8\" (UniqueName: \"kubernetes.io/projected/51806a0b-50a9-4d05-91db-3ba5340f3e07-kube-api-access-9pwp8\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:49.775556 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:49.775489 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51806a0b-50a9-4d05-91db-3ba5340f3e07-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:49.775556 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:49.775499 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51806a0b-50a9-4d05-91db-3ba5340f3e07-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:50.383610 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:50.383557 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:50.446647 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:50.446618 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6" Oct 30 17:16:50.446853 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:50.446652 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6" event={"ID":"51806a0b-50a9-4d05-91db-3ba5340f3e07","Type":"ContainerDied","Data":"819138d1dbe7db6751845f2fda5cbd899dae8063b07e972227ac7059be8bc9ad"} Oct 30 17:16:50.446853 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:50.446699 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="819138d1dbe7db6751845f2fda5cbd899dae8063b07e972227ac7059be8bc9ad" Oct 30 17:16:50.881985 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:50.881945 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:51.019868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.019821 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:51.022117 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.022059 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g"] Oct 30 17:16:51.034700 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.034671 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0b2348-bgb2g"] Oct 30 17:16:51.312356 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.312212 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:51.732088 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.732062 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 17:16:51.902580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.902548 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e38c4872-9809-48e2-a8f5-9a0076deddd5-kube-api-access-8bmg8\" (UniqueName: \"kubernetes.io/projected/e38c4872-9809-48e2-a8f5-9a0076deddd5-kube-api-access-8bmg8\") pod \"e38c4872-9809-48e2-a8f5-9a0076deddd5\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " Oct 30 17:16:51.902580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.902586 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e38c4872-9809-48e2-a8f5-9a0076deddd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e38c4872-9809-48e2-a8f5-9a0076deddd5-istiod-ca-cert\") pod \"e38c4872-9809-48e2-a8f5-9a0076deddd5\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " Oct 30 17:16:51.903190 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.902612 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-token\" (UniqueName: \"kubernetes.io/projected/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-token\") pod \"e38c4872-9809-48e2-a8f5-9a0076deddd5\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " Oct 30 17:16:51.903190 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.902733 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-workload-socket\") pod \"e38c4872-9809-48e2-a8f5-9a0076deddd5\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " Oct 30 17:16:51.903190 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.902827 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-credential-socket\") pod \"e38c4872-9809-48e2-a8f5-9a0076deddd5\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " Oct 30 17:16:51.903190 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.902871 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-podinfo\") pod \"e38c4872-9809-48e2-a8f5-9a0076deddd5\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " Oct 30 17:16:51.903190 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.902919 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-envoy\") pod \"e38c4872-9809-48e2-a8f5-9a0076deddd5\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " Oct 30 17:16:51.903190 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.902985 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-workload-certs\") pod \"e38c4872-9809-48e2-a8f5-9a0076deddd5\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " Oct 30 17:16:51.903190 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.902993 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e38c4872-9809-48e2-a8f5-9a0076deddd5-istiod-ca-cert" pod "e38c4872-9809-48e2-a8f5-9a0076deddd5" (UID: "e38c4872-9809-48e2-a8f5-9a0076deddd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:51.903190 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.903029 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-data\") pod \"e38c4872-9809-48e2-a8f5-9a0076deddd5\" (UID: \"e38c4872-9809-48e2-a8f5-9a0076deddd5\") " Oct 30 17:16:51.903190 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.903139 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-credential-socket" pod "e38c4872-9809-48e2-a8f5-9a0076deddd5" (UID: "e38c4872-9809-48e2-a8f5-9a0076deddd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:51.903190 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.903152 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-workload-socket" pod "e38c4872-9809-48e2-a8f5-9a0076deddd5" (UID: "e38c4872-9809-48e2-a8f5-9a0076deddd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:51.903966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.903343 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-workload-certs" pod "e38c4872-9809-48e2-a8f5-9a0076deddd5" (UID: "e38c4872-9809-48e2-a8f5-9a0076deddd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:51.903966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.903463 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-data" pod "e38c4872-9809-48e2-a8f5-9a0076deddd5" (UID: "e38c4872-9809-48e2-a8f5-9a0076deddd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:51.903966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.903829 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e38c4872-9809-48e2-a8f5-9a0076deddd5-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:51.903966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.903853 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:51.903966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.903868 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:51.903966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.903890 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:51.903966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.903906 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:51.905191 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.905163 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-envoy" pod "e38c4872-9809-48e2-a8f5-9a0076deddd5" (UID: "e38c4872-9809-48e2-a8f5-9a0076deddd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:51.905313 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.905201 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-token" pod "e38c4872-9809-48e2-a8f5-9a0076deddd5" (UID: "e38c4872-9809-48e2-a8f5-9a0076deddd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:51.905313 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.905270 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-podinfo" pod "e38c4872-9809-48e2-a8f5-9a0076deddd5" (UID: "e38c4872-9809-48e2-a8f5-9a0076deddd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:16:51.905399 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:51.905313 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e38c4872-9809-48e2-a8f5-9a0076deddd5-kube-api-access-8bmg8" pod "e38c4872-9809-48e2-a8f5-9a0076deddd5" (UID: "e38c4872-9809-48e2-a8f5-9a0076deddd5"). InnerVolumeSpecName "kube-api-access-8bmg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:52.005083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.005045 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:52.005083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.005077 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bmg8\" (UniqueName: \"kubernetes.io/projected/e38c4872-9809-48e2-a8f5-9a0076deddd5-kube-api-access-8bmg8\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:52.005083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.005088 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:52.005316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.005097 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e38c4872-9809-48e2-a8f5-9a0076deddd5-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:52.107207 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.107168 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:52.223022 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.222935 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:52.399835 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.399799 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j"] Oct 30 17:16:52.400876 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.400338 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" podUID="5018b303-b054-4954-a781-39523acd950f" containerName="istio-proxy" containerID="cri-o://f0d0b3293d58a510c5c3d9b40d5b680f266ba358017621f167787089cac14d82" gracePeriod=30 Oct 30 17:16:52.416405 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.416376 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b79b467-33a3-45f3-8523-a2eda2ef2bca" path="/var/lib/kubelet/pods/3b79b467-33a3-45f3-8523-a2eda2ef2bca/volumes" Oct 30 17:16:52.442624 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.442590 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:52.467416 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.467375 6202 generic.go:356] "Generic (PLEG): container finished" podID="e38c4872-9809-48e2-a8f5-9a0076deddd5" containerID="e4a7280f35037c8160667af6852578229aa6fa3eda75cb06cf4c8189696e6c32" exitCode=0 Oct 30 17:16:52.467615 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.467431 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" Oct 30 17:16:52.467615 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.467419 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" event={"ID":"e38c4872-9809-48e2-a8f5-9a0076deddd5","Type":"ContainerDied","Data":"e4a7280f35037c8160667af6852578229aa6fa3eda75cb06cf4c8189696e6c32"} Oct 30 17:16:52.467728 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.467640 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k" event={"ID":"e38c4872-9809-48e2-a8f5-9a0076deddd5","Type":"ContainerDied","Data":"2e161751cd9f1d2efb2d71f150bf5efe3d93615fa9ce36a45251cb187e7d6e1d"} Oct 30 17:16:52.467728 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.467675 6202 scope.go:122] "RemoveContainer" containerID="e4a7280f35037c8160667af6852578229aa6fa3eda75cb06cf4c8189696e6c32" Oct 30 17:16:52.483450 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.483427 6202 scope.go:122] "RemoveContainer" containerID="e4a7280f35037c8160667af6852578229aa6fa3eda75cb06cf4c8189696e6c32" Oct 30 17:16:52.483830 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:16:52.483793 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a7280f35037c8160667af6852578229aa6fa3eda75cb06cf4c8189696e6c32\": container with ID starting with e4a7280f35037c8160667af6852578229aa6fa3eda75cb06cf4c8189696e6c32 not found: ID does not exist" containerID="e4a7280f35037c8160667af6852578229aa6fa3eda75cb06cf4c8189696e6c32" Oct 30 17:16:52.483950 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.483834 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a7280f35037c8160667af6852578229aa6fa3eda75cb06cf4c8189696e6c32"} err="failed to get container status \"e4a7280f35037c8160667af6852578229aa6fa3eda75cb06cf4c8189696e6c32\": rpc error: code = NotFound desc = could not find container \"e4a7280f35037c8160667af6852578229aa6fa3eda75cb06cf4c8189696e6c32\": container with ID starting with e4a7280f35037c8160667af6852578229aa6fa3eda75cb06cf4c8189696e6c32 not found: ID does not exist" Oct 30 17:16:52.545904 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.545830 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k"] Oct 30 17:16:52.547327 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.547271 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-mzm7k"] Oct 30 17:16:52.878002 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.877951 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:52.990093 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:52.990065 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:53.094607 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.094575 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl"] Oct 30 17:16:53.095153 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.095122 6202 cpu_manager.go:414] "containerMap: removing container" podUID="e38c4872-9809-48e2-a8f5-9a0076deddd5" containerName="istio-proxy" Oct 30 17:16:53.095153 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.095141 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e38c4872-9809-48e2-a8f5-9a0076deddd5" containerName="istio-proxy" Oct 30 17:16:53.095379 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.095195 6202 cpu_manager.go:414] "containerMap: removing container" podUID="51806a0b-50a9-4d05-91db-3ba5340f3e07" containerName="revoke-keys" Oct 30 17:16:53.095379 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.095203 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51806a0b-50a9-4d05-91db-3ba5340f3e07" containerName="revoke-keys" Oct 30 17:16:53.095379 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.095322 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="51806a0b-50a9-4d05-91db-3ba5340f3e07" containerName="revoke-keys" Oct 30 17:16:53.095379 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.095342 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="e38c4872-9809-48e2-a8f5-9a0076deddd5" containerName="istio-proxy" Oct 30 17:16:53.098395 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.098373 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl" Oct 30 17:16:53.115798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.115758 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:16:53.115798 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.115783 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:16:53.116219 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.116181 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:16:53.116342 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.116270 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:16:53.118611 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.118586 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl"] Oct 30 17:16:53.224315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.224234 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d92701-cab0-4360-b5b5-87d49d16ec10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl\" (UID: \"99d92701-cab0-4360-b5b5-87d49d16ec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl" Oct 30 17:16:53.224467 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.224373 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9n5w\" (UniqueName: \"kubernetes.io/projected/99d92701-cab0-4360-b5b5-87d49d16ec10-kube-api-access-s9n5w\") pod \"maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl\" (UID: \"99d92701-cab0-4360-b5b5-87d49d16ec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl" Oct 30 17:16:53.224467 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.224413 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d92701-cab0-4360-b5b5-87d49d16ec10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl\" (UID: \"99d92701-cab0-4360-b5b5-87d49d16ec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl" Oct 30 17:16:53.325383 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.325346 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9n5w\" (UniqueName: \"kubernetes.io/projected/99d92701-cab0-4360-b5b5-87d49d16ec10-kube-api-access-s9n5w\") pod \"maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl\" (UID: \"99d92701-cab0-4360-b5b5-87d49d16ec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl" Oct 30 17:16:53.325581 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.325394 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d92701-cab0-4360-b5b5-87d49d16ec10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl\" (UID: \"99d92701-cab0-4360-b5b5-87d49d16ec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl" Oct 30 17:16:53.325581 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.325450 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d92701-cab0-4360-b5b5-87d49d16ec10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl\" (UID: \"99d92701-cab0-4360-b5b5-87d49d16ec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl" Oct 30 17:16:53.326088 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.326068 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d92701-cab0-4360-b5b5-87d49d16ec10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl\" (UID: \"99d92701-cab0-4360-b5b5-87d49d16ec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl" Oct 30 17:16:53.327967 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.327944 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d92701-cab0-4360-b5b5-87d49d16ec10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl\" (UID: \"99d92701-cab0-4360-b5b5-87d49d16ec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl" Oct 30 17:16:53.350646 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.350609 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9n5w\" (UniqueName: \"kubernetes.io/projected/99d92701-cab0-4360-b5b5-87d49d16ec10-kube-api-access-s9n5w\") pod \"maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl\" (UID: \"99d92701-cab0-4360-b5b5-87d49d16ec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl" Oct 30 17:16:53.411415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.411378 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl" Oct 30 17:16:53.709724 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:16:53.709693 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d92701_cab0_4360_b5b5_87d49d16ec10.slice/crio-7088be3f727260dfb2e58447577d4fe2b9a6331258e42522d30b494005ed8b15 WatchSource:0}: Error finding container 7088be3f727260dfb2e58447577d4fe2b9a6331258e42522d30b494005ed8b15: Status 404 returned error can't find the container with id 7088be3f727260dfb2e58447577d4fe2b9a6331258e42522d30b494005ed8b15 Oct 30 17:16:53.709965 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:53.709941 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl"] Oct 30 17:16:54.410640 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:54.410606 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e38c4872-9809-48e2-a8f5-9a0076deddd5" path="/var/lib/kubelet/pods/e38c4872-9809-48e2-a8f5-9a0076deddd5/volumes" Oct 30 17:16:54.501683 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:54.501641 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl" event={"ID":"99d92701-cab0-4360-b5b5-87d49d16ec10","Type":"ContainerStarted","Data":"f21995a9a5abd0fb09efc815d3f43dad289b981dcf7783fa5d90347a2cf8e0f0"} Oct 30 17:16:54.501890 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:54.501701 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl" event={"ID":"99d92701-cab0-4360-b5b5-87d49d16ec10","Type":"ContainerStarted","Data":"7088be3f727260dfb2e58447577d4fe2b9a6331258e42522d30b494005ed8b15"} Oct 30 17:16:54.544869 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:54.544753 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl" podStartSLOduration=1.5447348 podStartE2EDuration="1.5447348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:54.543620094 +0000 UTC m=+3328.254074842" watchObservedRunningTime="2026-10-30 17:16:54.5447348 +0000 UTC m=+3328.255189545" Oct 30 17:16:55.145286 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:55.145249 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh"] Oct 30 17:16:55.148365 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:55.148341 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh" Oct 30 17:16:55.174856 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:55.174825 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh"] Oct 30 17:16:55.249307 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:55.249272 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt42x\" (UniqueName: \"kubernetes.io/projected/337af4a5-d6e2-46b3-b89a-b4955cc92da2-kube-api-access-tt42x\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh\" (UID: \"337af4a5-d6e2-46b3-b89a-b4955cc92da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh" Oct 30 17:16:55.249490 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:55.249402 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337af4a5-d6e2-46b3-b89a-b4955cc92da2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh\" (UID: \"337af4a5-d6e2-46b3-b89a-b4955cc92da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh" Oct 30 17:16:55.249490 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:55.249477 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337af4a5-d6e2-46b3-b89a-b4955cc92da2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh\" (UID: \"337af4a5-d6e2-46b3-b89a-b4955cc92da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh" Oct 30 17:16:55.355397 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:55.354512 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337af4a5-d6e2-46b3-b89a-b4955cc92da2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh\" (UID: \"337af4a5-d6e2-46b3-b89a-b4955cc92da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh" Oct 30 17:16:55.355397 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:55.354600 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337af4a5-d6e2-46b3-b89a-b4955cc92da2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh\" (UID: \"337af4a5-d6e2-46b3-b89a-b4955cc92da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh" Oct 30 17:16:55.355397 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:55.354704 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt42x\" (UniqueName: \"kubernetes.io/projected/337af4a5-d6e2-46b3-b89a-b4955cc92da2-kube-api-access-tt42x\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh\" (UID: \"337af4a5-d6e2-46b3-b89a-b4955cc92da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh" Oct 30 17:16:55.356796 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:55.356381 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337af4a5-d6e2-46b3-b89a-b4955cc92da2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh\" (UID: \"337af4a5-d6e2-46b3-b89a-b4955cc92da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh" Oct 30 17:16:55.374366 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:55.370831 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337af4a5-d6e2-46b3-b89a-b4955cc92da2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh\" (UID: \"337af4a5-d6e2-46b3-b89a-b4955cc92da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh" Oct 30 17:16:55.385805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:55.383830 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt42x\" (UniqueName: \"kubernetes.io/projected/337af4a5-d6e2-46b3-b89a-b4955cc92da2-kube-api-access-tt42x\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh\" (UID: \"337af4a5-d6e2-46b3-b89a-b4955cc92da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh" Oct 30 17:16:55.461809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:55.461719 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh" Oct 30 17:16:55.497323 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:55.497283 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" podUID="1b94338f-f1d5-4e7c-880d-905f46fd9c7e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:55.499743 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:55.499706 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" podUID="f750fe64-42e5-4d6f-91f7-071c1493c881" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:55.513090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:55.513052 6202 generic.go:356] "Generic (PLEG): container finished" podID="99d92701-cab0-4360-b5b5-87d49d16ec10" containerID="f21995a9a5abd0fb09efc815d3f43dad289b981dcf7783fa5d90347a2cf8e0f0" exitCode=0 Oct 30 17:16:55.513284 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:55.513148 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl" event={"ID":"99d92701-cab0-4360-b5b5-87d49d16ec10","Type":"ContainerDied","Data":"f21995a9a5abd0fb09efc815d3f43dad289b981dcf7783fa5d90347a2cf8e0f0"} Oct 30 17:16:55.770986 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:55.770952 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh"] Oct 30 17:16:55.771943 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:16:55.771907 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337af4a5_d6e2_46b3_b89a_b4955cc92da2.slice/crio-6b0d8388a0ee9afd3793321cfa0d8f59999f0c7ccfe547fd6ff78460da1bb592 WatchSource:0}: Error finding container 6b0d8388a0ee9afd3793321cfa0d8f59999f0c7ccfe547fd6ff78460da1bb592: Status 404 returned error can't find the container with id 6b0d8388a0ee9afd3793321cfa0d8f59999f0c7ccfe547fd6ff78460da1bb592 Oct 30 17:16:56.521152 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:56.521115 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:56.524685 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:56.524647 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh" event={"ID":"337af4a5-d6e2-46b3-b89a-b4955cc92da2","Type":"ContainerStarted","Data":"3188b4d58a9d8439a45a93d0a07cabdb6db911183e052d205d9d116ce73d4468"} Oct 30 17:16:56.524864 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:56.524713 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh" event={"ID":"337af4a5-d6e2-46b3-b89a-b4955cc92da2","Type":"ContainerStarted","Data":"6b0d8388a0ee9afd3793321cfa0d8f59999f0c7ccfe547fd6ff78460da1bb592"} Oct 30 17:16:56.568791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:56.568564 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh" podStartSLOduration=1.568543466 podStartE2EDuration="1.568543466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:56.567335424 +0000 UTC m=+3330.277790170" watchObservedRunningTime="2026-10-30 17:16:56.568543466 +0000 UTC m=+3330.278998212" Oct 30 17:16:56.643847 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:56.642340 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:56.723980 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:56.723957 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl" Oct 30 17:16:56.882922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:56.880062 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d92701-cab0-4360-b5b5-87d49d16ec10-kube-api-access-s9n5w\" (UniqueName: \"kubernetes.io/projected/99d92701-cab0-4360-b5b5-87d49d16ec10-kube-api-access-s9n5w\") pod \"99d92701-cab0-4360-b5b5-87d49d16ec10\" (UID: \"99d92701-cab0-4360-b5b5-87d49d16ec10\") " Oct 30 17:16:56.882922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:56.880131 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99d92701-cab0-4360-b5b5-87d49d16ec10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d92701-cab0-4360-b5b5-87d49d16ec10-maas-api-xks-ca\") pod \"99d92701-cab0-4360-b5b5-87d49d16ec10\" (UID: \"99d92701-cab0-4360-b5b5-87d49d16ec10\") " Oct 30 17:16:56.882922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:56.880200 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99d92701-cab0-4360-b5b5-87d49d16ec10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d92701-cab0-4360-b5b5-87d49d16ec10-maas-api-openshift-ca\") pod \"99d92701-cab0-4360-b5b5-87d49d16ec10\" (UID: \"99d92701-cab0-4360-b5b5-87d49d16ec10\") " Oct 30 17:16:56.882922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:56.880945 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d92701-cab0-4360-b5b5-87d49d16ec10-maas-api-openshift-ca" pod "99d92701-cab0-4360-b5b5-87d49d16ec10" (UID: "99d92701-cab0-4360-b5b5-87d49d16ec10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:56.885889 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:56.885842 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d92701-cab0-4360-b5b5-87d49d16ec10-maas-api-xks-ca" pod "99d92701-cab0-4360-b5b5-87d49d16ec10" (UID: "99d92701-cab0-4360-b5b5-87d49d16ec10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:56.889430 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:56.889401 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d92701-cab0-4360-b5b5-87d49d16ec10-kube-api-access-s9n5w" pod "99d92701-cab0-4360-b5b5-87d49d16ec10" (UID: "99d92701-cab0-4360-b5b5-87d49d16ec10"). InnerVolumeSpecName "kube-api-access-s9n5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:56.914980 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:56.914947 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" podUID="b123429e-9af8-402a-b91a-1a0c2fdc7697" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:56.915097 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:56.915009 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" podUID="3807a7b3-8948-416d-97fe-e94528e239d2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:56.977638 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:56.977603 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:56.982084 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:56.982050 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d92701-cab0-4360-b5b5-87d49d16ec10-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:56.982084 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:56.982080 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d92701-cab0-4360-b5b5-87d49d16ec10-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:56.982084 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:56.982090 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9n5w\" (UniqueName: \"kubernetes.io/projected/99d92701-cab0-4360-b5b5-87d49d16ec10-kube-api-access-s9n5w\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:57.092684 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:57.092641 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:57.099894 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:57.099860 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd"] Oct 30 17:16:57.102907 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:57.102887 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-263f00-bwqmd"] Oct 30 17:16:57.208278 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:57.208181 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:57.532624 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:57.532538 6202 generic.go:356] "Generic (PLEG): container finished" podID="5018b303-b054-4954-a781-39523acd950f" containerID="f0d0b3293d58a510c5c3d9b40d5b680f266ba358017621f167787089cac14d82" exitCode=0 Oct 30 17:16:57.532624 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:57.532612 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" event={"ID":"5018b303-b054-4954-a781-39523acd950f","Type":"ContainerDied","Data":"f0d0b3293d58a510c5c3d9b40d5b680f266ba358017621f167787089cac14d82"} Oct 30 17:16:57.533163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:57.532676 6202 scope.go:122] "RemoveContainer" containerID="a9b9a6bc6f76cc0355d6d312416ef6df123cf4decec8bbfd6d84590129cbd548" Oct 30 17:16:57.534229 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:57.534207 6202 generic.go:356] "Generic (PLEG): container finished" podID="337af4a5-d6e2-46b3-b89a-b4955cc92da2" containerID="3188b4d58a9d8439a45a93d0a07cabdb6db911183e052d205d9d116ce73d4468" exitCode=0 Oct 30 17:16:57.534334 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:57.534296 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh" event={"ID":"337af4a5-d6e2-46b3-b89a-b4955cc92da2","Type":"ContainerDied","Data":"3188b4d58a9d8439a45a93d0a07cabdb6db911183e052d205d9d116ce73d4468"} Oct 30 17:16:57.535742 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:57.535724 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl" Oct 30 17:16:57.535857 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:57.535726 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl" event={"ID":"99d92701-cab0-4360-b5b5-87d49d16ec10","Type":"ContainerDied","Data":"7088be3f727260dfb2e58447577d4fe2b9a6331258e42522d30b494005ed8b15"} Oct 30 17:16:57.535857 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:57.535833 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7088be3f727260dfb2e58447577d4fe2b9a6331258e42522d30b494005ed8b15" Oct 30 17:16:57.595457 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:57.595419 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:57.884273 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:57.884241 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2"] Oct 30 17:16:57.925373 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:57.925338 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj"] Oct 30 17:16:57.996878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:57.994269 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:58.106204 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.103138 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" Oct 30 17:16:58.120571 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.119225 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:58.157810 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.155806 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" Oct 30 17:16:58.196100 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.196073 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f750fe64-42e5-4d6f-91f7-071c1493c881-kube-api-access-m2p77\" (UniqueName: \"kubernetes.io/projected/f750fe64-42e5-4d6f-91f7-071c1493c881-kube-api-access-m2p77\") pod \"f750fe64-42e5-4d6f-91f7-071c1493c881\" (UID: \"f750fe64-42e5-4d6f-91f7-071c1493c881\") " Oct 30 17:16:58.196262 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.196134 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f750fe64-42e5-4d6f-91f7-071c1493c881-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f750fe64-42e5-4d6f-91f7-071c1493c881-plugins-config-volume\") pod \"f750fe64-42e5-4d6f-91f7-071c1493c881\" (UID: \"f750fe64-42e5-4d6f-91f7-071c1493c881\") " Oct 30 17:16:58.196510 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.196487 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f750fe64-42e5-4d6f-91f7-071c1493c881-plugins-config-volume" pod "f750fe64-42e5-4d6f-91f7-071c1493c881" (UID: "f750fe64-42e5-4d6f-91f7-071c1493c881"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:58.196637 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.196613 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f750fe64-42e5-4d6f-91f7-071c1493c881-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:58.202276 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.202186 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f750fe64-42e5-4d6f-91f7-071c1493c881-kube-api-access-m2p77" pod "f750fe64-42e5-4d6f-91f7-071c1493c881" (UID: "f750fe64-42e5-4d6f-91f7-071c1493c881"). InnerVolumeSpecName "kube-api-access-m2p77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:58.208454 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.208432 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:16:58.297243 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.297211 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-istio-data\") pod \"5018b303-b054-4954-a781-39523acd950f\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " Oct 30 17:16:58.297420 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.297293 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5018b303-b054-4954-a781-39523acd950f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5018b303-b054-4954-a781-39523acd950f-istiod-ca-cert\") pod \"5018b303-b054-4954-a781-39523acd950f\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " Oct 30 17:16:58.297420 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.297332 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5018b303-b054-4954-a781-39523acd950f-kube-api-access-4hpxv\" (UniqueName: \"kubernetes.io/projected/5018b303-b054-4954-a781-39523acd950f-kube-api-access-4hpxv\") pod \"5018b303-b054-4954-a781-39523acd950f\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " Oct 30 17:16:58.297420 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.297358 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5018b303-b054-4954-a781-39523acd950f-istio-token\" (UniqueName: \"kubernetes.io/projected/5018b303-b054-4954-a781-39523acd950f-istio-token\") pod \"5018b303-b054-4954-a781-39523acd950f\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " Oct 30 17:16:58.297420 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.297394 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-workload-socket\") pod \"5018b303-b054-4954-a781-39523acd950f\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " Oct 30 17:16:58.297662 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.297423 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-istio-envoy\") pod \"5018b303-b054-4954-a781-39523acd950f\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " Oct 30 17:16:58.297662 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.297480 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b94338f-f1d5-4e7c-880d-905f46fd9c7e-kube-api-access-7pndw\" (UniqueName: \"kubernetes.io/projected/1b94338f-f1d5-4e7c-880d-905f46fd9c7e-kube-api-access-7pndw\") pod \"1b94338f-f1d5-4e7c-880d-905f46fd9c7e\" (UID: \"1b94338f-f1d5-4e7c-880d-905f46fd9c7e\") " Oct 30 17:16:58.297662 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.297511 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-istio-data" pod "5018b303-b054-4954-a781-39523acd950f" (UID: "5018b303-b054-4954-a781-39523acd950f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:58.297662 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.297596 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5018b303-b054-4954-a781-39523acd950f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5018b303-b054-4954-a781-39523acd950f-istio-podinfo\") pod \"5018b303-b054-4954-a781-39523acd950f\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " Oct 30 17:16:58.297662 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.297657 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-workload-certs\") pod \"5018b303-b054-4954-a781-39523acd950f\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " Oct 30 17:16:58.297968 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.297696 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-credential-socket\") pod \"5018b303-b054-4954-a781-39523acd950f\" (UID: \"5018b303-b054-4954-a781-39523acd950f\") " Oct 30 17:16:58.297968 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.297726 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b94338f-f1d5-4e7c-880d-905f46fd9c7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b94338f-f1d5-4e7c-880d-905f46fd9c7e-plugins-config-volume\") pod \"1b94338f-f1d5-4e7c-880d-905f46fd9c7e\" (UID: \"1b94338f-f1d5-4e7c-880d-905f46fd9c7e\") " Oct 30 17:16:58.297968 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.297821 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-workload-socket" pod "5018b303-b054-4954-a781-39523acd950f" (UID: "5018b303-b054-4954-a781-39523acd950f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:58.297968 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.297871 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5018b303-b054-4954-a781-39523acd950f-istiod-ca-cert" pod "5018b303-b054-4954-a781-39523acd950f" (UID: "5018b303-b054-4954-a781-39523acd950f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:58.298159 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.298034 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-workload-certs" pod "5018b303-b054-4954-a781-39523acd950f" (UID: "5018b303-b054-4954-a781-39523acd950f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:58.298159 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.298079 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-credential-socket" pod "5018b303-b054-4954-a781-39523acd950f" (UID: "5018b303-b054-4954-a781-39523acd950f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:58.298272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.298221 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:58.298272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.298235 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2p77\" (UniqueName: \"kubernetes.io/projected/f750fe64-42e5-4d6f-91f7-071c1493c881-kube-api-access-m2p77\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:58.298272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.298245 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:58.298272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.298253 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:58.298272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.298264 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:58.298730 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.298278 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5018b303-b054-4954-a781-39523acd950f-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:58.298730 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.298531 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b94338f-f1d5-4e7c-880d-905f46fd9c7e-plugins-config-volume" pod "1b94338f-f1d5-4e7c-880d-905f46fd9c7e" (UID: "1b94338f-f1d5-4e7c-880d-905f46fd9c7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:58.300072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.300041 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b94338f-f1d5-4e7c-880d-905f46fd9c7e-kube-api-access-7pndw" pod "1b94338f-f1d5-4e7c-880d-905f46fd9c7e" (UID: "1b94338f-f1d5-4e7c-880d-905f46fd9c7e"). InnerVolumeSpecName "kube-api-access-7pndw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:58.300072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.300051 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5018b303-b054-4954-a781-39523acd950f-istio-podinfo" pod "5018b303-b054-4954-a781-39523acd950f" (UID: "5018b303-b054-4954-a781-39523acd950f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:16:58.300351 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.300335 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5018b303-b054-4954-a781-39523acd950f-kube-api-access-4hpxv" pod "5018b303-b054-4954-a781-39523acd950f" (UID: "5018b303-b054-4954-a781-39523acd950f"). InnerVolumeSpecName "kube-api-access-4hpxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:58.300464 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.300443 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-istio-envoy" pod "5018b303-b054-4954-a781-39523acd950f" (UID: "5018b303-b054-4954-a781-39523acd950f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:58.300842 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.300822 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5018b303-b054-4954-a781-39523acd950f-istio-token" pod "5018b303-b054-4954-a781-39523acd950f" (UID: "5018b303-b054-4954-a781-39523acd950f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:58.399932 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.399833 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5018b303-b054-4954-a781-39523acd950f-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:58.399932 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.399868 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b94338f-f1d5-4e7c-880d-905f46fd9c7e-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:58.399932 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.399881 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hpxv\" (UniqueName: \"kubernetes.io/projected/5018b303-b054-4954-a781-39523acd950f-kube-api-access-4hpxv\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:58.399932 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.399902 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5018b303-b054-4954-a781-39523acd950f-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:58.399932 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.399924 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5018b303-b054-4954-a781-39523acd950f-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:58.400269 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.399935 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pndw\" (UniqueName: \"kubernetes.io/projected/1b94338f-f1d5-4e7c-880d-905f46fd9c7e-kube-api-access-7pndw\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:58.410195 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.410154 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ef8a93-8965-43f9-a86e-f2101beef171" path="/var/lib/kubelet/pods/75ef8a93-8965-43f9-a86e-f2101beef171/volumes" Oct 30 17:16:58.546929 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.546892 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" Oct 30 17:16:58.547370 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.546893 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j" event={"ID":"5018b303-b054-4954-a781-39523acd950f","Type":"ContainerDied","Data":"5eb212cbaa11adfc28c72e813d7977d0c10125e09b4e1943da647863ddd6052f"} Oct 30 17:16:58.547370 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.547043 6202 scope.go:122] "RemoveContainer" containerID="f0d0b3293d58a510c5c3d9b40d5b680f266ba358017621f167787089cac14d82" Oct 30 17:16:58.549872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.549828 6202 generic.go:356] "Generic (PLEG): container finished" podID="1b94338f-f1d5-4e7c-880d-905f46fd9c7e" containerID="f778a7ac0ea6b96d017c00add1efc0dbb8b7fa2135dae671c99b669e524868aa" exitCode=0 Oct 30 17:16:58.550017 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.549956 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" event={"ID":"1b94338f-f1d5-4e7c-880d-905f46fd9c7e","Type":"ContainerDied","Data":"f778a7ac0ea6b96d017c00add1efc0dbb8b7fa2135dae671c99b669e524868aa"} Oct 30 17:16:58.550017 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.550001 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" event={"ID":"1b94338f-f1d5-4e7c-880d-905f46fd9c7e","Type":"ContainerDied","Data":"1b00410367b983a4983853129c3023a732cb84c0a42d16626183f928ae657ad4"} Oct 30 17:16:58.550169 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.550092 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf" Oct 30 17:16:58.554449 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.552882 6202 generic.go:356] "Generic (PLEG): container finished" podID="f750fe64-42e5-4d6f-91f7-071c1493c881" containerID="704f712bbb2dbfd40ae22408a6c69ba1e475c9ff4d79039bced0825f04ae1a49" exitCode=0 Oct 30 17:16:58.554449 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.552945 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" Oct 30 17:16:58.554449 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.553343 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" event={"ID":"f750fe64-42e5-4d6f-91f7-071c1493c881","Type":"ContainerDied","Data":"704f712bbb2dbfd40ae22408a6c69ba1e475c9ff4d79039bced0825f04ae1a49"} Oct 30 17:16:58.554449 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.553384 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn" event={"ID":"f750fe64-42e5-4d6f-91f7-071c1493c881","Type":"ContainerDied","Data":"7e633b9ffe15bd888354d2c699b780a71c10d316541f219d36cd7470017c1e8b"} Oct 30 17:16:58.564411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.564389 6202 scope.go:122] "RemoveContainer" containerID="f778a7ac0ea6b96d017c00add1efc0dbb8b7fa2135dae671c99b669e524868aa" Oct 30 17:16:58.582542 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.582519 6202 scope.go:122] "RemoveContainer" containerID="f778a7ac0ea6b96d017c00add1efc0dbb8b7fa2135dae671c99b669e524868aa" Oct 30 17:16:58.582916 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:16:58.582890 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f778a7ac0ea6b96d017c00add1efc0dbb8b7fa2135dae671c99b669e524868aa\": container with ID starting with f778a7ac0ea6b96d017c00add1efc0dbb8b7fa2135dae671c99b669e524868aa not found: ID does not exist" containerID="f778a7ac0ea6b96d017c00add1efc0dbb8b7fa2135dae671c99b669e524868aa" Oct 30 17:16:58.582971 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.582930 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f778a7ac0ea6b96d017c00add1efc0dbb8b7fa2135dae671c99b669e524868aa"} err="failed to get container status \"f778a7ac0ea6b96d017c00add1efc0dbb8b7fa2135dae671c99b669e524868aa\": rpc error: code = NotFound desc = could not find container \"f778a7ac0ea6b96d017c00add1efc0dbb8b7fa2135dae671c99b669e524868aa\": container with ID starting with f778a7ac0ea6b96d017c00add1efc0dbb8b7fa2135dae671c99b669e524868aa not found: ID does not exist" Oct 30 17:16:58.582971 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.582956 6202 scope.go:122] "RemoveContainer" containerID="704f712bbb2dbfd40ae22408a6c69ba1e475c9ff4d79039bced0825f04ae1a49" Oct 30 17:16:58.593976 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.593956 6202 scope.go:122] "RemoveContainer" containerID="704f712bbb2dbfd40ae22408a6c69ba1e475c9ff4d79039bced0825f04ae1a49" Oct 30 17:16:58.594331 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:16:58.594309 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704f712bbb2dbfd40ae22408a6c69ba1e475c9ff4d79039bced0825f04ae1a49\": container with ID starting with 704f712bbb2dbfd40ae22408a6c69ba1e475c9ff4d79039bced0825f04ae1a49 not found: ID does not exist" containerID="704f712bbb2dbfd40ae22408a6c69ba1e475c9ff4d79039bced0825f04ae1a49" Oct 30 17:16:58.594401 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.594338 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704f712bbb2dbfd40ae22408a6c69ba1e475c9ff4d79039bced0825f04ae1a49"} err="failed to get container status \"704f712bbb2dbfd40ae22408a6c69ba1e475c9ff4d79039bced0825f04ae1a49\": rpc error: code = NotFound desc = could not find container \"704f712bbb2dbfd40ae22408a6c69ba1e475c9ff4d79039bced0825f04ae1a49\": container with ID starting with 704f712bbb2dbfd40ae22408a6c69ba1e475c9ff4d79039bced0825f04ae1a49 not found: ID does not exist" Oct 30 17:16:58.604102 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.604074 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j"] Oct 30 17:16:58.620464 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.618637 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ae5afd-openshift-default-7d94b865f-b7m2j"] Oct 30 17:16:58.649804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.649759 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn"] Oct 30 17:16:58.658810 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.653707 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-0b2348-64969865db-gndtn"] Oct 30 17:16:58.689602 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.689570 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf"] Oct 30 17:16:58.693589 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.693566 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0b2348-7bc9cf7c75-zdzjf"] Oct 30 17:16:58.712454 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.712429 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh" Oct 30 17:16:58.805114 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.805084 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337af4a5-d6e2-46b3-b89a-b4955cc92da2-kube-api-access-tt42x\" (UniqueName: \"kubernetes.io/projected/337af4a5-d6e2-46b3-b89a-b4955cc92da2-kube-api-access-tt42x\") pod \"337af4a5-d6e2-46b3-b89a-b4955cc92da2\" (UID: \"337af4a5-d6e2-46b3-b89a-b4955cc92da2\") " Oct 30 17:16:58.805325 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.805137 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/337af4a5-d6e2-46b3-b89a-b4955cc92da2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337af4a5-d6e2-46b3-b89a-b4955cc92da2-maas-api-xks-ca\") pod \"337af4a5-d6e2-46b3-b89a-b4955cc92da2\" (UID: \"337af4a5-d6e2-46b3-b89a-b4955cc92da2\") " Oct 30 17:16:58.805325 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.805190 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/337af4a5-d6e2-46b3-b89a-b4955cc92da2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337af4a5-d6e2-46b3-b89a-b4955cc92da2-maas-api-openshift-ca\") pod \"337af4a5-d6e2-46b3-b89a-b4955cc92da2\" (UID: \"337af4a5-d6e2-46b3-b89a-b4955cc92da2\") " Oct 30 17:16:58.805600 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.805575 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337af4a5-d6e2-46b3-b89a-b4955cc92da2-maas-api-openshift-ca" pod "337af4a5-d6e2-46b3-b89a-b4955cc92da2" (UID: "337af4a5-d6e2-46b3-b89a-b4955cc92da2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:58.807415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.807388 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337af4a5-d6e2-46b3-b89a-b4955cc92da2-maas-api-xks-ca" pod "337af4a5-d6e2-46b3-b89a-b4955cc92da2" (UID: "337af4a5-d6e2-46b3-b89a-b4955cc92da2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:58.807513 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.807476 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337af4a5-d6e2-46b3-b89a-b4955cc92da2-kube-api-access-tt42x" pod "337af4a5-d6e2-46b3-b89a-b4955cc92da2" (UID: "337af4a5-d6e2-46b3-b89a-b4955cc92da2"). InnerVolumeSpecName "kube-api-access-tt42x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:58.906739 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.906708 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt42x\" (UniqueName: \"kubernetes.io/projected/337af4a5-d6e2-46b3-b89a-b4955cc92da2-kube-api-access-tt42x\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:58.906739 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.906736 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337af4a5-d6e2-46b3-b89a-b4955cc92da2-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:58.906739 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:58.906745 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337af4a5-d6e2-46b3-b89a-b4955cc92da2-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:16:59.075283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:59.075252 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:59.568200 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:59.568104 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh" event={"ID":"337af4a5-d6e2-46b3-b89a-b4955cc92da2","Type":"ContainerDied","Data":"6b0d8388a0ee9afd3793321cfa0d8f59999f0c7ccfe547fd6ff78460da1bb592"} Oct 30 17:16:59.568200 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:59.568137 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh" Oct 30 17:16:59.568200 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:16:59.568165 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b0d8388a0ee9afd3793321cfa0d8f59999f0c7ccfe547fd6ff78460da1bb592" Oct 30 17:17:00.409569 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:00.409534 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b94338f-f1d5-4e7c-880d-905f46fd9c7e" path="/var/lib/kubelet/pods/1b94338f-f1d5-4e7c-880d-905f46fd9c7e/volumes" Oct 30 17:17:00.410020 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:00.410006 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5018b303-b054-4954-a781-39523acd950f" path="/var/lib/kubelet/pods/5018b303-b054-4954-a781-39523acd950f/volumes" Oct 30 17:17:00.410355 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:00.410343 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f750fe64-42e5-4d6f-91f7-071c1493c881" path="/var/lib/kubelet/pods/f750fe64-42e5-4d6f-91f7-071c1493c881/volumes" Oct 30 17:17:00.917491 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:00.917462 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:01.505289 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:01.505254 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:02.143359 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:02.143322 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:02.537121 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:02.537022 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:02.868363 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:02.868329 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:02.885056 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:02.885011 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" podUID="e14671cc-c74d-4fb5-b363-e060629ad7cc" containerName="payload-processing" containerID="cri-o://37a3befbc8522cf8318ac96e80acd10114a37051ff7d863e98650bbe37db96b6" gracePeriod=25 Oct 30 17:17:02.925665 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:02.925625 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" podUID="3f4c1389-5421-498a-bc8d-05ca2ec455f6" containerName="payload-pre-processing" containerID="cri-o://3aff97e1feebf4ff7b82943dbeb7dc020c1b6c5f719028b0669defb4de4d39d5" gracePeriod=25 Oct 30 17:17:03.068155 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:03.068117 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:03.549463 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:03.549428 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5"] Oct 30 17:17:03.578600 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:03.578574 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs"] Oct 30 17:17:03.677170 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:03.677136 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:03.876463 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:03.876435 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:04.156169 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.156079 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk"] Oct 30 17:17:04.157317 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.157285 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5018b303-b054-4954-a781-39523acd950f" containerName="istio-proxy" Oct 30 17:17:04.157476 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.157462 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5018b303-b054-4954-a781-39523acd950f" containerName="istio-proxy" Oct 30 17:17:04.157583 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.157572 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5018b303-b054-4954-a781-39523acd950f" containerName="istio-proxy" Oct 30 17:17:04.157658 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.157648 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5018b303-b054-4954-a781-39523acd950f" containerName="istio-proxy" Oct 30 17:17:04.157819 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.157739 6202 cpu_manager.go:414] "containerMap: removing container" podUID="1b94338f-f1d5-4e7c-880d-905f46fd9c7e" containerName="payload-pre-processing" Oct 30 17:17:04.157953 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.157938 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b94338f-f1d5-4e7c-880d-905f46fd9c7e" containerName="payload-pre-processing" Oct 30 17:17:04.158047 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.158036 6202 cpu_manager.go:414] "containerMap: removing container" podUID="99d92701-cab0-4360-b5b5-87d49d16ec10" containerName="revoke-keys" Oct 30 17:17:04.158139 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.158127 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d92701-cab0-4360-b5b5-87d49d16ec10" containerName="revoke-keys" Oct 30 17:17:04.158228 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.158211 6202 cpu_manager.go:414] "containerMap: removing container" podUID="f750fe64-42e5-4d6f-91f7-071c1493c881" containerName="payload-processing" Oct 30 17:17:04.158228 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.158226 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f750fe64-42e5-4d6f-91f7-071c1493c881" containerName="payload-processing" Oct 30 17:17:04.158367 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.158278 6202 cpu_manager.go:414] "containerMap: removing container" podUID="337af4a5-d6e2-46b3-b89a-b4955cc92da2" containerName="revoke-keys" Oct 30 17:17:04.158367 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.158286 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337af4a5-d6e2-46b3-b89a-b4955cc92da2" containerName="revoke-keys" Oct 30 17:17:04.158498 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.158484 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="337af4a5-d6e2-46b3-b89a-b4955cc92da2" containerName="revoke-keys" Oct 30 17:17:04.158567 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.158507 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5018b303-b054-4954-a781-39523acd950f" containerName="istio-proxy" Oct 30 17:17:04.158567 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.158520 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5018b303-b054-4954-a781-39523acd950f" containerName="istio-proxy" Oct 30 17:17:04.158567 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.158531 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b94338f-f1d5-4e7c-880d-905f46fd9c7e" containerName="payload-pre-processing" Oct 30 17:17:04.158567 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.158538 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d92701-cab0-4360-b5b5-87d49d16ec10" containerName="revoke-keys" Oct 30 17:17:04.158567 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.158548 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="f750fe64-42e5-4d6f-91f7-071c1493c881" containerName="payload-processing" Oct 30 17:17:04.170582 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.170556 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk" Oct 30 17:17:04.173956 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.173403 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l"] Oct 30 17:17:04.183200 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.183174 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk"] Oct 30 17:17:04.183504 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.183438 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l" Oct 30 17:17:04.191492 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.191454 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:17:04.192292 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.192200 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:17:04.193365 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.193345 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:17:04.193603 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.193578 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:17:04.212030 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.212001 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l"] Oct 30 17:17:04.273707 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.273674 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnqtj\" (UniqueName: \"kubernetes.io/projected/76f830ec-ce77-4535-8d17-a6d64e901053-kube-api-access-gnqtj\") pod \"maas-api-revoke-sub-fef032b4-2sd9l\" (UID: \"76f830ec-ce77-4535-8d17-a6d64e901053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l" Oct 30 17:17:04.273907 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.273718 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823d03c5-f042-4c15-8e73-d57c779e8a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91156806-gkcjk\" (UID: \"823d03c5-f042-4c15-8e73-d57c779e8a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk" Oct 30 17:17:04.273986 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.273902 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt2k9\" (UniqueName: \"kubernetes.io/projected/823d03c5-f042-4c15-8e73-d57c779e8a9a-kube-api-access-nt2k9\") pod \"maas-api-revoke-sub-91156806-gkcjk\" (UID: \"823d03c5-f042-4c15-8e73-d57c779e8a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk" Oct 30 17:17:04.273986 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.273956 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f830ec-ce77-4535-8d17-a6d64e901053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef032b4-2sd9l\" (UID: \"76f830ec-ce77-4535-8d17-a6d64e901053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l" Oct 30 17:17:04.273986 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.273976 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823d03c5-f042-4c15-8e73-d57c779e8a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91156806-gkcjk\" (UID: \"823d03c5-f042-4c15-8e73-d57c779e8a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk" Oct 30 17:17:04.274144 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.274016 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f830ec-ce77-4535-8d17-a6d64e901053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef032b4-2sd9l\" (UID: \"76f830ec-ce77-4535-8d17-a6d64e901053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l" Oct 30 17:17:04.375680 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.375633 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f830ec-ce77-4535-8d17-a6d64e901053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef032b4-2sd9l\" (UID: \"76f830ec-ce77-4535-8d17-a6d64e901053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l" Oct 30 17:17:04.375901 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.375737 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnqtj\" (UniqueName: \"kubernetes.io/projected/76f830ec-ce77-4535-8d17-a6d64e901053-kube-api-access-gnqtj\") pod \"maas-api-revoke-sub-fef032b4-2sd9l\" (UID: \"76f830ec-ce77-4535-8d17-a6d64e901053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l" Oct 30 17:17:04.375901 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.375816 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823d03c5-f042-4c15-8e73-d57c779e8a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91156806-gkcjk\" (UID: \"823d03c5-f042-4c15-8e73-d57c779e8a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk" Oct 30 17:17:04.375992 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.375936 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt2k9\" (UniqueName: \"kubernetes.io/projected/823d03c5-f042-4c15-8e73-d57c779e8a9a-kube-api-access-nt2k9\") pod \"maas-api-revoke-sub-91156806-gkcjk\" (UID: \"823d03c5-f042-4c15-8e73-d57c779e8a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk" Oct 30 17:17:04.375992 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.375970 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f830ec-ce77-4535-8d17-a6d64e901053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef032b4-2sd9l\" (UID: \"76f830ec-ce77-4535-8d17-a6d64e901053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l" Oct 30 17:17:04.376065 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.375995 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823d03c5-f042-4c15-8e73-d57c779e8a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91156806-gkcjk\" (UID: \"823d03c5-f042-4c15-8e73-d57c779e8a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk" Oct 30 17:17:04.376444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.376412 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f830ec-ce77-4535-8d17-a6d64e901053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef032b4-2sd9l\" (UID: \"76f830ec-ce77-4535-8d17-a6d64e901053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l" Oct 30 17:17:04.376622 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.376600 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823d03c5-f042-4c15-8e73-d57c779e8a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91156806-gkcjk\" (UID: \"823d03c5-f042-4c15-8e73-d57c779e8a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk" Oct 30 17:17:04.378378 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.378352 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f830ec-ce77-4535-8d17-a6d64e901053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef032b4-2sd9l\" (UID: \"76f830ec-ce77-4535-8d17-a6d64e901053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l" Oct 30 17:17:04.378459 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.378441 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823d03c5-f042-4c15-8e73-d57c779e8a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91156806-gkcjk\" (UID: \"823d03c5-f042-4c15-8e73-d57c779e8a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk" Oct 30 17:17:04.400516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.400492 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt2k9\" (UniqueName: \"kubernetes.io/projected/823d03c5-f042-4c15-8e73-d57c779e8a9a-kube-api-access-nt2k9\") pod \"maas-api-revoke-sub-91156806-gkcjk\" (UID: \"823d03c5-f042-4c15-8e73-d57c779e8a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk" Oct 30 17:17:04.400642 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.400540 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnqtj\" (UniqueName: \"kubernetes.io/projected/76f830ec-ce77-4535-8d17-a6d64e901053-kube-api-access-gnqtj\") pod \"maas-api-revoke-sub-fef032b4-2sd9l\" (UID: \"76f830ec-ce77-4535-8d17-a6d64e901053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l" Oct 30 17:17:04.486642 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.486566 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:04.504749 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.504724 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk" Oct 30 17:17:04.524361 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.523013 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l" Oct 30 17:17:04.828469 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:17:04.828440 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823d03c5_f042_4c15_8e73_d57c779e8a9a.slice/crio-ed549eabf788aaf59c115a661a8d7a6365c1cc1d4421168e0551a7cda15c7bc7 WatchSource:0}: Error finding container ed549eabf788aaf59c115a661a8d7a6365c1cc1d4421168e0551a7cda15c7bc7: Status 404 returned error can't find the container with id ed549eabf788aaf59c115a661a8d7a6365c1cc1d4421168e0551a7cda15c7bc7 Oct 30 17:17:04.828838 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.828651 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk"] Oct 30 17:17:04.852412 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:04.852388 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l"] Oct 30 17:17:05.482148 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:05.482037 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:05.557775 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:05.557731 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7"] Oct 30 17:17:05.588276 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:05.588246 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk"] Oct 30 17:17:05.600477 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:05.600447 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:05.634751 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:05.634709 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l" event={"ID":"76f830ec-ce77-4535-8d17-a6d64e901053","Type":"ContainerStarted","Data":"53eba797a2a163e4a954974c1ef60a3903a914adb4f064239f8077d6ee1bb3c4"} Oct 30 17:17:05.634932 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:05.634794 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l" event={"ID":"76f830ec-ce77-4535-8d17-a6d64e901053","Type":"ContainerStarted","Data":"8cb4ed9c87260e3007682d6724eb110547dcc2f6b11e6b6d652e8970f86d00e4"} Oct 30 17:17:05.638747 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:05.638622 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk" event={"ID":"823d03c5-f042-4c15-8e73-d57c779e8a9a","Type":"ContainerStarted","Data":"71473a71dafd4e04c390038d2599446d1bab0df6e60c56f2f3b94467fb05a556"} Oct 30 17:17:05.638897 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:05.638803 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk" event={"ID":"823d03c5-f042-4c15-8e73-d57c779e8a9a","Type":"ContainerStarted","Data":"ed549eabf788aaf59c115a661a8d7a6365c1cc1d4421168e0551a7cda15c7bc7"} Oct 30 17:17:05.694888 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:05.694839 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk" podStartSLOduration=1.694827057 podStartE2EDuration="1.694827057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:05.692953774 +0000 UTC m=+3339.403408518" watchObservedRunningTime="2026-10-30 17:17:05.694827057 +0000 UTC m=+3339.405281801" Oct 30 17:17:05.717720 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:05.717686 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:05.737966 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:05.737857 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l" podStartSLOduration=1.737838083 podStartE2EDuration="1.737838083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:05.736152014 +0000 UTC m=+3339.446606758" watchObservedRunningTime="2026-10-30 17:17:05.737838083 +0000 UTC m=+3339.448292828" Oct 30 17:17:05.852243 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:05.851501 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:05.974610 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:05.974553 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:06.088637 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:06.088596 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:06.433192 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:06.433106 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:06.632454 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:06.632419 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:06.648285 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:06.648253 6202 generic.go:356] "Generic (PLEG): container finished" podID="76f830ec-ce77-4535-8d17-a6d64e901053" containerID="53eba797a2a163e4a954974c1ef60a3903a914adb4f064239f8077d6ee1bb3c4" exitCode=0 Oct 30 17:17:06.648472 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:06.648337 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l" event={"ID":"76f830ec-ce77-4535-8d17-a6d64e901053","Type":"ContainerDied","Data":"53eba797a2a163e4a954974c1ef60a3903a914adb4f064239f8077d6ee1bb3c4"} Oct 30 17:17:06.650448 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:06.650424 6202 generic.go:356] "Generic (PLEG): container finished" podID="823d03c5-f042-4c15-8e73-d57c779e8a9a" containerID="71473a71dafd4e04c390038d2599446d1bab0df6e60c56f2f3b94467fb05a556" exitCode=0 Oct 30 17:17:06.650588 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:06.650469 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk" event={"ID":"823d03c5-f042-4c15-8e73-d57c779e8a9a","Type":"ContainerDied","Data":"71473a71dafd4e04c390038d2599446d1bab0df6e60c56f2f3b94467fb05a556"} Oct 30 17:17:06.684054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:06.683974 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6"] Oct 30 17:17:06.688418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:06.688385 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c8db42-fhvj6"] Oct 30 17:17:06.837783 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:06.837725 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:06.914753 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:06.914718 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" podUID="3807a7b3-8948-416d-97fe-e94528e239d2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:17:06.914753 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:06.914741 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" podUID="b123429e-9af8-402a-b91a-1a0c2fdc7697" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:17:07.055099 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.055012 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:07.203751 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.203710 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh"] Oct 30 17:17:07.205897 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.205871 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:07.209448 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.209419 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-bfecdd-fdvwh"] Oct 30 17:17:07.331802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.331752 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:07.687197 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.687161 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:07.818780 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.818731 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l" Oct 30 17:17:07.844081 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.844059 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk" Oct 30 17:17:07.935920 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.935841 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76f830ec-ce77-4535-8d17-a6d64e901053-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f830ec-ce77-4535-8d17-a6d64e901053-maas-api-xks-ca\") pod \"76f830ec-ce77-4535-8d17-a6d64e901053\" (UID: \"76f830ec-ce77-4535-8d17-a6d64e901053\") " Oct 30 17:17:07.935920 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.935894 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823d03c5-f042-4c15-8e73-d57c779e8a9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823d03c5-f042-4c15-8e73-d57c779e8a9a-maas-api-openshift-ca\") pod \"823d03c5-f042-4c15-8e73-d57c779e8a9a\" (UID: \"823d03c5-f042-4c15-8e73-d57c779e8a9a\") " Oct 30 17:17:07.936444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.935932 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f830ec-ce77-4535-8d17-a6d64e901053-kube-api-access-gnqtj\" (UniqueName: \"kubernetes.io/projected/76f830ec-ce77-4535-8d17-a6d64e901053-kube-api-access-gnqtj\") pod \"76f830ec-ce77-4535-8d17-a6d64e901053\" (UID: \"76f830ec-ce77-4535-8d17-a6d64e901053\") " Oct 30 17:17:07.936444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.935959 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823d03c5-f042-4c15-8e73-d57c779e8a9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823d03c5-f042-4c15-8e73-d57c779e8a9a-maas-api-xks-ca\") pod \"823d03c5-f042-4c15-8e73-d57c779e8a9a\" (UID: \"823d03c5-f042-4c15-8e73-d57c779e8a9a\") " Oct 30 17:17:07.936444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.935984 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823d03c5-f042-4c15-8e73-d57c779e8a9a-kube-api-access-nt2k9\" (UniqueName: \"kubernetes.io/projected/823d03c5-f042-4c15-8e73-d57c779e8a9a-kube-api-access-nt2k9\") pod \"823d03c5-f042-4c15-8e73-d57c779e8a9a\" (UID: \"823d03c5-f042-4c15-8e73-d57c779e8a9a\") " Oct 30 17:17:07.936444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.936035 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76f830ec-ce77-4535-8d17-a6d64e901053-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f830ec-ce77-4535-8d17-a6d64e901053-maas-api-openshift-ca\") pod \"76f830ec-ce77-4535-8d17-a6d64e901053\" (UID: \"76f830ec-ce77-4535-8d17-a6d64e901053\") " Oct 30 17:17:07.936444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.936295 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823d03c5-f042-4c15-8e73-d57c779e8a9a-maas-api-openshift-ca" pod "823d03c5-f042-4c15-8e73-d57c779e8a9a" (UID: "823d03c5-f042-4c15-8e73-d57c779e8a9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:07.936675 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.936463 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823d03c5-f042-4c15-8e73-d57c779e8a9a-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:07.936735 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.936675 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f830ec-ce77-4535-8d17-a6d64e901053-maas-api-openshift-ca" pod "76f830ec-ce77-4535-8d17-a6d64e901053" (UID: "76f830ec-ce77-4535-8d17-a6d64e901053"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:07.938539 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.938511 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823d03c5-f042-4c15-8e73-d57c779e8a9a-maas-api-xks-ca" pod "823d03c5-f042-4c15-8e73-d57c779e8a9a" (UID: "823d03c5-f042-4c15-8e73-d57c779e8a9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:07.939013 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.938986 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823d03c5-f042-4c15-8e73-d57c779e8a9a-kube-api-access-nt2k9" pod "823d03c5-f042-4c15-8e73-d57c779e8a9a" (UID: "823d03c5-f042-4c15-8e73-d57c779e8a9a"). InnerVolumeSpecName "kube-api-access-nt2k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:07.939124 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.939026 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f830ec-ce77-4535-8d17-a6d64e901053-kube-api-access-gnqtj" pod "76f830ec-ce77-4535-8d17-a6d64e901053" (UID: "76f830ec-ce77-4535-8d17-a6d64e901053"). InnerVolumeSpecName "kube-api-access-gnqtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:07.939124 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:07.939027 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76f830ec-ce77-4535-8d17-a6d64e901053-maas-api-xks-ca" pod "76f830ec-ce77-4535-8d17-a6d64e901053" (UID: "76f830ec-ce77-4535-8d17-a6d64e901053"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:08.037683 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.037649 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f830ec-ce77-4535-8d17-a6d64e901053-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:08.037683 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.037680 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnqtj\" (UniqueName: \"kubernetes.io/projected/76f830ec-ce77-4535-8d17-a6d64e901053-kube-api-access-gnqtj\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:08.037683 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.037691 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823d03c5-f042-4c15-8e73-d57c779e8a9a-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:08.037940 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.037702 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt2k9\" (UniqueName: \"kubernetes.io/projected/823d03c5-f042-4c15-8e73-d57c779e8a9a-kube-api-access-nt2k9\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:08.037940 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.037710 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f830ec-ce77-4535-8d17-a6d64e901053-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:08.413863 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.413833 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="337af4a5-d6e2-46b3-b89a-b4955cc92da2" path="/var/lib/kubelet/pods/337af4a5-d6e2-46b3-b89a-b4955cc92da2/volumes" Oct 30 17:17:08.414334 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.414320 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51806a0b-50a9-4d05-91db-3ba5340f3e07" path="/var/lib/kubelet/pods/51806a0b-50a9-4d05-91db-3ba5340f3e07/volumes" Oct 30 17:17:08.550805 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.550326 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5" podUID="f02f9f64-d9f3-4fe2-b2fb-a994a92796aa" containerName="payload-processing" containerID="cri-o://18908458f95ad16dfa16ab9ead851c033912044c516345cb51193ed4db04a70a" gracePeriod=25 Oct 30 17:17:08.578869 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.578824 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs" podUID="9a88a4db-f16f-4663-b7b3-acce72983c69" containerName="payload-pre-processing" containerID="cri-o://1c11d50b0915ece3776001fbd9d79322054587ef0ca9b233cded56bb3beaa93e" gracePeriod=25 Oct 30 17:17:08.671243 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.671211 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l" Oct 30 17:17:08.671243 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.671226 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l" event={"ID":"76f830ec-ce77-4535-8d17-a6d64e901053","Type":"ContainerDied","Data":"8cb4ed9c87260e3007682d6724eb110547dcc2f6b11e6b6d652e8970f86d00e4"} Oct 30 17:17:08.671511 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.671278 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cb4ed9c87260e3007682d6724eb110547dcc2f6b11e6b6d652e8970f86d00e4" Oct 30 17:17:08.673160 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.673134 6202 generic.go:356] "Generic (PLEG): container finished" podID="b123429e-9af8-402a-b91a-1a0c2fdc7697" containerID="267285a07aaec76c93ac695ab23734819cd5035e61889d38d0c8a760d990e767" exitCode=0 Oct 30 17:17:08.673332 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.673222 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" event={"ID":"b123429e-9af8-402a-b91a-1a0c2fdc7697","Type":"ContainerDied","Data":"267285a07aaec76c93ac695ab23734819cd5035e61889d38d0c8a760d990e767"} Oct 30 17:17:08.674921 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.674892 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk" Oct 30 17:17:08.674921 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.674897 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk" event={"ID":"823d03c5-f042-4c15-8e73-d57c779e8a9a","Type":"ContainerDied","Data":"ed549eabf788aaf59c115a661a8d7a6365c1cc1d4421168e0551a7cda15c7bc7"} Oct 30 17:17:08.675116 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.674952 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed549eabf788aaf59c115a661a8d7a6365c1cc1d4421168e0551a7cda15c7bc7" Oct 30 17:17:08.676617 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.676596 6202 generic.go:356] "Generic (PLEG): container finished" podID="f02f9f64-d9f3-4fe2-b2fb-a994a92796aa" containerID="18908458f95ad16dfa16ab9ead851c033912044c516345cb51193ed4db04a70a" exitCode=0 Oct 30 17:17:08.676742 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.676689 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5" event={"ID":"f02f9f64-d9f3-4fe2-b2fb-a994a92796aa","Type":"ContainerDied","Data":"18908458f95ad16dfa16ab9ead851c033912044c516345cb51193ed4db04a70a"} Oct 30 17:17:08.678781 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.678748 6202 generic.go:356] "Generic (PLEG): container finished" podID="3807a7b3-8948-416d-97fe-e94528e239d2" containerID="771bb3e055ce6ba9ef32a1134ff9a5894652e6924b717421b4cb5e7ea4242003" exitCode=0 Oct 30 17:17:08.678887 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.678808 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" event={"ID":"3807a7b3-8948-416d-97fe-e94528e239d2","Type":"ContainerDied","Data":"771bb3e055ce6ba9ef32a1134ff9a5894652e6924b717421b4cb5e7ea4242003"} Oct 30 17:17:08.692063 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.691750 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" Oct 30 17:17:08.696098 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.696074 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" Oct 30 17:17:08.751046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.745971 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3807a7b3-8948-416d-97fe-e94528e239d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3807a7b3-8948-416d-97fe-e94528e239d2-plugins-config-volume\") pod \"3807a7b3-8948-416d-97fe-e94528e239d2\" (UID: \"3807a7b3-8948-416d-97fe-e94528e239d2\") " Oct 30 17:17:08.751046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.746031 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b123429e-9af8-402a-b91a-1a0c2fdc7697-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b123429e-9af8-402a-b91a-1a0c2fdc7697-plugins-config-volume\") pod \"b123429e-9af8-402a-b91a-1a0c2fdc7697\" (UID: \"b123429e-9af8-402a-b91a-1a0c2fdc7697\") " Oct 30 17:17:08.751046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.746099 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b123429e-9af8-402a-b91a-1a0c2fdc7697-kube-api-access-w82nc\" (UniqueName: \"kubernetes.io/projected/b123429e-9af8-402a-b91a-1a0c2fdc7697-kube-api-access-w82nc\") pod \"b123429e-9af8-402a-b91a-1a0c2fdc7697\" (UID: \"b123429e-9af8-402a-b91a-1a0c2fdc7697\") " Oct 30 17:17:08.751046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.746166 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3807a7b3-8948-416d-97fe-e94528e239d2-kube-api-access-qbddf\" (UniqueName: \"kubernetes.io/projected/3807a7b3-8948-416d-97fe-e94528e239d2-kube-api-access-qbddf\") pod \"3807a7b3-8948-416d-97fe-e94528e239d2\" (UID: \"3807a7b3-8948-416d-97fe-e94528e239d2\") " Oct 30 17:17:08.751046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.747808 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3807a7b3-8948-416d-97fe-e94528e239d2-plugins-config-volume" pod "3807a7b3-8948-416d-97fe-e94528e239d2" (UID: "3807a7b3-8948-416d-97fe-e94528e239d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:08.751046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.748295 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b123429e-9af8-402a-b91a-1a0c2fdc7697-plugins-config-volume" pod "b123429e-9af8-402a-b91a-1a0c2fdc7697" (UID: "b123429e-9af8-402a-b91a-1a0c2fdc7697"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:08.763251 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.753297 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b123429e-9af8-402a-b91a-1a0c2fdc7697-kube-api-access-w82nc" pod "b123429e-9af8-402a-b91a-1a0c2fdc7697" (UID: "b123429e-9af8-402a-b91a-1a0c2fdc7697"). InnerVolumeSpecName "kube-api-access-w82nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:08.763251 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.754896 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3807a7b3-8948-416d-97fe-e94528e239d2-kube-api-access-qbddf" pod "3807a7b3-8948-416d-97fe-e94528e239d2" (UID: "3807a7b3-8948-416d-97fe-e94528e239d2"). InnerVolumeSpecName "kube-api-access-qbddf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:08.849342 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.848234 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3807a7b3-8948-416d-97fe-e94528e239d2-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:08.849342 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.848267 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b123429e-9af8-402a-b91a-1a0c2fdc7697-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:08.849342 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.848281 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w82nc\" (UniqueName: \"kubernetes.io/projected/b123429e-9af8-402a-b91a-1a0c2fdc7697-kube-api-access-w82nc\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:08.849342 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.848297 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbddf\" (UniqueName: \"kubernetes.io/projected/3807a7b3-8948-416d-97fe-e94528e239d2-kube-api-access-qbddf\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:08.852489 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.852459 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5" Oct 30 17:17:08.909104 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.909067 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs" Oct 30 17:17:08.949002 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.948965 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f02f9f64-d9f3-4fe2-b2fb-a994a92796aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02f9f64-d9f3-4fe2-b2fb-a994a92796aa-plugins-config-volume\") pod \"f02f9f64-d9f3-4fe2-b2fb-a994a92796aa\" (UID: \"f02f9f64-d9f3-4fe2-b2fb-a994a92796aa\") " Oct 30 17:17:08.949471 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.949024 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a88a4db-f16f-4663-b7b3-acce72983c69-kube-api-access-xtwk6\" (UniqueName: \"kubernetes.io/projected/9a88a4db-f16f-4663-b7b3-acce72983c69-kube-api-access-xtwk6\") pod \"9a88a4db-f16f-4663-b7b3-acce72983c69\" (UID: \"9a88a4db-f16f-4663-b7b3-acce72983c69\") " Oct 30 17:17:08.949471 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.949054 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02f9f64-d9f3-4fe2-b2fb-a994a92796aa-kube-api-access-gwmd2\" (UniqueName: \"kubernetes.io/projected/f02f9f64-d9f3-4fe2-b2fb-a994a92796aa-kube-api-access-gwmd2\") pod \"f02f9f64-d9f3-4fe2-b2fb-a994a92796aa\" (UID: \"f02f9f64-d9f3-4fe2-b2fb-a994a92796aa\") " Oct 30 17:17:08.949471 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.949211 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a88a4db-f16f-4663-b7b3-acce72983c69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a88a4db-f16f-4663-b7b3-acce72983c69-plugins-config-volume\") pod \"9a88a4db-f16f-4663-b7b3-acce72983c69\" (UID: \"9a88a4db-f16f-4663-b7b3-acce72983c69\") " Oct 30 17:17:08.949471 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.949387 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02f9f64-d9f3-4fe2-b2fb-a994a92796aa-plugins-config-volume" pod "f02f9f64-d9f3-4fe2-b2fb-a994a92796aa" (UID: "f02f9f64-d9f3-4fe2-b2fb-a994a92796aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:08.949701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.949674 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02f9f64-d9f3-4fe2-b2fb-a994a92796aa-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:08.950105 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.950074 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a88a4db-f16f-4663-b7b3-acce72983c69-plugins-config-volume" pod "9a88a4db-f16f-4663-b7b3-acce72983c69" (UID: "9a88a4db-f16f-4663-b7b3-acce72983c69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:08.952472 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.952427 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a88a4db-f16f-4663-b7b3-acce72983c69-kube-api-access-xtwk6" pod "9a88a4db-f16f-4663-b7b3-acce72983c69" (UID: "9a88a4db-f16f-4663-b7b3-acce72983c69"). InnerVolumeSpecName "kube-api-access-xtwk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:08.953028 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:08.953005 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02f9f64-d9f3-4fe2-b2fb-a994a92796aa-kube-api-access-gwmd2" pod "f02f9f64-d9f3-4fe2-b2fb-a994a92796aa" (UID: "f02f9f64-d9f3-4fe2-b2fb-a994a92796aa"). InnerVolumeSpecName "kube-api-access-gwmd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:09.050499 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.050461 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a88a4db-f16f-4663-b7b3-acce72983c69-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:09.050499 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.050488 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtwk6\" (UniqueName: \"kubernetes.io/projected/9a88a4db-f16f-4663-b7b3-acce72983c69-kube-api-access-xtwk6\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:09.050499 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.050498 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwmd2\" (UniqueName: \"kubernetes.io/projected/f02f9f64-d9f3-4fe2-b2fb-a994a92796aa-kube-api-access-gwmd2\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:09.058225 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.058165 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:09.182100 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.182048 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:09.688205 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.688174 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" Oct 30 17:17:09.690559 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.688170 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx" event={"ID":"b123429e-9af8-402a-b91a-1a0c2fdc7697","Type":"ContainerDied","Data":"a4ffbeb4ab72ad6d96ab4c85d41eeb1592bd11cb54cc5f7454a7ef19528d6930"} Oct 30 17:17:09.690559 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.688324 6202 scope.go:122] "RemoveContainer" containerID="267285a07aaec76c93ac695ab23734819cd5035e61889d38d0c8a760d990e767" Oct 30 17:17:09.690559 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.689813 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5" event={"ID":"f02f9f64-d9f3-4fe2-b2fb-a994a92796aa","Type":"ContainerDied","Data":"f24415a61a3e970c1ea872d1d0c9147fe3ae9116598555268cabd2c4686244ec"} Oct 30 17:17:09.690559 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.689853 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5" Oct 30 17:17:09.691519 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.691492 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" Oct 30 17:17:09.691672 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.691547 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28" event={"ID":"3807a7b3-8948-416d-97fe-e94528e239d2","Type":"ContainerDied","Data":"3c325d24a4edd01e8151fd6b218ff1a6459fb317d7c936438d5ee6abf60272ed"} Oct 30 17:17:09.698359 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.698326 6202 generic.go:356] "Generic (PLEG): container finished" podID="9a88a4db-f16f-4663-b7b3-acce72983c69" containerID="1c11d50b0915ece3776001fbd9d79322054587ef0ca9b233cded56bb3beaa93e" exitCode=0 Oct 30 17:17:09.698359 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.698355 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs" Oct 30 17:17:09.698557 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.698402 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs" event={"ID":"9a88a4db-f16f-4663-b7b3-acce72983c69","Type":"ContainerDied","Data":"1c11d50b0915ece3776001fbd9d79322054587ef0ca9b233cded56bb3beaa93e"} Oct 30 17:17:09.698557 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.698449 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs" event={"ID":"9a88a4db-f16f-4663-b7b3-acce72983c69","Type":"ContainerDied","Data":"520a53e3821bd2939f57f1de2a1b7c911408b889643a8e29a5b4db72c318e264"} Oct 30 17:17:09.699972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.699732 6202 scope.go:122] "RemoveContainer" containerID="18908458f95ad16dfa16ab9ead851c033912044c516345cb51193ed4db04a70a" Oct 30 17:17:09.709903 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.709823 6202 scope.go:122] "RemoveContainer" containerID="771bb3e055ce6ba9ef32a1134ff9a5894652e6924b717421b4cb5e7ea4242003" Oct 30 17:17:09.718879 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.718860 6202 scope.go:122] "RemoveContainer" containerID="1c11d50b0915ece3776001fbd9d79322054587ef0ca9b233cded56bb3beaa93e" Oct 30 17:17:09.727495 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.727478 6202 scope.go:122] "RemoveContainer" containerID="1c11d50b0915ece3776001fbd9d79322054587ef0ca9b233cded56bb3beaa93e" Oct 30 17:17:09.727755 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:17:09.727732 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c11d50b0915ece3776001fbd9d79322054587ef0ca9b233cded56bb3beaa93e\": container with ID starting with 1c11d50b0915ece3776001fbd9d79322054587ef0ca9b233cded56bb3beaa93e not found: ID does not exist" containerID="1c11d50b0915ece3776001fbd9d79322054587ef0ca9b233cded56bb3beaa93e" Oct 30 17:17:09.727834 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.727788 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c11d50b0915ece3776001fbd9d79322054587ef0ca9b233cded56bb3beaa93e"} err="failed to get container status \"1c11d50b0915ece3776001fbd9d79322054587ef0ca9b233cded56bb3beaa93e\": rpc error: code = NotFound desc = could not find container \"1c11d50b0915ece3776001fbd9d79322054587ef0ca9b233cded56bb3beaa93e\": container with ID starting with 1c11d50b0915ece3776001fbd9d79322054587ef0ca9b233cded56bb3beaa93e not found: ID does not exist" Oct 30 17:17:09.760691 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.760639 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx"] Oct 30 17:17:09.773711 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.773670 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-ae5afd-85b649c995-fnpzx"] Oct 30 17:17:09.825521 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.825487 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5"] Oct 30 17:17:09.828934 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.828893 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a8c22d-585f98f7c6-fljj5"] Oct 30 17:17:09.865650 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.865615 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28"] Oct 30 17:17:09.869007 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.868980 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-ae5afd-5dcc467768-lnz28"] Oct 30 17:17:09.902400 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.902375 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs"] Oct 30 17:17:09.904496 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:09.904469 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a8c22d-7c5689b95f-wmrvs"] Oct 30 17:17:10.409098 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:10.409064 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3807a7b3-8948-416d-97fe-e94528e239d2" path="/var/lib/kubelet/pods/3807a7b3-8948-416d-97fe-e94528e239d2/volumes" Oct 30 17:17:10.409489 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:10.409445 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a88a4db-f16f-4663-b7b3-acce72983c69" path="/var/lib/kubelet/pods/9a88a4db-f16f-4663-b7b3-acce72983c69/volumes" Oct 30 17:17:10.409777 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:10.409751 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b123429e-9af8-402a-b91a-1a0c2fdc7697" path="/var/lib/kubelet/pods/b123429e-9af8-402a-b91a-1a0c2fdc7697/volumes" Oct 30 17:17:10.410098 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:10.410086 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f02f9f64-d9f3-4fe2-b2fb-a994a92796aa" path="/var/lib/kubelet/pods/f02f9f64-d9f3-4fe2-b2fb-a994a92796aa/volumes" Oct 30 17:17:10.558903 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:10.558843 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" podUID="cf706793-9392-4041-8753-f65a7f1afe24" containerName="payload-processing" containerID="cri-o://75b1301afbc1d6d370ce272a13cf03e3ca26b8949d5ccc25316e725ff22a56fd" gracePeriod=25 Oct 30 17:17:10.589579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:10.589532 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" podUID="052f7204-67ca-4d46-8b48-5f44e951a8f1" containerName="payload-pre-processing" containerID="cri-o://070515d09bed052bee8ec9fa902a8ba75eaa17ad24eb57f3f3a68b7959da17e5" gracePeriod=25 Oct 30 17:17:10.786282 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:10.786177 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:12.352108 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:12.352067 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" podUID="3f4c1389-5421-498a-bc8d-05ca2ec455f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:17:12.352594 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:12.352525 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" podUID="e14671cc-c74d-4fb5-b363-e060629ad7cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:17:12.742499 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:12.742342 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:12.860429 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:12.860390 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:14.074532 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:17:14.074477 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:17:14.075304 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:14.075263 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" podUID="052f7204-67ca-4d46-8b48-5f44e951a8f1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:17:14.075439 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:14.075306 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" podUID="cf706793-9392-4041-8753-f65a7f1afe24" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:17:14.408045 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:14.407954 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:14.518515 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:14.518480 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:14.817518 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:14.817440 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb"] Oct 30 17:17:14.817685 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:14.817661 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" podUID="a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" containerName="istio-proxy" containerID="cri-o://5e75df24ec30f457fda6ba267ef67755946b41012e320b0ce62365cdeaf1a3b6" gracePeriod=30 Oct 30 17:17:14.866151 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:14.865367 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:15.382296 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.382258 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:15.498286 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.498256 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:15.555515 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.555464 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742"] Oct 30 17:17:15.556243 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.556215 6202 cpu_manager.go:414] "containerMap: removing container" podUID="823d03c5-f042-4c15-8e73-d57c779e8a9a" containerName="revoke-keys" Oct 30 17:17:15.556243 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.556241 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823d03c5-f042-4c15-8e73-d57c779e8a9a" containerName="revoke-keys" Oct 30 17:17:15.556461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.556269 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3807a7b3-8948-416d-97fe-e94528e239d2" containerName="payload-processing" Oct 30 17:17:15.556461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.556279 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3807a7b3-8948-416d-97fe-e94528e239d2" containerName="payload-processing" Oct 30 17:17:15.556461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.556298 6202 cpu_manager.go:414] "containerMap: removing container" podUID="f02f9f64-d9f3-4fe2-b2fb-a994a92796aa" containerName="payload-processing" Oct 30 17:17:15.556461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.556306 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02f9f64-d9f3-4fe2-b2fb-a994a92796aa" containerName="payload-processing" Oct 30 17:17:15.556461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.556324 6202 cpu_manager.go:414] "containerMap: removing container" podUID="b123429e-9af8-402a-b91a-1a0c2fdc7697" containerName="payload-pre-processing" Oct 30 17:17:15.556461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.556331 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b123429e-9af8-402a-b91a-1a0c2fdc7697" containerName="payload-pre-processing" Oct 30 17:17:15.556461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.556343 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9a88a4db-f16f-4663-b7b3-acce72983c69" containerName="payload-pre-processing" Oct 30 17:17:15.556461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.556350 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a88a4db-f16f-4663-b7b3-acce72983c69" containerName="payload-pre-processing" Oct 30 17:17:15.556461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.556383 6202 cpu_manager.go:414] "containerMap: removing container" podUID="76f830ec-ce77-4535-8d17-a6d64e901053" containerName="revoke-keys" Oct 30 17:17:15.556461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.556390 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f830ec-ce77-4535-8d17-a6d64e901053" containerName="revoke-keys" Oct 30 17:17:15.557035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.556478 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f830ec-ce77-4535-8d17-a6d64e901053" containerName="revoke-keys" Oct 30 17:17:15.557035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.556493 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="823d03c5-f042-4c15-8e73-d57c779e8a9a" containerName="revoke-keys" Oct 30 17:17:15.557035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.556503 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3807a7b3-8948-416d-97fe-e94528e239d2" containerName="payload-processing" Oct 30 17:17:15.557035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.556512 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02f9f64-d9f3-4fe2-b2fb-a994a92796aa" containerName="payload-processing" Oct 30 17:17:15.557035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.556524 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="b123429e-9af8-402a-b91a-1a0c2fdc7697" containerName="payload-pre-processing" Oct 30 17:17:15.557035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.556534 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a88a4db-f16f-4663-b7b3-acce72983c69" containerName="payload-pre-processing" Oct 30 17:17:15.562043 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.562017 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742" Oct 30 17:17:15.571136 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.571108 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742"] Oct 30 17:17:15.577690 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.577663 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:17:15.583808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.581370 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:17:15.583808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.581435 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:17:15.583808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.581462 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:17:15.618723 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.618694 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:15.626042 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.626016 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742\" (UID: \"b71c1827-6c8f-4116-9fd3-79ffedf89b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742" Oct 30 17:17:15.626164 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.626126 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxjcc\" (UniqueName: \"kubernetes.io/projected/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-kube-api-access-mxjcc\") pod \"maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742\" (UID: \"b71c1827-6c8f-4116-9fd3-79ffedf89b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742" Oct 30 17:17:15.626164 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.626150 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742\" (UID: \"b71c1827-6c8f-4116-9fd3-79ffedf89b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742" Oct 30 17:17:15.727331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.727240 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742\" (UID: \"b71c1827-6c8f-4116-9fd3-79ffedf89b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742" Oct 30 17:17:15.727331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.727325 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxjcc\" (UniqueName: \"kubernetes.io/projected/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-kube-api-access-mxjcc\") pod \"maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742\" (UID: \"b71c1827-6c8f-4116-9fd3-79ffedf89b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742" Oct 30 17:17:15.727593 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.727472 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742\" (UID: \"b71c1827-6c8f-4116-9fd3-79ffedf89b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742" Oct 30 17:17:15.728129 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.728110 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742\" (UID: \"b71c1827-6c8f-4116-9fd3-79ffedf89b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742" Oct 30 17:17:15.729840 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.729822 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742\" (UID: \"b71c1827-6c8f-4116-9fd3-79ffedf89b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742" Oct 30 17:17:15.751702 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.751674 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxjcc\" (UniqueName: \"kubernetes.io/projected/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-kube-api-access-mxjcc\") pod \"maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742\" (UID: \"b71c1827-6c8f-4116-9fd3-79ffedf89b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742" Oct 30 17:17:15.848695 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.848657 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl"] Oct 30 17:17:15.848952 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.848926 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" podUID="e2315e06-69db-44cc-ad5f-f875d7758205" containerName="istio-proxy" containerID="cri-o://f901609bbc9425608a465bf3601cff6bbb1347324a8d93ffd4f87befbd6de8d4" gracePeriod=30 Oct 30 17:17:15.879543 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.879510 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742" Oct 30 17:17:15.881984 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:15.881955 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:16.182923 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:17:16.182887 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71c1827_6c8f_4116_9fd3_79ffedf89b5f.slice/crio-32455934edfb9400a4df01a88c255fb200cedafa82072d91e0aff755e68440b1 WatchSource:0}: Error finding container 32455934edfb9400a4df01a88c255fb200cedafa82072d91e0aff755e68440b1: Status 404 returned error can't find the container with id 32455934edfb9400a4df01a88c255fb200cedafa82072d91e0aff755e68440b1 Oct 30 17:17:16.183312 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:16.183283 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742"] Oct 30 17:17:16.304693 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:16.304653 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:16.773705 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:16.773603 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742" event={"ID":"b71c1827-6c8f-4116-9fd3-79ffedf89b5f","Type":"ContainerStarted","Data":"2398d113d49d69c48014065a60e4cfe8e1816f9abaa7474a6396e16ca24a0c6c"} Oct 30 17:17:16.773705 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:16.773700 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742" event={"ID":"b71c1827-6c8f-4116-9fd3-79ffedf89b5f","Type":"ContainerStarted","Data":"32455934edfb9400a4df01a88c255fb200cedafa82072d91e0aff755e68440b1"} Oct 30 17:17:16.818226 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:16.818170 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742" podStartSLOduration=1.818156525 podStartE2EDuration="1.818156525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:16.815854091 +0000 UTC m=+3350.526308834" watchObservedRunningTime="2026-10-30 17:17:16.818156525 +0000 UTC m=+3350.528611303" Oct 30 17:17:17.296898 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:17.296809 6202 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:17:17.296898 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:17.296890 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" podUID="a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:17:17.782631 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:17.782599 6202 generic.go:356] "Generic (PLEG): container finished" podID="b71c1827-6c8f-4116-9fd3-79ffedf89b5f" containerID="2398d113d49d69c48014065a60e4cfe8e1816f9abaa7474a6396e16ca24a0c6c" exitCode=0 Oct 30 17:17:17.783054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:17.782690 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742" event={"ID":"b71c1827-6c8f-4116-9fd3-79ffedf89b5f","Type":"ContainerDied","Data":"2398d113d49d69c48014065a60e4cfe8e1816f9abaa7474a6396e16ca24a0c6c"} Oct 30 17:17:18.185368 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:18.185335 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:18.930406 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:18.930378 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742" Oct 30 17:17:19.065964 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:19.065887 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-kube-api-access-mxjcc\" (UniqueName: \"kubernetes.io/projected/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-kube-api-access-mxjcc\") pod \"b71c1827-6c8f-4116-9fd3-79ffedf89b5f\" (UID: \"b71c1827-6c8f-4116-9fd3-79ffedf89b5f\") " Oct 30 17:17:19.065964 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:19.065959 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-maas-api-xks-ca\") pod \"b71c1827-6c8f-4116-9fd3-79ffedf89b5f\" (UID: \"b71c1827-6c8f-4116-9fd3-79ffedf89b5f\") " Oct 30 17:17:19.066155 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:19.065984 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-maas-api-openshift-ca\") pod \"b71c1827-6c8f-4116-9fd3-79ffedf89b5f\" (UID: \"b71c1827-6c8f-4116-9fd3-79ffedf89b5f\") " Oct 30 17:17:19.066398 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:19.066367 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-maas-api-openshift-ca" pod "b71c1827-6c8f-4116-9fd3-79ffedf89b5f" (UID: "b71c1827-6c8f-4116-9fd3-79ffedf89b5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:19.068195 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:19.068170 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-maas-api-xks-ca" pod "b71c1827-6c8f-4116-9fd3-79ffedf89b5f" (UID: "b71c1827-6c8f-4116-9fd3-79ffedf89b5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:19.068295 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:19.068213 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-kube-api-access-mxjcc" pod "b71c1827-6c8f-4116-9fd3-79ffedf89b5f" (UID: "b71c1827-6c8f-4116-9fd3-79ffedf89b5f"). InnerVolumeSpecName "kube-api-access-mxjcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:19.168113 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:19.168076 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxjcc\" (UniqueName: \"kubernetes.io/projected/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-kube-api-access-mxjcc\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:19.168113 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:19.168110 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:19.168301 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:19.168124 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71c1827-6c8f-4116-9fd3-79ffedf89b5f-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:19.407593 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:19.407561 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:19.798297 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:19.798216 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742" Oct 30 17:17:19.798558 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:19.798252 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742" event={"ID":"b71c1827-6c8f-4116-9fd3-79ffedf89b5f","Type":"ContainerDied","Data":"32455934edfb9400a4df01a88c255fb200cedafa82072d91e0aff755e68440b1"} Oct 30 17:17:19.798707 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:19.798574 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32455934edfb9400a4df01a88c255fb200cedafa82072d91e0aff755e68440b1" Oct 30 17:17:19.821531 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:19.821504 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:20.115793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.111969 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 17:17:20.195175 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.195145 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-credential-socket\") pod \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " Oct 30 17:17:20.195367 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.195200 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-token\" (UniqueName: \"kubernetes.io/projected/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-token\") pod \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " Oct 30 17:17:20.195367 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.195239 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-workload-socket\") pod \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " Oct 30 17:17:20.195367 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.195278 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-kube-api-access-6hxrj\" (UniqueName: \"kubernetes.io/projected/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-kube-api-access-6hxrj\") pod \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " Oct 30 17:17:20.195367 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.195317 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istiod-ca-cert\") pod \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " Oct 30 17:17:20.195367 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.195337 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-data\") pod \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " Oct 30 17:17:20.195367 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.195360 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-workload-certs\") pod \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " Oct 30 17:17:20.195709 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.195420 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-envoy\") pod \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " Oct 30 17:17:20.195709 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.195467 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-podinfo\") pod \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\" (UID: \"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2\") " Oct 30 17:17:20.195709 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.195497 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-credential-socket" pod "a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" (UID: "a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:20.195709 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.195587 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-workload-socket" pod "a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" (UID: "a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:20.195972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.195709 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istiod-ca-cert" pod "a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" (UID: "a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:20.195972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.195799 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-workload-certs" pod "a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" (UID: "a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:20.195972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.195937 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-data" pod "a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" (UID: "a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:20.196148 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.196097 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:20.196148 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.196120 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:20.196148 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.196134 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:20.196148 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.196147 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:20.196537 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.196160 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:20.197814 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.197757 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-kube-api-access-6hxrj" pod "a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" (UID: "a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2"). InnerVolumeSpecName "kube-api-access-6hxrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:20.198058 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.198035 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-envoy" pod "a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" (UID: "a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:20.198193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.198109 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-token" pod "a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" (UID: "a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:20.198267 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.198242 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-podinfo" pod "a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" (UID: "a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:17:20.297235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.297189 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:20.297235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.297224 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hxrj\" (UniqueName: \"kubernetes.io/projected/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-kube-api-access-6hxrj\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:20.297235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.297236 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:20.297235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.297245 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:20.453416 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.453384 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:20.576540 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.576509 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:20.621927 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.621884 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl"] Oct 30 17:17:20.626283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.626252 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a8c22d-bz7rl"] Oct 30 17:17:20.692638 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.692540 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:20.807381 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.807344 6202 generic.go:356] "Generic (PLEG): container finished" podID="a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" containerID="5e75df24ec30f457fda6ba267ef67755946b41012e320b0ce62365cdeaf1a3b6" exitCode=0 Oct 30 17:17:20.807573 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.807397 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" Oct 30 17:17:20.807573 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.807420 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" event={"ID":"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2","Type":"ContainerDied","Data":"5e75df24ec30f457fda6ba267ef67755946b41012e320b0ce62365cdeaf1a3b6"} Oct 30 17:17:20.807573 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.807471 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb" event={"ID":"a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2","Type":"ContainerDied","Data":"e340adcaf5c1805299743b777184fffe14c1a3c450c9c2ad91b325acc496c5c6"} Oct 30 17:17:20.807573 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.807495 6202 scope.go:122] "RemoveContainer" containerID="5e75df24ec30f457fda6ba267ef67755946b41012e320b0ce62365cdeaf1a3b6" Oct 30 17:17:20.818342 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.818321 6202 scope.go:122] "RemoveContainer" containerID="7d88a428f5b39bb6d46ebb63910aaf26c2b899da2884e7ff3b06eee592b30ce5" Oct 30 17:17:20.828377 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.828357 6202 scope.go:122] "RemoveContainer" containerID="5e75df24ec30f457fda6ba267ef67755946b41012e320b0ce62365cdeaf1a3b6" Oct 30 17:17:20.828676 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:17:20.828650 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e75df24ec30f457fda6ba267ef67755946b41012e320b0ce62365cdeaf1a3b6\": container with ID starting with 5e75df24ec30f457fda6ba267ef67755946b41012e320b0ce62365cdeaf1a3b6 not found: ID does not exist" containerID="5e75df24ec30f457fda6ba267ef67755946b41012e320b0ce62365cdeaf1a3b6" Oct 30 17:17:20.828727 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.828710 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e75df24ec30f457fda6ba267ef67755946b41012e320b0ce62365cdeaf1a3b6"} err="failed to get container status \"5e75df24ec30f457fda6ba267ef67755946b41012e320b0ce62365cdeaf1a3b6\": rpc error: code = NotFound desc = could not find container \"5e75df24ec30f457fda6ba267ef67755946b41012e320b0ce62365cdeaf1a3b6\": container with ID starting with 5e75df24ec30f457fda6ba267ef67755946b41012e320b0ce62365cdeaf1a3b6 not found: ID does not exist" Oct 30 17:17:20.828841 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.828728 6202 scope.go:122] "RemoveContainer" containerID="7d88a428f5b39bb6d46ebb63910aaf26c2b899da2884e7ff3b06eee592b30ce5" Oct 30 17:17:20.829050 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:17:20.829030 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d88a428f5b39bb6d46ebb63910aaf26c2b899da2884e7ff3b06eee592b30ce5\": container with ID starting with 7d88a428f5b39bb6d46ebb63910aaf26c2b899da2884e7ff3b06eee592b30ce5 not found: ID does not exist" containerID="7d88a428f5b39bb6d46ebb63910aaf26c2b899da2884e7ff3b06eee592b30ce5" Oct 30 17:17:20.829125 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.829053 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d88a428f5b39bb6d46ebb63910aaf26c2b899da2884e7ff3b06eee592b30ce5"} err="failed to get container status \"7d88a428f5b39bb6d46ebb63910aaf26c2b899da2884e7ff3b06eee592b30ce5\": rpc error: code = NotFound desc = could not find container \"7d88a428f5b39bb6d46ebb63910aaf26c2b899da2884e7ff3b06eee592b30ce5\": container with ID starting with 7d88a428f5b39bb6d46ebb63910aaf26c2b899da2884e7ff3b06eee592b30ce5 not found: ID does not exist" Oct 30 17:17:20.863524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.863493 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb"] Oct 30 17:17:20.880177 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:20.880150 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c8db42-openshift-default-f44f6fd9f-fhgcb"] Oct 30 17:17:21.111232 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.111196 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:21.150494 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.150464 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:17:21.308006 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.307919 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2315e06-69db-44cc-ad5f-f875d7758205-istio-token\" (UniqueName: \"kubernetes.io/projected/e2315e06-69db-44cc-ad5f-f875d7758205-istio-token\") pod \"e2315e06-69db-44cc-ad5f-f875d7758205\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " Oct 30 17:17:21.308006 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.307962 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-workload-certs\") pod \"e2315e06-69db-44cc-ad5f-f875d7758205\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " Oct 30 17:17:21.308006 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.307993 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2315e06-69db-44cc-ad5f-f875d7758205-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2315e06-69db-44cc-ad5f-f875d7758205-istio-podinfo\") pod \"e2315e06-69db-44cc-ad5f-f875d7758205\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " Oct 30 17:17:21.308287 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.308036 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-credential-socket\") pod \"e2315e06-69db-44cc-ad5f-f875d7758205\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " Oct 30 17:17:21.308287 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.308114 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-istio-data\") pod \"e2315e06-69db-44cc-ad5f-f875d7758205\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " Oct 30 17:17:21.308287 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.308166 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2315e06-69db-44cc-ad5f-f875d7758205-kube-api-access-pwk7t\" (UniqueName: \"kubernetes.io/projected/e2315e06-69db-44cc-ad5f-f875d7758205-kube-api-access-pwk7t\") pod \"e2315e06-69db-44cc-ad5f-f875d7758205\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " Oct 30 17:17:21.308287 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.308242 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2315e06-69db-44cc-ad5f-f875d7758205-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2315e06-69db-44cc-ad5f-f875d7758205-istiod-ca-cert\") pod \"e2315e06-69db-44cc-ad5f-f875d7758205\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " Oct 30 17:17:21.308287 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.308243 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-workload-certs" pod "e2315e06-69db-44cc-ad5f-f875d7758205" (UID: "e2315e06-69db-44cc-ad5f-f875d7758205"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:21.308579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.308289 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-workload-socket\") pod \"e2315e06-69db-44cc-ad5f-f875d7758205\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " Oct 30 17:17:21.308579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.308355 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-istio-envoy\") pod \"e2315e06-69db-44cc-ad5f-f875d7758205\" (UID: \"e2315e06-69db-44cc-ad5f-f875d7758205\") " Oct 30 17:17:21.308579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.308430 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-credential-socket" pod "e2315e06-69db-44cc-ad5f-f875d7758205" (UID: "e2315e06-69db-44cc-ad5f-f875d7758205"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:21.308579 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.308444 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-istio-data" pod "e2315e06-69db-44cc-ad5f-f875d7758205" (UID: "e2315e06-69db-44cc-ad5f-f875d7758205"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:21.308858 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.308638 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-workload-socket" pod "e2315e06-69db-44cc-ad5f-f875d7758205" (UID: "e2315e06-69db-44cc-ad5f-f875d7758205"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:21.308858 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.308690 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2315e06-69db-44cc-ad5f-f875d7758205-istiod-ca-cert" pod "e2315e06-69db-44cc-ad5f-f875d7758205" (UID: "e2315e06-69db-44cc-ad5f-f875d7758205"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:21.309008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.308918 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:21.309075 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.309006 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:21.309075 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.309021 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:21.309075 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.309034 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:21.310439 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.310416 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2315e06-69db-44cc-ad5f-f875d7758205-istio-podinfo" pod "e2315e06-69db-44cc-ad5f-f875d7758205" (UID: "e2315e06-69db-44cc-ad5f-f875d7758205"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:17:21.310897 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.310865 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-istio-envoy" pod "e2315e06-69db-44cc-ad5f-f875d7758205" (UID: "e2315e06-69db-44cc-ad5f-f875d7758205"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:21.311117 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.311101 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2315e06-69db-44cc-ad5f-f875d7758205-kube-api-access-pwk7t" pod "e2315e06-69db-44cc-ad5f-f875d7758205" (UID: "e2315e06-69db-44cc-ad5f-f875d7758205"). InnerVolumeSpecName "kube-api-access-pwk7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:21.311193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.311175 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2315e06-69db-44cc-ad5f-f875d7758205-istio-token" pod "e2315e06-69db-44cc-ad5f-f875d7758205" (UID: "e2315e06-69db-44cc-ad5f-f875d7758205"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:21.409972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.409940 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2315e06-69db-44cc-ad5f-f875d7758205-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:21.409972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.409968 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2315e06-69db-44cc-ad5f-f875d7758205-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:21.409972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.409976 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2315e06-69db-44cc-ad5f-f875d7758205-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:21.410237 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.409984 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2315e06-69db-44cc-ad5f-f875d7758205-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:21.410237 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.409995 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwk7t\" (UniqueName: \"kubernetes.io/projected/e2315e06-69db-44cc-ad5f-f875d7758205-kube-api-access-pwk7t\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:21.816409 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.816375 6202 generic.go:356] "Generic (PLEG): container finished" podID="e2315e06-69db-44cc-ad5f-f875d7758205" containerID="f901609bbc9425608a465bf3601cff6bbb1347324a8d93ffd4f87befbd6de8d4" exitCode=0 Oct 30 17:17:21.816639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.816427 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" Oct 30 17:17:21.816639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.816468 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" event={"ID":"e2315e06-69db-44cc-ad5f-f875d7758205","Type":"ContainerDied","Data":"f901609bbc9425608a465bf3601cff6bbb1347324a8d93ffd4f87befbd6de8d4"} Oct 30 17:17:21.816639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.816528 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl" event={"ID":"e2315e06-69db-44cc-ad5f-f875d7758205","Type":"ContainerDied","Data":"c2f6283d375cdf1bec5a0e350cd6fb3e2db2bd3e4b17a316ac853cef68e1622e"} Oct 30 17:17:21.816639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.816558 6202 scope.go:122] "RemoveContainer" containerID="f901609bbc9425608a465bf3601cff6bbb1347324a8d93ffd4f87befbd6de8d4" Oct 30 17:17:21.827659 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.827638 6202 scope.go:122] "RemoveContainer" containerID="f901609bbc9425608a465bf3601cff6bbb1347324a8d93ffd4f87befbd6de8d4" Oct 30 17:17:21.830710 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:17:21.830687 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f901609bbc9425608a465bf3601cff6bbb1347324a8d93ffd4f87befbd6de8d4\": container with ID starting with f901609bbc9425608a465bf3601cff6bbb1347324a8d93ffd4f87befbd6de8d4 not found: ID does not exist" containerID="f901609bbc9425608a465bf3601cff6bbb1347324a8d93ffd4f87befbd6de8d4" Oct 30 17:17:21.830810 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.830717 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f901609bbc9425608a465bf3601cff6bbb1347324a8d93ffd4f87befbd6de8d4"} err="failed to get container status \"f901609bbc9425608a465bf3601cff6bbb1347324a8d93ffd4f87befbd6de8d4\": rpc error: code = NotFound desc = could not find container \"f901609bbc9425608a465bf3601cff6bbb1347324a8d93ffd4f87befbd6de8d4\": container with ID starting with f901609bbc9425608a465bf3601cff6bbb1347324a8d93ffd4f87befbd6de8d4 not found: ID does not exist" Oct 30 17:17:21.894516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.894478 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl"] Oct 30 17:17:21.907550 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:21.907521 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lf2wl"] Oct 30 17:17:22.351955 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:22.351909 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" podUID="3f4c1389-5421-498a-bc8d-05ca2ec455f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:17:22.352472 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:22.352307 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" podUID="e14671cc-c74d-4fb5-b363-e060629ad7cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:17:22.409789 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:22.409745 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d92701-cab0-4360-b5b5-87d49d16ec10" path="/var/lib/kubelet/pods/99d92701-cab0-4360-b5b5-87d49d16ec10/volumes" Oct 30 17:17:22.410222 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:22.410207 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" path="/var/lib/kubelet/pods/a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2/volumes" Oct 30 17:17:22.410619 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:22.410598 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2315e06-69db-44cc-ad5f-f875d7758205" path="/var/lib/kubelet/pods/e2315e06-69db-44cc-ad5f-f875d7758205/volumes" Oct 30 17:17:22.465311 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:22.465276 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:23.200337 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.200258 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" Oct 30 17:17:23.219957 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.219931 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" Oct 30 17:17:23.328398 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.328370 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f4c1389-5421-498a-bc8d-05ca2ec455f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f4c1389-5421-498a-bc8d-05ca2ec455f6-plugins-config-volume\") pod \"3f4c1389-5421-498a-bc8d-05ca2ec455f6\" (UID: \"3f4c1389-5421-498a-bc8d-05ca2ec455f6\") " Oct 30 17:17:23.328398 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.328410 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e14671cc-c74d-4fb5-b363-e060629ad7cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14671cc-c74d-4fb5-b363-e060629ad7cc-plugins-config-volume\") pod \"e14671cc-c74d-4fb5-b363-e060629ad7cc\" (UID: \"e14671cc-c74d-4fb5-b363-e060629ad7cc\") " Oct 30 17:17:23.328670 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.328465 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e14671cc-c74d-4fb5-b363-e060629ad7cc-kube-api-access-6s8k4\" (UniqueName: \"kubernetes.io/projected/e14671cc-c74d-4fb5-b363-e060629ad7cc-kube-api-access-6s8k4\") pod \"e14671cc-c74d-4fb5-b363-e060629ad7cc\" (UID: \"e14671cc-c74d-4fb5-b363-e060629ad7cc\") " Oct 30 17:17:23.328670 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.328572 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4c1389-5421-498a-bc8d-05ca2ec455f6-kube-api-access-jtjn2\" (UniqueName: \"kubernetes.io/projected/3f4c1389-5421-498a-bc8d-05ca2ec455f6-kube-api-access-jtjn2\") pod \"3f4c1389-5421-498a-bc8d-05ca2ec455f6\" (UID: \"3f4c1389-5421-498a-bc8d-05ca2ec455f6\") " Oct 30 17:17:23.328836 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.328753 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4c1389-5421-498a-bc8d-05ca2ec455f6-plugins-config-volume" pod "3f4c1389-5421-498a-bc8d-05ca2ec455f6" (UID: "3f4c1389-5421-498a-bc8d-05ca2ec455f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:23.328926 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.328893 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e14671cc-c74d-4fb5-b363-e060629ad7cc-plugins-config-volume" pod "e14671cc-c74d-4fb5-b363-e060629ad7cc" (UID: "e14671cc-c74d-4fb5-b363-e060629ad7cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:23.328926 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.328918 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f4c1389-5421-498a-bc8d-05ca2ec455f6-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:23.330812 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.330756 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14671cc-c74d-4fb5-b363-e060629ad7cc-kube-api-access-6s8k4" pod "e14671cc-c74d-4fb5-b363-e060629ad7cc" (UID: "e14671cc-c74d-4fb5-b363-e060629ad7cc"). InnerVolumeSpecName "kube-api-access-6s8k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:23.330929 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.330840 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4c1389-5421-498a-bc8d-05ca2ec455f6-kube-api-access-jtjn2" pod "3f4c1389-5421-498a-bc8d-05ca2ec455f6" (UID: "3f4c1389-5421-498a-bc8d-05ca2ec455f6"). InnerVolumeSpecName "kube-api-access-jtjn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:23.430051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.429978 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s8k4\" (UniqueName: \"kubernetes.io/projected/e14671cc-c74d-4fb5-b363-e060629ad7cc-kube-api-access-6s8k4\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:23.430051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.430003 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtjn2\" (UniqueName: \"kubernetes.io/projected/3f4c1389-5421-498a-bc8d-05ca2ec455f6-kube-api-access-jtjn2\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:23.430051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.430012 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14671cc-c74d-4fb5-b363-e060629ad7cc-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:23.833436 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.833399 6202 generic.go:356] "Generic (PLEG): container finished" podID="e14671cc-c74d-4fb5-b363-e060629ad7cc" containerID="37a3befbc8522cf8318ac96e80acd10114a37051ff7d863e98650bbe37db96b6" exitCode=0 Oct 30 17:17:23.833618 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.833454 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" Oct 30 17:17:23.833618 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.833488 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" event={"ID":"e14671cc-c74d-4fb5-b363-e060629ad7cc","Type":"ContainerDied","Data":"37a3befbc8522cf8318ac96e80acd10114a37051ff7d863e98650bbe37db96b6"} Oct 30 17:17:23.833618 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.833546 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2" event={"ID":"e14671cc-c74d-4fb5-b363-e060629ad7cc","Type":"ContainerDied","Data":"177c08a83743d6c16a7d1477a149f02ff138e2d72b1f70fbb7840516e2c09164"} Oct 30 17:17:23.833618 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.833573 6202 scope.go:122] "RemoveContainer" containerID="37a3befbc8522cf8318ac96e80acd10114a37051ff7d863e98650bbe37db96b6" Oct 30 17:17:23.835257 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.835232 6202 generic.go:356] "Generic (PLEG): container finished" podID="3f4c1389-5421-498a-bc8d-05ca2ec455f6" containerID="3aff97e1feebf4ff7b82943dbeb7dc020c1b6c5f719028b0669defb4de4d39d5" exitCode=0 Oct 30 17:17:23.835393 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.835283 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" Oct 30 17:17:23.835393 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.835307 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" event={"ID":"3f4c1389-5421-498a-bc8d-05ca2ec455f6","Type":"ContainerDied","Data":"3aff97e1feebf4ff7b82943dbeb7dc020c1b6c5f719028b0669defb4de4d39d5"} Oct 30 17:17:23.835393 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.835357 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj" event={"ID":"3f4c1389-5421-498a-bc8d-05ca2ec455f6","Type":"ContainerDied","Data":"15f9edff52b5317a4d6e5a03a9aaaab513414ac3625264b3c758763cbcea5530"} Oct 30 17:17:23.846345 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.846321 6202 scope.go:122] "RemoveContainer" containerID="37a3befbc8522cf8318ac96e80acd10114a37051ff7d863e98650bbe37db96b6" Oct 30 17:17:23.846645 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:17:23.846627 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a3befbc8522cf8318ac96e80acd10114a37051ff7d863e98650bbe37db96b6\": container with ID starting with 37a3befbc8522cf8318ac96e80acd10114a37051ff7d863e98650bbe37db96b6 not found: ID does not exist" containerID="37a3befbc8522cf8318ac96e80acd10114a37051ff7d863e98650bbe37db96b6" Oct 30 17:17:23.846740 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.846657 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a3befbc8522cf8318ac96e80acd10114a37051ff7d863e98650bbe37db96b6"} err="failed to get container status \"37a3befbc8522cf8318ac96e80acd10114a37051ff7d863e98650bbe37db96b6\": rpc error: code = NotFound desc = could not find container \"37a3befbc8522cf8318ac96e80acd10114a37051ff7d863e98650bbe37db96b6\": container with ID starting with 37a3befbc8522cf8318ac96e80acd10114a37051ff7d863e98650bbe37db96b6 not found: ID does not exist" Oct 30 17:17:23.846740 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.846680 6202 scope.go:122] "RemoveContainer" containerID="3aff97e1feebf4ff7b82943dbeb7dc020c1b6c5f719028b0669defb4de4d39d5" Oct 30 17:17:23.857239 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.857219 6202 scope.go:122] "RemoveContainer" containerID="3aff97e1feebf4ff7b82943dbeb7dc020c1b6c5f719028b0669defb4de4d39d5" Oct 30 17:17:23.857513 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:17:23.857494 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aff97e1feebf4ff7b82943dbeb7dc020c1b6c5f719028b0669defb4de4d39d5\": container with ID starting with 3aff97e1feebf4ff7b82943dbeb7dc020c1b6c5f719028b0669defb4de4d39d5 not found: ID does not exist" containerID="3aff97e1feebf4ff7b82943dbeb7dc020c1b6c5f719028b0669defb4de4d39d5" Oct 30 17:17:23.857604 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.857520 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aff97e1feebf4ff7b82943dbeb7dc020c1b6c5f719028b0669defb4de4d39d5"} err="failed to get container status \"3aff97e1feebf4ff7b82943dbeb7dc020c1b6c5f719028b0669defb4de4d39d5\": rpc error: code = NotFound desc = could not find container \"3aff97e1feebf4ff7b82943dbeb7dc020c1b6c5f719028b0669defb4de4d39d5\": container with ID starting with 3aff97e1feebf4ff7b82943dbeb7dc020c1b6c5f719028b0669defb4de4d39d5 not found: ID does not exist" Oct 30 17:17:23.888596 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.888566 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2"] Oct 30 17:17:23.901663 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.901632 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c8db42-7496d6cdb4-x9sw2"] Oct 30 17:17:23.944686 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.944639 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj"] Oct 30 17:17:23.946643 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:23.946618 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c8db42-5b6d4d5c5-ww4hj"] Oct 30 17:17:24.075283 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:24.075238 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" podUID="052f7204-67ca-4d46-8b48-5f44e951a8f1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:17:24.075481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:24.075245 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" podUID="cf706793-9392-4041-8753-f65a7f1afe24" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:17:24.214898 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:24.214795 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:24.409421 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:24.409390 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4c1389-5421-498a-bc8d-05ca2ec455f6" path="/var/lib/kubelet/pods/3f4c1389-5421-498a-bc8d-05ca2ec455f6/volumes" Oct 30 17:17:24.409806 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:24.409791 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e14671cc-c74d-4fb5-b363-e060629ad7cc" path="/var/lib/kubelet/pods/e14671cc-c74d-4fb5-b363-e060629ad7cc/volumes" Oct 30 17:17:25.967931 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:25.967889 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw"] Oct 30 17:17:25.993387 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:25.993354 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:26.003921 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:26.003880 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj"] Oct 30 17:17:26.104893 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:26.104856 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:26.211953 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:26.211920 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:26.794157 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:26.794117 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:26.904316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:26.904270 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:27.642077 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:27.642040 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:28.234541 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:28.234501 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:28.350780 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:28.350726 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:28.613221 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:28.613171 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:29.441341 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:29.441303 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:30.832163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.832135 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg"] Oct 30 17:17:30.832607 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.832592 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" containerName="istio-proxy" Oct 30 17:17:30.832665 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.832608 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" containerName="istio-proxy" Oct 30 17:17:30.832665 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.832621 6202 cpu_manager.go:414] "containerMap: removing container" podUID="e14671cc-c74d-4fb5-b363-e060629ad7cc" containerName="payload-processing" Oct 30 17:17:30.832665 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.832626 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e14671cc-c74d-4fb5-b363-e060629ad7cc" containerName="payload-processing" Oct 30 17:17:30.832665 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.832663 6202 cpu_manager.go:414] "containerMap: removing container" podUID="e2315e06-69db-44cc-ad5f-f875d7758205" containerName="istio-proxy" Oct 30 17:17:30.832843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.832668 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2315e06-69db-44cc-ad5f-f875d7758205" containerName="istio-proxy" Oct 30 17:17:30.832843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.832677 6202 cpu_manager.go:414] "containerMap: removing container" podUID="3f4c1389-5421-498a-bc8d-05ca2ec455f6" containerName="payload-pre-processing" Oct 30 17:17:30.832843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.832681 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4c1389-5421-498a-bc8d-05ca2ec455f6" containerName="payload-pre-processing" Oct 30 17:17:30.832843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.832690 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" containerName="istio-proxy" Oct 30 17:17:30.832843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.832695 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" containerName="istio-proxy" Oct 30 17:17:30.832843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.832703 6202 cpu_manager.go:414] "containerMap: removing container" podUID="b71c1827-6c8f-4116-9fd3-79ffedf89b5f" containerName="revoke-keys" Oct 30 17:17:30.832843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.832709 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b71c1827-6c8f-4116-9fd3-79ffedf89b5f" containerName="revoke-keys" Oct 30 17:17:30.832843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.832797 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="e14671cc-c74d-4fb5-b363-e060629ad7cc" containerName="payload-processing" Oct 30 17:17:30.832843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.832810 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2315e06-69db-44cc-ad5f-f875d7758205" containerName="istio-proxy" Oct 30 17:17:30.832843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.832817 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4c1389-5421-498a-bc8d-05ca2ec455f6" containerName="payload-pre-processing" Oct 30 17:17:30.832843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.832823 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" containerName="istio-proxy" Oct 30 17:17:30.832843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.832829 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="b71c1827-6c8f-4116-9fd3-79ffedf89b5f" containerName="revoke-keys" Oct 30 17:17:30.836379 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.836310 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg" Oct 30 17:17:30.857479 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.857284 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg"] Oct 30 17:17:30.871606 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.870677 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:17:30.871606 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.870818 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:17:30.871606 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.870850 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:17:30.871606 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.870941 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:17:30.904628 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.904581 6202 generic.go:356] "Generic (PLEG): container finished" podID="cf706793-9392-4041-8753-f65a7f1afe24" containerID="75b1301afbc1d6d370ce272a13cf03e3ca26b8949d5ccc25316e725ff22a56fd" exitCode=0 Oct 30 17:17:30.904791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.904650 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" event={"ID":"cf706793-9392-4041-8753-f65a7f1afe24","Type":"ContainerDied","Data":"75b1301afbc1d6d370ce272a13cf03e3ca26b8949d5ccc25316e725ff22a56fd"} Oct 30 17:17:30.904791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.904711 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" event={"ID":"cf706793-9392-4041-8753-f65a7f1afe24","Type":"ContainerDied","Data":"c96f5c650e5d680e5d74b5a2d479f48d8b0c5def611f7847e74150d192634b4d"} Oct 30 17:17:30.904791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.904738 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c96f5c650e5d680e5d74b5a2d479f48d8b0c5def611f7847e74150d192634b4d" Oct 30 17:17:30.905793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.905755 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" Oct 30 17:17:30.906238 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.906214 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzjfz\" (UniqueName: \"kubernetes.io/projected/a3c2d2f9-1429-4441-88f2-78b89feabbee-kube-api-access-dzjfz\") pod \"maas-api-revoke-sub-3d33b897-8hqvg\" (UID: \"a3c2d2f9-1429-4441-88f2-78b89feabbee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg" Oct 30 17:17:30.906350 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.906272 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c2d2f9-1429-4441-88f2-78b89feabbee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d33b897-8hqvg\" (UID: \"a3c2d2f9-1429-4441-88f2-78b89feabbee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg" Oct 30 17:17:30.906424 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.906386 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c2d2f9-1429-4441-88f2-78b89feabbee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d33b897-8hqvg\" (UID: \"a3c2d2f9-1429-4441-88f2-78b89feabbee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg" Oct 30 17:17:30.908027 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.908003 6202 generic.go:356] "Generic (PLEG): container finished" podID="052f7204-67ca-4d46-8b48-5f44e951a8f1" containerID="070515d09bed052bee8ec9fa902a8ba75eaa17ad24eb57f3f3a68b7959da17e5" exitCode=0 Oct 30 17:17:30.908120 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.908078 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" event={"ID":"052f7204-67ca-4d46-8b48-5f44e951a8f1","Type":"ContainerDied","Data":"070515d09bed052bee8ec9fa902a8ba75eaa17ad24eb57f3f3a68b7959da17e5"} Oct 30 17:17:30.910353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.910334 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" Oct 30 17:17:30.956809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.953075 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:30.968834 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:30.968793 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" podUID="a46e1ced-667f-4218-9fe5-4dafa958b55e" containerName="payload-processing" containerID="cri-o://c28e76120e6e4732e94b57d46b9af757d11697d2663091c37de9ea1bf377fae5" gracePeriod=25 Oct 30 17:17:31.004649 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.004614 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" podUID="5b428320-6e0b-463f-be2e-74ec2cfda977" containerName="payload-pre-processing" containerID="cri-o://ad737be37cbc8782fdb8841ac5d2ecb9a056d237074637a76214be8e40184441" gracePeriod=25 Oct 30 17:17:31.007384 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.007358 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052f7204-67ca-4d46-8b48-5f44e951a8f1-kube-api-access-pdtws\" (UniqueName: \"kubernetes.io/projected/052f7204-67ca-4d46-8b48-5f44e951a8f1-kube-api-access-pdtws\") pod \"052f7204-67ca-4d46-8b48-5f44e951a8f1\" (UID: \"052f7204-67ca-4d46-8b48-5f44e951a8f1\") " Oct 30 17:17:31.007514 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.007431 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/052f7204-67ca-4d46-8b48-5f44e951a8f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/052f7204-67ca-4d46-8b48-5f44e951a8f1-plugins-config-volume\") pod \"052f7204-67ca-4d46-8b48-5f44e951a8f1\" (UID: \"052f7204-67ca-4d46-8b48-5f44e951a8f1\") " Oct 30 17:17:31.007585 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.007517 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf706793-9392-4041-8753-f65a7f1afe24-kube-api-access-zvz22\" (UniqueName: \"kubernetes.io/projected/cf706793-9392-4041-8753-f65a7f1afe24-kube-api-access-zvz22\") pod \"cf706793-9392-4041-8753-f65a7f1afe24\" (UID: \"cf706793-9392-4041-8753-f65a7f1afe24\") " Oct 30 17:17:31.007643 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.007616 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf706793-9392-4041-8753-f65a7f1afe24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf706793-9392-4041-8753-f65a7f1afe24-plugins-config-volume\") pod \"cf706793-9392-4041-8753-f65a7f1afe24\" (UID: \"cf706793-9392-4041-8753-f65a7f1afe24\") " Oct 30 17:17:31.007841 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.007816 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052f7204-67ca-4d46-8b48-5f44e951a8f1-plugins-config-volume" pod "052f7204-67ca-4d46-8b48-5f44e951a8f1" (UID: "052f7204-67ca-4d46-8b48-5f44e951a8f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:31.007931 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.007848 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c2d2f9-1429-4441-88f2-78b89feabbee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d33b897-8hqvg\" (UID: \"a3c2d2f9-1429-4441-88f2-78b89feabbee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg" Oct 30 17:17:31.008065 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.008036 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzjfz\" (UniqueName: \"kubernetes.io/projected/a3c2d2f9-1429-4441-88f2-78b89feabbee-kube-api-access-dzjfz\") pod \"maas-api-revoke-sub-3d33b897-8hqvg\" (UID: \"a3c2d2f9-1429-4441-88f2-78b89feabbee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg" Oct 30 17:17:31.008153 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.008118 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf706793-9392-4041-8753-f65a7f1afe24-plugins-config-volume" pod "cf706793-9392-4041-8753-f65a7f1afe24" (UID: "cf706793-9392-4041-8753-f65a7f1afe24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:31.008697 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.008664 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c2d2f9-1429-4441-88f2-78b89feabbee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d33b897-8hqvg\" (UID: \"a3c2d2f9-1429-4441-88f2-78b89feabbee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg" Oct 30 17:17:31.008831 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.008806 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf706793-9392-4041-8753-f65a7f1afe24-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:31.008904 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.008841 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/052f7204-67ca-4d46-8b48-5f44e951a8f1-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:31.009625 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.009456 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c2d2f9-1429-4441-88f2-78b89feabbee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d33b897-8hqvg\" (UID: \"a3c2d2f9-1429-4441-88f2-78b89feabbee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg" Oct 30 17:17:31.010203 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.010174 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf706793-9392-4041-8753-f65a7f1afe24-kube-api-access-zvz22" pod "cf706793-9392-4041-8753-f65a7f1afe24" (UID: "cf706793-9392-4041-8753-f65a7f1afe24"). InnerVolumeSpecName "kube-api-access-zvz22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:31.010724 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.010704 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052f7204-67ca-4d46-8b48-5f44e951a8f1-kube-api-access-pdtws" pod "052f7204-67ca-4d46-8b48-5f44e951a8f1" (UID: "052f7204-67ca-4d46-8b48-5f44e951a8f1"). InnerVolumeSpecName "kube-api-access-pdtws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:31.010845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.010824 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c2d2f9-1429-4441-88f2-78b89feabbee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d33b897-8hqvg\" (UID: \"a3c2d2f9-1429-4441-88f2-78b89feabbee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg" Oct 30 17:17:31.032533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.032505 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzjfz\" (UniqueName: \"kubernetes.io/projected/a3c2d2f9-1429-4441-88f2-78b89feabbee-kube-api-access-dzjfz\") pod \"maas-api-revoke-sub-3d33b897-8hqvg\" (UID: \"a3c2d2f9-1429-4441-88f2-78b89feabbee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg" Oct 30 17:17:31.109582 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.109496 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdtws\" (UniqueName: \"kubernetes.io/projected/052f7204-67ca-4d46-8b48-5f44e951a8f1-kube-api-access-pdtws\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:31.109582 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.109532 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvz22\" (UniqueName: \"kubernetes.io/projected/cf706793-9392-4041-8753-f65a7f1afe24-kube-api-access-zvz22\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:31.194466 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.194433 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg" Oct 30 17:17:31.428975 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.428863 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:31.492650 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.492615 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg"] Oct 30 17:17:31.492925 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:17:31.492895 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c2d2f9_1429_4441_88f2_78b89feabbee.slice/crio-b53898585406480a6499a39825bbdfd8641e189fcda3e89a01e25ef4573f3914 WatchSource:0}: Error finding container b53898585406480a6499a39825bbdfd8641e189fcda3e89a01e25ef4573f3914: Status 404 returned error can't find the container with id b53898585406480a6499a39825bbdfd8641e189fcda3e89a01e25ef4573f3914 Oct 30 17:17:31.541199 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.541168 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:31.917256 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.917224 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" Oct 30 17:17:31.917702 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.917263 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk" event={"ID":"052f7204-67ca-4d46-8b48-5f44e951a8f1","Type":"ContainerDied","Data":"b2c294902fb0e9e4c8b8db626b04347803fd9940f5422e8ad6bfca921b80d705"} Oct 30 17:17:31.917702 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.917328 6202 scope.go:122] "RemoveContainer" containerID="070515d09bed052bee8ec9fa902a8ba75eaa17ad24eb57f3f3a68b7959da17e5" Oct 30 17:17:31.919383 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.919356 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg" event={"ID":"a3c2d2f9-1429-4441-88f2-78b89feabbee","Type":"ContainerStarted","Data":"f267735ff757c5b712f19418671adfb1f1007e408b068dc89f077e695d920d47"} Oct 30 17:17:31.919543 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.919408 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7" Oct 30 17:17:31.919543 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.919408 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg" event={"ID":"a3c2d2f9-1429-4441-88f2-78b89feabbee","Type":"ContainerStarted","Data":"b53898585406480a6499a39825bbdfd8641e189fcda3e89a01e25ef4573f3914"} Oct 30 17:17:31.956026 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:17:31.955993 6202 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052f7204_67ca_4d46_8b48_5f44e951a8f1.slice/crio-b2c294902fb0e9e4c8b8db626b04347803fd9940f5422e8ad6bfca921b80d705\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf706793_9392_4041_8753_f65a7f1afe24.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052f7204_67ca_4d46_8b48_5f44e951a8f1.slice\": RecentStats: unable to find data in memory cache]" Oct 30 17:17:31.972931 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.972905 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7"] Oct 30 17:17:31.985951 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:31.985924 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-bfecdd-bc5cdfbbd-qdcr7"] Oct 30 17:17:32.022699 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:32.022621 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk"] Oct 30 17:17:32.026383 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:32.026355 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-bfecdd-56f764f46c-kj2gk"] Oct 30 17:17:32.067643 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:32.067589 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg" podStartSLOduration=2.067575294 podStartE2EDuration="2.067575294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:32.066729974 +0000 UTC m=+3365.777184717" watchObservedRunningTime="2026-10-30 17:17:32.067575294 +0000 UTC m=+3365.778030038" Oct 30 17:17:32.088857 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:32.088807 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:32.409740 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:32.409698 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="052f7204-67ca-4d46-8b48-5f44e951a8f1" path="/var/lib/kubelet/pods/052f7204-67ca-4d46-8b48-5f44e951a8f1/volumes" Oct 30 17:17:32.410192 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:32.410177 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf706793-9392-4041-8753-f65a7f1afe24" path="/var/lib/kubelet/pods/cf706793-9392-4041-8753-f65a7f1afe24/volumes" Oct 30 17:17:32.928171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:32.928134 6202 generic.go:356] "Generic (PLEG): container finished" podID="a3c2d2f9-1429-4441-88f2-78b89feabbee" containerID="f267735ff757c5b712f19418671adfb1f1007e408b068dc89f077e695d920d47" exitCode=0 Oct 30 17:17:32.928572 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:32.928183 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg" event={"ID":"a3c2d2f9-1429-4441-88f2-78b89feabbee","Type":"ContainerDied","Data":"f267735ff757c5b712f19418671adfb1f1007e408b068dc89f077e695d920d47"} Oct 30 17:17:33.337368 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:33.337333 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:33.479494 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:33.479459 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" podUID="5b428320-6e0b-463f-be2e-74ec2cfda977" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:17:33.479694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:33.479459 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" podUID="a46e1ced-667f-4218-9fe5-4dafa958b55e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:17:34.081150 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:34.081126 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg" Oct 30 17:17:34.143079 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:34.143048 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3c2d2f9-1429-4441-88f2-78b89feabbee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c2d2f9-1429-4441-88f2-78b89feabbee-maas-api-openshift-ca\") pod \"a3c2d2f9-1429-4441-88f2-78b89feabbee\" (UID: \"a3c2d2f9-1429-4441-88f2-78b89feabbee\") " Oct 30 17:17:34.143248 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:34.143104 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3c2d2f9-1429-4441-88f2-78b89feabbee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c2d2f9-1429-4441-88f2-78b89feabbee-maas-api-xks-ca\") pod \"a3c2d2f9-1429-4441-88f2-78b89feabbee\" (UID: \"a3c2d2f9-1429-4441-88f2-78b89feabbee\") " Oct 30 17:17:34.143248 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:34.143150 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c2d2f9-1429-4441-88f2-78b89feabbee-kube-api-access-dzjfz\" (UniqueName: \"kubernetes.io/projected/a3c2d2f9-1429-4441-88f2-78b89feabbee-kube-api-access-dzjfz\") pod \"a3c2d2f9-1429-4441-88f2-78b89feabbee\" (UID: \"a3c2d2f9-1429-4441-88f2-78b89feabbee\") " Oct 30 17:17:34.143504 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:34.143475 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c2d2f9-1429-4441-88f2-78b89feabbee-maas-api-openshift-ca" pod "a3c2d2f9-1429-4441-88f2-78b89feabbee" (UID: "a3c2d2f9-1429-4441-88f2-78b89feabbee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:34.145504 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:34.145462 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3c2d2f9-1429-4441-88f2-78b89feabbee-maas-api-xks-ca" pod "a3c2d2f9-1429-4441-88f2-78b89feabbee" (UID: "a3c2d2f9-1429-4441-88f2-78b89feabbee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:34.145646 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:34.145613 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c2d2f9-1429-4441-88f2-78b89feabbee-kube-api-access-dzjfz" pod "a3c2d2f9-1429-4441-88f2-78b89feabbee" (UID: "a3c2d2f9-1429-4441-88f2-78b89feabbee"). InnerVolumeSpecName "kube-api-access-dzjfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:34.244481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:34.244345 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c2d2f9-1429-4441-88f2-78b89feabbee-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:34.244481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:34.244385 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c2d2f9-1429-4441-88f2-78b89feabbee-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:34.244481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:34.244403 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzjfz\" (UniqueName: \"kubernetes.io/projected/a3c2d2f9-1429-4441-88f2-78b89feabbee-kube-api-access-dzjfz\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:34.647447 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:34.647413 6202 scope.go:122] "RemoveContainer" containerID="75b1301afbc1d6d370ce272a13cf03e3ca26b8949d5ccc25316e725ff22a56fd" Oct 30 17:17:34.659748 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:34.659717 6202 scope.go:122] "RemoveContainer" containerID="273da7149749dd0e9902855ac586532004f13f2e6c96e5dfdbebb382ba0248af" Oct 30 17:17:34.856120 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:34.856080 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:34.943401 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:34.943323 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg" Oct 30 17:17:34.943401 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:34.943359 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg" event={"ID":"a3c2d2f9-1429-4441-88f2-78b89feabbee","Type":"ContainerDied","Data":"b53898585406480a6499a39825bbdfd8641e189fcda3e89a01e25ef4573f3914"} Oct 30 17:17:34.943591 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:34.943412 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b53898585406480a6499a39825bbdfd8641e189fcda3e89a01e25ef4573f3914" Oct 30 17:17:35.070079 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:35.070044 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:35.186586 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:35.186531 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:35.995219 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:35.995186 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:36.794463 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:36.794426 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:38.403536 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:38.403474 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:38.666060 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:38.665976 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:38.785835 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:38.785803 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:38.989068 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:38.988929 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q"] Oct 30 17:17:38.989254 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:38.989170 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" podUID="320b55f1-a255-4f32-8b97-26cd86755a2d" containerName="istio-proxy" containerID="cri-o://2351ddff425996d75d55bc9dd9a5452efcb8987562784bc7a02a50d68b535e1f" gracePeriod=30 Oct 30 17:17:39.035876 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:39.035845 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:39.527696 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:39.527661 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:40.478014 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:40.477980 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:42.162926 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:42.162885 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:42.387903 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:42.387862 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:43.236700 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:43.236654 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:43.361442 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:43.361394 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:43.479613 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:43.479567 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" podUID="a46e1ced-667f-4218-9fe5-4dafa958b55e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:17:43.479876 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:43.479757 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" podUID="5b428320-6e0b-463f-be2e-74ec2cfda977" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:17:43.867683 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:43.867630 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:44.084159 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:17:44.084114 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:17:44.243715 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.243689 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:17:44.347958 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.347921 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-workload-certs\") pod \"320b55f1-a255-4f32-8b97-26cd86755a2d\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " Oct 30 17:17:44.348178 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.347996 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-workload-socket\") pod \"320b55f1-a255-4f32-8b97-26cd86755a2d\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " Oct 30 17:17:44.348178 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.348036 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-credential-socket\") pod \"320b55f1-a255-4f32-8b97-26cd86755a2d\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " Oct 30 17:17:44.348178 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.348075 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320b55f1-a255-4f32-8b97-26cd86755a2d-kube-api-access-gz44k\" (UniqueName: \"kubernetes.io/projected/320b55f1-a255-4f32-8b97-26cd86755a2d-kube-api-access-gz44k\") pod \"320b55f1-a255-4f32-8b97-26cd86755a2d\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " Oct 30 17:17:44.348178 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.348130 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-token\" (UniqueName: \"kubernetes.io/projected/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-token\") pod \"320b55f1-a255-4f32-8b97-26cd86755a2d\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " Oct 30 17:17:44.348178 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.348169 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-envoy\") pod \"320b55f1-a255-4f32-8b97-26cd86755a2d\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " Oct 30 17:17:44.348485 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.348202 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-data\") pod \"320b55f1-a255-4f32-8b97-26cd86755a2d\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " Oct 30 17:17:44.348485 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.348259 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/320b55f1-a255-4f32-8b97-26cd86755a2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320b55f1-a255-4f32-8b97-26cd86755a2d-istiod-ca-cert\") pod \"320b55f1-a255-4f32-8b97-26cd86755a2d\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " Oct 30 17:17:44.348485 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.348282 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-workload-socket" pod "320b55f1-a255-4f32-8b97-26cd86755a2d" (UID: "320b55f1-a255-4f32-8b97-26cd86755a2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:44.348485 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.348358 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-podinfo\") pod \"320b55f1-a255-4f32-8b97-26cd86755a2d\" (UID: \"320b55f1-a255-4f32-8b97-26cd86755a2d\") " Oct 30 17:17:44.348809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.348721 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.348957 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.348821 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-workload-certs" pod "320b55f1-a255-4f32-8b97-26cd86755a2d" (UID: "320b55f1-a255-4f32-8b97-26cd86755a2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:44.349025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.349013 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-data" pod "320b55f1-a255-4f32-8b97-26cd86755a2d" (UID: "320b55f1-a255-4f32-8b97-26cd86755a2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:44.349349 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.349302 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-credential-socket" pod "320b55f1-a255-4f32-8b97-26cd86755a2d" (UID: "320b55f1-a255-4f32-8b97-26cd86755a2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:44.350114 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.350086 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/320b55f1-a255-4f32-8b97-26cd86755a2d-istiod-ca-cert" pod "320b55f1-a255-4f32-8b97-26cd86755a2d" (UID: "320b55f1-a255-4f32-8b97-26cd86755a2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:44.352861 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.352826 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-podinfo" pod "320b55f1-a255-4f32-8b97-26cd86755a2d" (UID: "320b55f1-a255-4f32-8b97-26cd86755a2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:17:44.352969 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.352900 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320b55f1-a255-4f32-8b97-26cd86755a2d-kube-api-access-gz44k" pod "320b55f1-a255-4f32-8b97-26cd86755a2d" (UID: "320b55f1-a255-4f32-8b97-26cd86755a2d"). InnerVolumeSpecName "kube-api-access-gz44k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:44.353204 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.353155 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-token" pod "320b55f1-a255-4f32-8b97-26cd86755a2d" (UID: "320b55f1-a255-4f32-8b97-26cd86755a2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:44.355173 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.355140 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-envoy" pod "320b55f1-a255-4f32-8b97-26cd86755a2d" (UID: "320b55f1-a255-4f32-8b97-26cd86755a2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:44.441578 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.441539 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:44.450267 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.450240 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.450267 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.450265 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.450504 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.450274 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.450504 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.450284 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320b55f1-a255-4f32-8b97-26cd86755a2d-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.450504 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.450292 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320b55f1-a255-4f32-8b97-26cd86755a2d-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.450504 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.450310 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.450504 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.450322 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b55f1-a255-4f32-8b97-26cd86755a2d-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.450504 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.450331 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz44k\" (UniqueName: \"kubernetes.io/projected/320b55f1-a255-4f32-8b97-26cd86755a2d-kube-api-access-gz44k\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.567008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.566974 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9"] Oct 30 17:17:44.569594 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.569560 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:44.588179 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.587949 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742"] Oct 30 17:17:44.594461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.594420 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-750dcb-jb742"] Oct 30 17:17:44.678008 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:44.677925 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:45.015279 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:45.015188 6202 generic.go:356] "Generic (PLEG): container finished" podID="320b55f1-a255-4f32-8b97-26cd86755a2d" containerID="2351ddff425996d75d55bc9dd9a5452efcb8987562784bc7a02a50d68b535e1f" exitCode=0 Oct 30 17:17:45.015279 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:45.015244 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" Oct 30 17:17:45.015488 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:45.015273 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" event={"ID":"320b55f1-a255-4f32-8b97-26cd86755a2d","Type":"ContainerDied","Data":"2351ddff425996d75d55bc9dd9a5452efcb8987562784bc7a02a50d68b535e1f"} Oct 30 17:17:45.015488 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:45.015332 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q" event={"ID":"320b55f1-a255-4f32-8b97-26cd86755a2d","Type":"ContainerDied","Data":"2c1ebf7390ed91334d94aab7072d611e205a437eece005b12911e79b55e2a61c"} Oct 30 17:17:45.015488 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:45.015358 6202 scope.go:122] "RemoveContainer" containerID="2351ddff425996d75d55bc9dd9a5452efcb8987562784bc7a02a50d68b535e1f" Oct 30 17:17:45.026603 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:45.026583 6202 scope.go:122] "RemoveContainer" containerID="2351ddff425996d75d55bc9dd9a5452efcb8987562784bc7a02a50d68b535e1f" Oct 30 17:17:45.027009 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:17:45.026981 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2351ddff425996d75d55bc9dd9a5452efcb8987562784bc7a02a50d68b535e1f\": container with ID starting with 2351ddff425996d75d55bc9dd9a5452efcb8987562784bc7a02a50d68b535e1f not found: ID does not exist" containerID="2351ddff425996d75d55bc9dd9a5452efcb8987562784bc7a02a50d68b535e1f" Oct 30 17:17:45.027068 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:45.027019 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2351ddff425996d75d55bc9dd9a5452efcb8987562784bc7a02a50d68b535e1f"} err="failed to get container status \"2351ddff425996d75d55bc9dd9a5452efcb8987562784bc7a02a50d68b535e1f\": rpc error: code = NotFound desc = could not find container \"2351ddff425996d75d55bc9dd9a5452efcb8987562784bc7a02a50d68b535e1f\": container with ID starting with 2351ddff425996d75d55bc9dd9a5452efcb8987562784bc7a02a50d68b535e1f not found: ID does not exist" Oct 30 17:17:45.071335 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:45.071311 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q"] Oct 30 17:17:45.072435 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:45.072410 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:45.089014 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:45.086854 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-j247q"] Oct 30 17:17:46.361300 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:46.361265 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:46.409577 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:46.409535 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="320b55f1-a255-4f32-8b97-26cd86755a2d" path="/var/lib/kubelet/pods/320b55f1-a255-4f32-8b97-26cd86755a2d/volumes" Oct 30 17:17:46.410061 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:46.410039 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b71c1827-6c8f-4116-9fd3-79ffedf89b5f" path="/var/lib/kubelet/pods/b71c1827-6c8f-4116-9fd3-79ffedf89b5f/volumes" Oct 30 17:17:48.140718 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:48.140686 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:49.750399 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:49.750361 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:50.162721 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:50.162681 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth"] Oct 30 17:17:50.193896 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:50.193854 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf"] Oct 30 17:17:50.297815 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:50.297754 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:50.412003 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:50.411968 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:51.241802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.240722 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh"] Oct 30 17:17:51.241802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.241367 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a3c2d2f9-1429-4441-88f2-78b89feabbee" containerName="revoke-keys" Oct 30 17:17:51.241802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.241385 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c2d2f9-1429-4441-88f2-78b89feabbee" containerName="revoke-keys" Oct 30 17:17:51.241802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.241397 6202 cpu_manager.go:414] "containerMap: removing container" podUID="cf706793-9392-4041-8753-f65a7f1afe24" containerName="payload-processing" Oct 30 17:17:51.241802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.241405 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf706793-9392-4041-8753-f65a7f1afe24" containerName="payload-processing" Oct 30 17:17:51.241802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.241433 6202 cpu_manager.go:414] "containerMap: removing container" podUID="052f7204-67ca-4d46-8b48-5f44e951a8f1" containerName="payload-pre-processing" Oct 30 17:17:51.241802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.241441 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052f7204-67ca-4d46-8b48-5f44e951a8f1" containerName="payload-pre-processing" Oct 30 17:17:51.241802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.241467 6202 cpu_manager.go:414] "containerMap: removing container" podUID="320b55f1-a255-4f32-8b97-26cd86755a2d" containerName="istio-proxy" Oct 30 17:17:51.241802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.241475 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320b55f1-a255-4f32-8b97-26cd86755a2d" containerName="istio-proxy" Oct 30 17:17:51.241802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.241568 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c2d2f9-1429-4441-88f2-78b89feabbee" containerName="revoke-keys" Oct 30 17:17:51.241802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.241578 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf706793-9392-4041-8753-f65a7f1afe24" containerName="payload-processing" Oct 30 17:17:51.241802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.241592 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="052f7204-67ca-4d46-8b48-5f44e951a8f1" containerName="payload-pre-processing" Oct 30 17:17:51.241802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.241603 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="320b55f1-a255-4f32-8b97-26cd86755a2d" containerName="istio-proxy" Oct 30 17:17:51.241802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.241619 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83d85c0-d7aa-4bdd-9e23-0fe2e30bdaa2" containerName="istio-proxy" Oct 30 17:17:51.260665 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.260630 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh"] Oct 30 17:17:51.260906 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.260760 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh" Oct 30 17:17:51.265068 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.264950 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" Oct 30 17:17:51.277804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.277761 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:17:51.290915 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.290891 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:17:51.302264 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.302239 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" Oct 30 17:17:51.324701 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.324667 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:17:51.324903 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.324788 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:17:51.330157 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.330134 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a46e1ced-667f-4218-9fe5-4dafa958b55e-kube-api-access-blngh\" (UniqueName: \"kubernetes.io/projected/a46e1ced-667f-4218-9fe5-4dafa958b55e-kube-api-access-blngh\") pod \"a46e1ced-667f-4218-9fe5-4dafa958b55e\" (UID: \"a46e1ced-667f-4218-9fe5-4dafa958b55e\") " Oct 30 17:17:51.330301 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.330177 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a46e1ced-667f-4218-9fe5-4dafa958b55e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a46e1ced-667f-4218-9fe5-4dafa958b55e-plugins-config-volume\") pod \"a46e1ced-667f-4218-9fe5-4dafa958b55e\" (UID: \"a46e1ced-667f-4218-9fe5-4dafa958b55e\") " Oct 30 17:17:51.330378 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.330316 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f10fc70-1bf2-4cca-ace8-47fd5063eada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5190542-kjvsh\" (UID: \"9f10fc70-1bf2-4cca-ace8-47fd5063eada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh" Oct 30 17:17:51.330444 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.330376 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f10fc70-1bf2-4cca-ace8-47fd5063eada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5190542-kjvsh\" (UID: \"9f10fc70-1bf2-4cca-ace8-47fd5063eada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh" Oct 30 17:17:51.330559 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.330537 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a46e1ced-667f-4218-9fe5-4dafa958b55e-plugins-config-volume" pod "a46e1ced-667f-4218-9fe5-4dafa958b55e" (UID: "a46e1ced-667f-4218-9fe5-4dafa958b55e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:51.330613 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.330558 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rpfz\" (UniqueName: \"kubernetes.io/projected/9f10fc70-1bf2-4cca-ace8-47fd5063eada-kube-api-access-9rpfz\") pod \"maas-api-revoke-sub-e5190542-kjvsh\" (UID: \"9f10fc70-1bf2-4cca-ace8-47fd5063eada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh" Oct 30 17:17:51.330751 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.330733 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a46e1ced-667f-4218-9fe5-4dafa958b55e-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:51.332581 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.332561 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a46e1ced-667f-4218-9fe5-4dafa958b55e-kube-api-access-blngh" pod "a46e1ced-667f-4218-9fe5-4dafa958b55e" (UID: "a46e1ced-667f-4218-9fe5-4dafa958b55e"). InnerVolumeSpecName "kube-api-access-blngh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:51.431135 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.431100 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b428320-6e0b-463f-be2e-74ec2cfda977-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b428320-6e0b-463f-be2e-74ec2cfda977-plugins-config-volume\") pod \"5b428320-6e0b-463f-be2e-74ec2cfda977\" (UID: \"5b428320-6e0b-463f-be2e-74ec2cfda977\") " Oct 30 17:17:51.431335 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.431156 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b428320-6e0b-463f-be2e-74ec2cfda977-kube-api-access-h8m57\" (UniqueName: \"kubernetes.io/projected/5b428320-6e0b-463f-be2e-74ec2cfda977-kube-api-access-h8m57\") pod \"5b428320-6e0b-463f-be2e-74ec2cfda977\" (UID: \"5b428320-6e0b-463f-be2e-74ec2cfda977\") " Oct 30 17:17:51.431426 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.431403 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rpfz\" (UniqueName: \"kubernetes.io/projected/9f10fc70-1bf2-4cca-ace8-47fd5063eada-kube-api-access-9rpfz\") pod \"maas-api-revoke-sub-e5190542-kjvsh\" (UID: \"9f10fc70-1bf2-4cca-ace8-47fd5063eada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh" Oct 30 17:17:51.431498 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.431459 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f10fc70-1bf2-4cca-ace8-47fd5063eada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5190542-kjvsh\" (UID: \"9f10fc70-1bf2-4cca-ace8-47fd5063eada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh" Oct 30 17:17:51.431498 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.431458 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b428320-6e0b-463f-be2e-74ec2cfda977-plugins-config-volume" pod "5b428320-6e0b-463f-be2e-74ec2cfda977" (UID: "5b428320-6e0b-463f-be2e-74ec2cfda977"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:51.431625 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.431595 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f10fc70-1bf2-4cca-ace8-47fd5063eada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5190542-kjvsh\" (UID: \"9f10fc70-1bf2-4cca-ace8-47fd5063eada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh" Oct 30 17:17:51.431877 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.431853 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b428320-6e0b-463f-be2e-74ec2cfda977-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:51.431877 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.431878 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blngh\" (UniqueName: \"kubernetes.io/projected/a46e1ced-667f-4218-9fe5-4dafa958b55e-kube-api-access-blngh\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:51.432563 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.432538 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f10fc70-1bf2-4cca-ace8-47fd5063eada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5190542-kjvsh\" (UID: \"9f10fc70-1bf2-4cca-ace8-47fd5063eada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh" Oct 30 17:17:51.433623 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.433593 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b428320-6e0b-463f-be2e-74ec2cfda977-kube-api-access-h8m57" pod "5b428320-6e0b-463f-be2e-74ec2cfda977" (UID: "5b428320-6e0b-463f-be2e-74ec2cfda977"). InnerVolumeSpecName "kube-api-access-h8m57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:51.433859 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.433843 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f10fc70-1bf2-4cca-ace8-47fd5063eada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5190542-kjvsh\" (UID: \"9f10fc70-1bf2-4cca-ace8-47fd5063eada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh" Oct 30 17:17:51.456524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.456501 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rpfz\" (UniqueName: \"kubernetes.io/projected/9f10fc70-1bf2-4cca-ace8-47fd5063eada-kube-api-access-9rpfz\") pod \"maas-api-revoke-sub-e5190542-kjvsh\" (UID: \"9f10fc70-1bf2-4cca-ace8-47fd5063eada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh" Oct 30 17:17:51.532976 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.532898 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8m57\" (UniqueName: \"kubernetes.io/projected/5b428320-6e0b-463f-be2e-74ec2cfda977-kube-api-access-h8m57\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:51.585111 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.585058 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh" Oct 30 17:17:51.791693 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.791607 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:51.906388 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.906356 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh"] Oct 30 17:17:51.910101 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:51.910069 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:52.065355 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.065269 6202 generic.go:356] "Generic (PLEG): container finished" podID="5b428320-6e0b-463f-be2e-74ec2cfda977" containerID="ad737be37cbc8782fdb8841ac5d2ecb9a056d237074637a76214be8e40184441" exitCode=0 Oct 30 17:17:52.065355 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.065327 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" Oct 30 17:17:52.065584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.065365 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" event={"ID":"5b428320-6e0b-463f-be2e-74ec2cfda977","Type":"ContainerDied","Data":"ad737be37cbc8782fdb8841ac5d2ecb9a056d237074637a76214be8e40184441"} Oct 30 17:17:52.065584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.065425 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj" event={"ID":"5b428320-6e0b-463f-be2e-74ec2cfda977","Type":"ContainerDied","Data":"f3a8c366e3c89e827924543e10831945073e882d0978b9659d47a5f3b7cc154e"} Oct 30 17:17:52.065584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.065455 6202 scope.go:122] "RemoveContainer" containerID="ad737be37cbc8782fdb8841ac5d2ecb9a056d237074637a76214be8e40184441" Oct 30 17:17:52.066994 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.066971 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh" event={"ID":"9f10fc70-1bf2-4cca-ace8-47fd5063eada","Type":"ContainerStarted","Data":"eeb9a188e3ce65ddd85d9b5db74c4c3002cf314a629d29d15b23c6c52c53b289"} Oct 30 17:17:52.068483 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.068462 6202 generic.go:356] "Generic (PLEG): container finished" podID="a46e1ced-667f-4218-9fe5-4dafa958b55e" containerID="c28e76120e6e4732e94b57d46b9af757d11697d2663091c37de9ea1bf377fae5" exitCode=0 Oct 30 17:17:52.068582 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.068497 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" event={"ID":"a46e1ced-667f-4218-9fe5-4dafa958b55e","Type":"ContainerDied","Data":"c28e76120e6e4732e94b57d46b9af757d11697d2663091c37de9ea1bf377fae5"} Oct 30 17:17:52.068582 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.068536 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" event={"ID":"a46e1ced-667f-4218-9fe5-4dafa958b55e","Type":"ContainerDied","Data":"ba67980ca4502ecc1b4ed4e7755b2c0c144e0e8e62330a4e7b040cc2d79d2603"} Oct 30 17:17:52.068582 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.068534 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw" Oct 30 17:17:52.092483 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.087097 6202 scope.go:122] "RemoveContainer" containerID="ad737be37cbc8782fdb8841ac5d2ecb9a056d237074637a76214be8e40184441" Oct 30 17:17:52.093250 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:17:52.093181 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad737be37cbc8782fdb8841ac5d2ecb9a056d237074637a76214be8e40184441\": container with ID starting with ad737be37cbc8782fdb8841ac5d2ecb9a056d237074637a76214be8e40184441 not found: ID does not exist" containerID="ad737be37cbc8782fdb8841ac5d2ecb9a056d237074637a76214be8e40184441" Oct 30 17:17:52.093680 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.093277 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad737be37cbc8782fdb8841ac5d2ecb9a056d237074637a76214be8e40184441"} err="failed to get container status \"ad737be37cbc8782fdb8841ac5d2ecb9a056d237074637a76214be8e40184441\": rpc error: code = NotFound desc = could not find container \"ad737be37cbc8782fdb8841ac5d2ecb9a056d237074637a76214be8e40184441\": container with ID starting with ad737be37cbc8782fdb8841ac5d2ecb9a056d237074637a76214be8e40184441 not found: ID does not exist" Oct 30 17:17:52.093680 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.093307 6202 scope.go:122] "RemoveContainer" containerID="c28e76120e6e4732e94b57d46b9af757d11697d2663091c37de9ea1bf377fae5" Oct 30 17:17:52.108675 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.108644 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb"] Oct 30 17:17:52.109109 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.109078 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" podUID="ac56359c-6664-4198-88ef-2208c355cd11" containerName="istio-proxy" containerID="cri-o://9254c8218a3a8994970b2d2cdd37035c6e03cdb3863de34b26ad6a3f1b377ce6" gracePeriod=30 Oct 30 17:17:52.116064 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.116034 6202 scope.go:122] "RemoveContainer" containerID="c28e76120e6e4732e94b57d46b9af757d11697d2663091c37de9ea1bf377fae5" Oct 30 17:17:52.116403 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:17:52.116373 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28e76120e6e4732e94b57d46b9af757d11697d2663091c37de9ea1bf377fae5\": container with ID starting with c28e76120e6e4732e94b57d46b9af757d11697d2663091c37de9ea1bf377fae5 not found: ID does not exist" containerID="c28e76120e6e4732e94b57d46b9af757d11697d2663091c37de9ea1bf377fae5" Oct 30 17:17:52.116503 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.116415 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28e76120e6e4732e94b57d46b9af757d11697d2663091c37de9ea1bf377fae5"} err="failed to get container status \"c28e76120e6e4732e94b57d46b9af757d11697d2663091c37de9ea1bf377fae5\": rpc error: code = NotFound desc = could not find container \"c28e76120e6e4732e94b57d46b9af757d11697d2663091c37de9ea1bf377fae5\": container with ID starting with c28e76120e6e4732e94b57d46b9af757d11697d2663091c37de9ea1bf377fae5 not found: ID does not exist" Oct 30 17:17:52.139584 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.139548 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj"] Oct 30 17:17:52.141357 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.141326 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-750dcb-5899ccf779-ct8tj"] Oct 30 17:17:52.174915 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.173144 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:52.176564 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.176537 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw"] Oct 30 17:17:52.180603 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.180575 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-750dcb-5cc94687cb-z7vpw"] Oct 30 17:17:52.410045 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.410009 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b428320-6e0b-463f-be2e-74ec2cfda977" path="/var/lib/kubelet/pods/5b428320-6e0b-463f-be2e-74ec2cfda977/volumes" Oct 30 17:17:52.410428 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:52.410382 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a46e1ced-667f-4218-9fe5-4dafa958b55e" path="/var/lib/kubelet/pods/a46e1ced-667f-4218-9fe5-4dafa958b55e/volumes" Oct 30 17:17:53.077349 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:53.077306 6202 generic.go:356] "Generic (PLEG): container finished" podID="9f10fc70-1bf2-4cca-ace8-47fd5063eada" containerID="58b1b0a0eda9cac728d84716a448d200055d01c3e5689bc7baef18b265b4549f" exitCode=0 Oct 30 17:17:53.077545 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:53.077367 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh" event={"ID":"9f10fc70-1bf2-4cca-ace8-47fd5063eada","Type":"ContainerDied","Data":"58b1b0a0eda9cac728d84716a448d200055d01c3e5689bc7baef18b265b4549f"} Oct 30 17:17:54.229943 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:54.229916 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh" Oct 30 17:17:54.359024 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:54.358940 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f10fc70-1bf2-4cca-ace8-47fd5063eada-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f10fc70-1bf2-4cca-ace8-47fd5063eada-maas-api-xks-ca\") pod \"9f10fc70-1bf2-4cca-ace8-47fd5063eada\" (UID: \"9f10fc70-1bf2-4cca-ace8-47fd5063eada\") " Oct 30 17:17:54.359193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:54.359118 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f10fc70-1bf2-4cca-ace8-47fd5063eada-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f10fc70-1bf2-4cca-ace8-47fd5063eada-maas-api-openshift-ca\") pod \"9f10fc70-1bf2-4cca-ace8-47fd5063eada\" (UID: \"9f10fc70-1bf2-4cca-ace8-47fd5063eada\") " Oct 30 17:17:54.359193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:54.359178 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f10fc70-1bf2-4cca-ace8-47fd5063eada-kube-api-access-9rpfz\" (UniqueName: \"kubernetes.io/projected/9f10fc70-1bf2-4cca-ace8-47fd5063eada-kube-api-access-9rpfz\") pod \"9f10fc70-1bf2-4cca-ace8-47fd5063eada\" (UID: \"9f10fc70-1bf2-4cca-ace8-47fd5063eada\") " Oct 30 17:17:54.359488 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:54.359464 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f10fc70-1bf2-4cca-ace8-47fd5063eada-maas-api-openshift-ca" pod "9f10fc70-1bf2-4cca-ace8-47fd5063eada" (UID: "9f10fc70-1bf2-4cca-ace8-47fd5063eada"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:54.361264 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:54.361228 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f10fc70-1bf2-4cca-ace8-47fd5063eada-maas-api-xks-ca" pod "9f10fc70-1bf2-4cca-ace8-47fd5063eada" (UID: "9f10fc70-1bf2-4cca-ace8-47fd5063eada"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:54.361375 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:54.361279 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f10fc70-1bf2-4cca-ace8-47fd5063eada-kube-api-access-9rpfz" pod "9f10fc70-1bf2-4cca-ace8-47fd5063eada" (UID: "9f10fc70-1bf2-4cca-ace8-47fd5063eada"). InnerVolumeSpecName "kube-api-access-9rpfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:54.460736 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:54.460702 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f10fc70-1bf2-4cca-ace8-47fd5063eada-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:54.460736 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:54.460732 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rpfz\" (UniqueName: \"kubernetes.io/projected/9f10fc70-1bf2-4cca-ace8-47fd5063eada-kube-api-access-9rpfz\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:54.460736 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:54.460741 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f10fc70-1bf2-4cca-ace8-47fd5063eada-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:55.095423 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:55.095390 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh" Oct 30 17:17:55.095423 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:55.095412 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh" event={"ID":"9f10fc70-1bf2-4cca-ace8-47fd5063eada","Type":"ContainerDied","Data":"eeb9a188e3ce65ddd85d9b5db74c4c3002cf314a629d29d15b23c6c52c53b289"} Oct 30 17:17:55.095663 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:55.095463 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeb9a188e3ce65ddd85d9b5db74c4c3002cf314a629d29d15b23c6c52c53b289" Oct 30 17:17:55.163513 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:55.163474 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" podUID="19c858ea-1556-446d-ac02-5c3b2cc6f709" containerName="payload-processing" containerID="cri-o://657c26198569ee21c59d233188a82629bed1150f4e579a0ca1bb0f43d2c75ef8" gracePeriod=25 Oct 30 17:17:55.194199 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:55.194137 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" podUID="dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5" containerName="payload-pre-processing" containerID="cri-o://340e823abc09bb194758ad312e8dd977383f446bbfcf391d454d4135b024443e" gracePeriod=25 Oct 30 17:17:56.951118 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:56.951078 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:57.065359 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.065319 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:57.382819 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.382783 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 17:17:57.490984 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.490902 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-workload-certs\") pod \"ac56359c-6664-4198-88ef-2208c355cd11\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " Oct 30 17:17:57.490984 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.490944 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-workload-socket\") pod \"ac56359c-6664-4198-88ef-2208c355cd11\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " Oct 30 17:17:57.490984 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.490978 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-istio-envoy\") pod \"ac56359c-6664-4198-88ef-2208c355cd11\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " Oct 30 17:17:57.491275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.491106 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac56359c-6664-4198-88ef-2208c355cd11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac56359c-6664-4198-88ef-2208c355cd11-istiod-ca-cert\") pod \"ac56359c-6664-4198-88ef-2208c355cd11\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " Oct 30 17:17:57.491275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.491144 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac56359c-6664-4198-88ef-2208c355cd11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac56359c-6664-4198-88ef-2208c355cd11-istio-podinfo\") pod \"ac56359c-6664-4198-88ef-2208c355cd11\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " Oct 30 17:17:57.491275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.491181 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac56359c-6664-4198-88ef-2208c355cd11-istio-token\" (UniqueName: \"kubernetes.io/projected/ac56359c-6664-4198-88ef-2208c355cd11-istio-token\") pod \"ac56359c-6664-4198-88ef-2208c355cd11\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " Oct 30 17:17:57.491275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.491218 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-workload-certs" pod "ac56359c-6664-4198-88ef-2208c355cd11" (UID: "ac56359c-6664-4198-88ef-2208c355cd11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:57.491275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.491243 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac56359c-6664-4198-88ef-2208c355cd11-kube-api-access-swsfq\" (UniqueName: \"kubernetes.io/projected/ac56359c-6664-4198-88ef-2208c355cd11-kube-api-access-swsfq\") pod \"ac56359c-6664-4198-88ef-2208c355cd11\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " Oct 30 17:17:57.491275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.491251 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-workload-socket" pod "ac56359c-6664-4198-88ef-2208c355cd11" (UID: "ac56359c-6664-4198-88ef-2208c355cd11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:57.491626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.491281 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-credential-socket\") pod \"ac56359c-6664-4198-88ef-2208c355cd11\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " Oct 30 17:17:57.491626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.491330 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-istio-data\") pod \"ac56359c-6664-4198-88ef-2208c355cd11\" (UID: \"ac56359c-6664-4198-88ef-2208c355cd11\") " Oct 30 17:17:57.491626 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.491537 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac56359c-6664-4198-88ef-2208c355cd11-istiod-ca-cert" pod "ac56359c-6664-4198-88ef-2208c355cd11" (UID: "ac56359c-6664-4198-88ef-2208c355cd11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:57.491814 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.491729 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-credential-socket" pod "ac56359c-6664-4198-88ef-2208c355cd11" (UID: "ac56359c-6664-4198-88ef-2208c355cd11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:57.492144 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.492121 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac56359c-6664-4198-88ef-2208c355cd11-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:57.492371 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.492331 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:57.492371 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.492360 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:57.492562 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.492376 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:57.492562 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.492134 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-istio-data" pod "ac56359c-6664-4198-88ef-2208c355cd11" (UID: "ac56359c-6664-4198-88ef-2208c355cd11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:57.493699 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.493675 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac56359c-6664-4198-88ef-2208c355cd11-istio-podinfo" pod "ac56359c-6664-4198-88ef-2208c355cd11" (UID: "ac56359c-6664-4198-88ef-2208c355cd11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:17:57.493809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.493715 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac56359c-6664-4198-88ef-2208c355cd11-istio-token" pod "ac56359c-6664-4198-88ef-2208c355cd11" (UID: "ac56359c-6664-4198-88ef-2208c355cd11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:57.493898 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.493881 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-istio-envoy" pod "ac56359c-6664-4198-88ef-2208c355cd11" (UID: "ac56359c-6664-4198-88ef-2208c355cd11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:57.493969 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.493946 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac56359c-6664-4198-88ef-2208c355cd11-kube-api-access-swsfq" pod "ac56359c-6664-4198-88ef-2208c355cd11" (UID: "ac56359c-6664-4198-88ef-2208c355cd11"). InnerVolumeSpecName "kube-api-access-swsfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:57.593281 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.593249 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:57.593281 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.593277 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac56359c-6664-4198-88ef-2208c355cd11-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:57.593281 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.593286 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac56359c-6664-4198-88ef-2208c355cd11-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:57.593532 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.593295 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swsfq\" (UniqueName: \"kubernetes.io/projected/ac56359c-6664-4198-88ef-2208c355cd11-kube-api-access-swsfq\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:57.593532 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:57.593304 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac56359c-6664-4198-88ef-2208c355cd11-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:17:58.116320 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:58.116288 6202 generic.go:356] "Generic (PLEG): container finished" podID="ac56359c-6664-4198-88ef-2208c355cd11" containerID="9254c8218a3a8994970b2d2cdd37035c6e03cdb3863de34b26ad6a3f1b377ce6" exitCode=0 Oct 30 17:17:58.116741 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:58.116346 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" Oct 30 17:17:58.116741 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:58.116380 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" event={"ID":"ac56359c-6664-4198-88ef-2208c355cd11","Type":"ContainerDied","Data":"9254c8218a3a8994970b2d2cdd37035c6e03cdb3863de34b26ad6a3f1b377ce6"} Oct 30 17:17:58.116741 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:58.116437 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb" event={"ID":"ac56359c-6664-4198-88ef-2208c355cd11","Type":"ContainerDied","Data":"2bf945ea40fd87e5401d0392f90ad43c07cbe413b4821996528e8bafeaab98da"} Oct 30 17:17:58.116741 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:58.116468 6202 scope.go:122] "RemoveContainer" containerID="9254c8218a3a8994970b2d2cdd37035c6e03cdb3863de34b26ad6a3f1b377ce6" Oct 30 17:17:58.128556 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:58.128538 6202 scope.go:122] "RemoveContainer" containerID="9fc6e07fb0b1ae86bcbee07e00402a998487aec7d191ffb882dd8c5806ac3409" Oct 30 17:17:58.137741 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:58.137711 6202 scope.go:122] "RemoveContainer" containerID="9254c8218a3a8994970b2d2cdd37035c6e03cdb3863de34b26ad6a3f1b377ce6" Oct 30 17:17:58.138086 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:17:58.138065 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9254c8218a3a8994970b2d2cdd37035c6e03cdb3863de34b26ad6a3f1b377ce6\": container with ID starting with 9254c8218a3a8994970b2d2cdd37035c6e03cdb3863de34b26ad6a3f1b377ce6 not found: ID does not exist" containerID="9254c8218a3a8994970b2d2cdd37035c6e03cdb3863de34b26ad6a3f1b377ce6" Oct 30 17:17:58.138152 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:58.138097 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9254c8218a3a8994970b2d2cdd37035c6e03cdb3863de34b26ad6a3f1b377ce6"} err="failed to get container status \"9254c8218a3a8994970b2d2cdd37035c6e03cdb3863de34b26ad6a3f1b377ce6\": rpc error: code = NotFound desc = could not find container \"9254c8218a3a8994970b2d2cdd37035c6e03cdb3863de34b26ad6a3f1b377ce6\": container with ID starting with 9254c8218a3a8994970b2d2cdd37035c6e03cdb3863de34b26ad6a3f1b377ce6 not found: ID does not exist" Oct 30 17:17:58.138152 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:58.138118 6202 scope.go:122] "RemoveContainer" containerID="9fc6e07fb0b1ae86bcbee07e00402a998487aec7d191ffb882dd8c5806ac3409" Oct 30 17:17:58.138377 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:17:58.138361 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc6e07fb0b1ae86bcbee07e00402a998487aec7d191ffb882dd8c5806ac3409\": container with ID starting with 9fc6e07fb0b1ae86bcbee07e00402a998487aec7d191ffb882dd8c5806ac3409 not found: ID does not exist" containerID="9fc6e07fb0b1ae86bcbee07e00402a998487aec7d191ffb882dd8c5806ac3409" Oct 30 17:17:58.138433 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:58.138381 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc6e07fb0b1ae86bcbee07e00402a998487aec7d191ffb882dd8c5806ac3409"} err="failed to get container status \"9fc6e07fb0b1ae86bcbee07e00402a998487aec7d191ffb882dd8c5806ac3409\": rpc error: code = NotFound desc = could not find container \"9fc6e07fb0b1ae86bcbee07e00402a998487aec7d191ffb882dd8c5806ac3409\": container with ID starting with 9fc6e07fb0b1ae86bcbee07e00402a998487aec7d191ffb882dd8c5806ac3409 not found: ID does not exist" Oct 30 17:17:58.190483 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:58.190450 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb"] Oct 30 17:17:58.195498 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:58.195459 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-q6frb"] Oct 30 17:17:58.409380 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:58.409298 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac56359c-6664-4198-88ef-2208c355cd11" path="/var/lib/kubelet/pods/ac56359c-6664-4198-88ef-2208c355cd11/volumes" Oct 30 17:17:58.858275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:58.858239 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:59.589149 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:59.589105 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" containerID="cri-o://cccf30eb422850080fab1fb40248b883b0e865f4c1983cf9c69298e04715e9fe" gracePeriod=45 Oct 30 17:17:59.646205 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:59.646166 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": dial tcp 10.133.0.48:8000: connect: connection refused" Oct 30 17:17:59.843521 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:59.843453 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 17:17:59.918003 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:59.917970 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-dshm\") pod \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " Oct 30 17:17:59.918219 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:59.918032 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-tls-certs\") pod \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " Oct 30 17:17:59.918219 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:59.918055 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-kube-api-access-tnjss\" (UniqueName: \"kubernetes.io/projected/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-kube-api-access-tnjss\") pod \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " Oct 30 17:17:59.918219 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:59.918091 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-model-cache\") pod \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " Oct 30 17:17:59.918219 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:59.918126 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-home\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-home\") pod \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " Oct 30 17:17:59.918219 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:59.918161 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-tmp-dir\") pod \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\" (UID: \"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4\") " Oct 30 17:17:59.918548 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:59.918496 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-tmp-dir" pod "8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" (UID: "8cbb0ec9-d73e-4b74-bce1-15b919acb9b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:59.918619 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:59.918551 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-model-cache" pod "8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" (UID: "8cbb0ec9-d73e-4b74-bce1-15b919acb9b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:59.918619 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:59.918597 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-home" pod "8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" (UID: "8cbb0ec9-d73e-4b74-bce1-15b919acb9b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:59.920841 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:59.920812 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-tls-certs" pod "8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" (UID: "8cbb0ec9-d73e-4b74-bce1-15b919acb9b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:59.921037 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:59.920997 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-kube-api-access-tnjss" pod "8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" (UID: "8cbb0ec9-d73e-4b74-bce1-15b919acb9b4"). InnerVolumeSpecName "kube-api-access-tnjss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:59.921133 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:17:59.921071 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-dshm" pod "8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" (UID: "8cbb0ec9-d73e-4b74-bce1-15b919acb9b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:18:00.019384 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:00.019346 6202 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-dshm\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:00.019384 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:00.019380 6202 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-tls-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:00.019643 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:00.019395 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnjss\" (UniqueName: \"kubernetes.io/projected/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-kube-api-access-tnjss\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:00.019643 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:00.019407 6202 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-model-cache\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:00.019643 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:00.019419 6202 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-home\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:00.019643 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:00.019429 6202 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4-tmp-dir\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:00.133195 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:00.133099 6202 generic.go:356] "Generic (PLEG): container finished" podID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerID="cccf30eb422850080fab1fb40248b883b0e865f4c1983cf9c69298e04715e9fe" exitCode=0 Oct 30 17:18:00.133195 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:00.133145 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" Oct 30 17:18:00.133434 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:00.133188 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" event={"ID":"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4","Type":"ContainerDied","Data":"cccf30eb422850080fab1fb40248b883b0e865f4c1983cf9c69298e04715e9fe"} Oct 30 17:18:00.133434 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:00.133238 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9" event={"ID":"8cbb0ec9-d73e-4b74-bce1-15b919acb9b4","Type":"ContainerDied","Data":"05ddf959af12446be02bde15d96b8d9d0c1fe74171b03b2ac031456f837f893e"} Oct 30 17:18:00.133434 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:00.133260 6202 scope.go:122] "RemoveContainer" containerID="cccf30eb422850080fab1fb40248b883b0e865f4c1983cf9c69298e04715e9fe" Oct 30 17:18:00.143490 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:00.143469 6202 scope.go:122] "RemoveContainer" containerID="cccf30eb422850080fab1fb40248b883b0e865f4c1983cf9c69298e04715e9fe" Oct 30 17:18:00.143747 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:18:00.143728 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cccf30eb422850080fab1fb40248b883b0e865f4c1983cf9c69298e04715e9fe\": container with ID starting with cccf30eb422850080fab1fb40248b883b0e865f4c1983cf9c69298e04715e9fe not found: ID does not exist" containerID="cccf30eb422850080fab1fb40248b883b0e865f4c1983cf9c69298e04715e9fe" Oct 30 17:18:00.143843 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:00.143753 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cccf30eb422850080fab1fb40248b883b0e865f4c1983cf9c69298e04715e9fe"} err="failed to get container status \"cccf30eb422850080fab1fb40248b883b0e865f4c1983cf9c69298e04715e9fe\": rpc error: code = NotFound desc = could not find container \"cccf30eb422850080fab1fb40248b883b0e865f4c1983cf9c69298e04715e9fe\": container with ID starting with cccf30eb422850080fab1fb40248b883b0e865f4c1983cf9c69298e04715e9fe not found: ID does not exist" Oct 30 17:18:00.190250 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:00.190216 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9"] Oct 30 17:18:00.194583 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:00.194559 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/premium-w0-3e2f96-kserve-7df5689c5f-p69q9"] Oct 30 17:18:00.408503 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:00.408424 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" path="/var/lib/kubelet/pods/8cbb0ec9-d73e-4b74-bce1-15b919acb9b4/volumes" Oct 30 17:18:02.460967 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:02.460928 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" podUID="dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:18:03.479335 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:03.479291 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" podUID="19c858ea-1556-446d-ac02-5c3b2cc6f709" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:18:04.613791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:04.613717 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:06.438650 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:06.438610 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:06.870385 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:06.870351 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:06.987014 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:06.986980 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:07.542480 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:07.542442 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:12.461253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:12.461213 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" podUID="dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:18:13.480073 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:13.480029 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" podUID="19c858ea-1556-446d-ac02-5c3b2cc6f709" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:18:14.070494 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:18:14.070454 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:18:14.113094 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:14.113060 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:14.228884 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:14.228848 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:14.438260 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:14.438218 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8"] Oct 30 17:18:14.438494 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:14.438458 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" podUID="dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" containerName="istio-proxy" containerID="cri-o://62766045165b4eaf44f22b983bc3d8b585d76a3109bf8bb6d94b4d3187ee0252" gracePeriod=30 Oct 30 17:18:14.482081 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:14.482041 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:15.438411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:15.438383 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" Oct 30 17:18:15.466744 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:15.466717 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" Oct 30 17:18:15.467923 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:15.467897 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c858ea-1556-446d-ac02-5c3b2cc6f709-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c858ea-1556-446d-ac02-5c3b2cc6f709-plugins-config-volume\") pod \"19c858ea-1556-446d-ac02-5c3b2cc6f709\" (UID: \"19c858ea-1556-446d-ac02-5c3b2cc6f709\") " Oct 30 17:18:15.468070 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:15.467989 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5-plugins-config-volume\") pod \"dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5\" (UID: \"dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5\") " Oct 30 17:18:15.468158 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:15.468115 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5-kube-api-access-4d9wz\" (UniqueName: \"kubernetes.io/projected/dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5-kube-api-access-4d9wz\") pod \"dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5\" (UID: \"dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5\") " Oct 30 17:18:15.468231 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:15.468184 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c858ea-1556-446d-ac02-5c3b2cc6f709-kube-api-access-jfwrb\" (UniqueName: \"kubernetes.io/projected/19c858ea-1556-446d-ac02-5c3b2cc6f709-kube-api-access-jfwrb\") pod \"19c858ea-1556-446d-ac02-5c3b2cc6f709\" (UID: \"19c858ea-1556-446d-ac02-5c3b2cc6f709\") " Oct 30 17:18:15.468351 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:15.468324 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c858ea-1556-446d-ac02-5c3b2cc6f709-plugins-config-volume" pod "19c858ea-1556-446d-ac02-5c3b2cc6f709" (UID: "19c858ea-1556-446d-ac02-5c3b2cc6f709"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:18:15.468485 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:15.468460 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5-plugins-config-volume" pod "dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5" (UID: "dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:18:15.468661 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:15.468633 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:15.468661 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:15.468661 6202 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c858ea-1556-446d-ac02-5c3b2cc6f709-plugins-config-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:15.470829 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:15.470759 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5-kube-api-access-4d9wz" pod "dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5" (UID: "dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5"). InnerVolumeSpecName "kube-api-access-4d9wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:18:15.470946 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:15.470924 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c858ea-1556-446d-ac02-5c3b2cc6f709-kube-api-access-jfwrb" pod "19c858ea-1556-446d-ac02-5c3b2cc6f709" (UID: "19c858ea-1556-446d-ac02-5c3b2cc6f709"). InnerVolumeSpecName "kube-api-access-jfwrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:18:15.569642 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:15.569610 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d9wz\" (UniqueName: \"kubernetes.io/projected/dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5-kube-api-access-4d9wz\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:15.569642 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:15.569641 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfwrb\" (UniqueName: \"kubernetes.io/projected/19c858ea-1556-446d-ac02-5c3b2cc6f709-kube-api-access-jfwrb\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:16.241380 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.241345 6202 generic.go:356] "Generic (PLEG): container finished" podID="dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5" containerID="340e823abc09bb194758ad312e8dd977383f446bbfcf391d454d4135b024443e" exitCode=0 Oct 30 17:18:16.241597 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.241393 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" Oct 30 17:18:16.241597 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.241435 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" event={"ID":"dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5","Type":"ContainerDied","Data":"340e823abc09bb194758ad312e8dd977383f446bbfcf391d454d4135b024443e"} Oct 30 17:18:16.241597 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.241496 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf" event={"ID":"dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5","Type":"ContainerDied","Data":"d58b963c9c8a62581fc991f1f232a3e056370767eaa80797fc7281559901b2c5"} Oct 30 17:18:16.241597 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.241530 6202 scope.go:122] "RemoveContainer" containerID="340e823abc09bb194758ad312e8dd977383f446bbfcf391d454d4135b024443e" Oct 30 17:18:16.243380 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.243339 6202 generic.go:356] "Generic (PLEG): container finished" podID="19c858ea-1556-446d-ac02-5c3b2cc6f709" containerID="657c26198569ee21c59d233188a82629bed1150f4e579a0ca1bb0f43d2c75ef8" exitCode=0 Oct 30 17:18:16.243532 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.243382 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" Oct 30 17:18:16.243532 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.243420 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" event={"ID":"19c858ea-1556-446d-ac02-5c3b2cc6f709","Type":"ContainerDied","Data":"657c26198569ee21c59d233188a82629bed1150f4e579a0ca1bb0f43d2c75ef8"} Oct 30 17:18:16.243532 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.243485 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth" event={"ID":"19c858ea-1556-446d-ac02-5c3b2cc6f709","Type":"ContainerDied","Data":"6d450e48f0faf2c1e62b66751d0fbcfff168c80ceead8a96e1cc2d4abc86379d"} Oct 30 17:18:16.253171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.253075 6202 scope.go:122] "RemoveContainer" containerID="340e823abc09bb194758ad312e8dd977383f446bbfcf391d454d4135b024443e" Oct 30 17:18:16.253480 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:18:16.253454 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340e823abc09bb194758ad312e8dd977383f446bbfcf391d454d4135b024443e\": container with ID starting with 340e823abc09bb194758ad312e8dd977383f446bbfcf391d454d4135b024443e not found: ID does not exist" containerID="340e823abc09bb194758ad312e8dd977383f446bbfcf391d454d4135b024443e" Oct 30 17:18:16.253542 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.253493 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340e823abc09bb194758ad312e8dd977383f446bbfcf391d454d4135b024443e"} err="failed to get container status \"340e823abc09bb194758ad312e8dd977383f446bbfcf391d454d4135b024443e\": rpc error: code = NotFound desc = could not find container \"340e823abc09bb194758ad312e8dd977383f446bbfcf391d454d4135b024443e\": container with ID starting with 340e823abc09bb194758ad312e8dd977383f446bbfcf391d454d4135b024443e not found: ID does not exist" Oct 30 17:18:16.253542 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.253522 6202 scope.go:122] "RemoveContainer" containerID="657c26198569ee21c59d233188a82629bed1150f4e579a0ca1bb0f43d2c75ef8" Oct 30 17:18:16.263841 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.263814 6202 scope.go:122] "RemoveContainer" containerID="657c26198569ee21c59d233188a82629bed1150f4e579a0ca1bb0f43d2c75ef8" Oct 30 17:18:16.264228 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:18:16.264205 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657c26198569ee21c59d233188a82629bed1150f4e579a0ca1bb0f43d2c75ef8\": container with ID starting with 657c26198569ee21c59d233188a82629bed1150f4e579a0ca1bb0f43d2c75ef8 not found: ID does not exist" containerID="657c26198569ee21c59d233188a82629bed1150f4e579a0ca1bb0f43d2c75ef8" Oct 30 17:18:16.264305 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.264237 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657c26198569ee21c59d233188a82629bed1150f4e579a0ca1bb0f43d2c75ef8"} err="failed to get container status \"657c26198569ee21c59d233188a82629bed1150f4e579a0ca1bb0f43d2c75ef8\": rpc error: code = NotFound desc = could not find container \"657c26198569ee21c59d233188a82629bed1150f4e579a0ca1bb0f43d2c75ef8\": container with ID starting with 657c26198569ee21c59d233188a82629bed1150f4e579a0ca1bb0f43d2c75ef8 not found: ID does not exist" Oct 30 17:18:16.296565 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.296525 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf"] Oct 30 17:18:16.309388 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.309355 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3e2f96-76c8465d7-54qdf"] Oct 30 17:18:16.337298 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.337258 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth"] Oct 30 17:18:16.340989 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.340957 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3e2f96-7bf8989788-4tsth"] Oct 30 17:18:16.409586 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.409546 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c858ea-1556-446d-ac02-5c3b2cc6f709" path="/var/lib/kubelet/pods/19c858ea-1556-446d-ac02-5c3b2cc6f709/volumes" Oct 30 17:18:16.410023 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.410003 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5" path="/var/lib/kubelet/pods/dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5/volumes" Oct 30 17:18:16.552712 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.552627 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:16.665328 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:16.665291 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:19.692786 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.692738 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 17:18:19.714207 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.714154 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-podinfo\") pod \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " Oct 30 17:18:19.714207 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.714195 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-kube-api-access-8lcbr\" (UniqueName: \"kubernetes.io/projected/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-kube-api-access-8lcbr\") pod \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " Oct 30 17:18:19.714368 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.714224 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-envoy\") pod \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " Oct 30 17:18:19.714368 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.714290 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-credential-socket\") pod \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " Oct 30 17:18:19.714368 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.714310 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-workload-certs\") pod \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " Oct 30 17:18:19.714368 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.714326 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istiod-ca-cert\") pod \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " Oct 30 17:18:19.714368 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.714354 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-data\") pod \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " Oct 30 17:18:19.714621 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.714406 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-token\" (UniqueName: \"kubernetes.io/projected/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-token\") pod \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " Oct 30 17:18:19.714621 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.714447 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-workload-socket\") pod \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\" (UID: \"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a\") " Oct 30 17:18:19.714748 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.714657 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-credential-socket" pod "dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" (UID: "dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:18:19.714748 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.714691 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-workload-certs" pod "dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" (UID: "dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:18:19.714894 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.714830 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istiod-ca-cert" pod "dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" (UID: "dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:18:19.714894 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.714832 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-data" pod "dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" (UID: "dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:18:19.714998 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.714930 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-workload-socket" pod "dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" (UID: "dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:18:19.715133 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.714839 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:19.715230 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.715148 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:19.716541 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.716517 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-kube-api-access-8lcbr" pod "dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" (UID: "dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a"). InnerVolumeSpecName "kube-api-access-8lcbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:18:19.716641 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.716520 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-podinfo" pod "dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" (UID: "dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:18:19.716702 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.716667 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-token" pod "dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" (UID: "dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:18:19.717053 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.717038 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-envoy" pod "dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" (UID: "dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:18:19.816189 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.816154 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:19.816189 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.816182 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lcbr\" (UniqueName: \"kubernetes.io/projected/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-kube-api-access-8lcbr\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:19.816189 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.816192 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:19.816434 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.816201 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:19.816434 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.816208 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:19.816434 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.816216 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:19.816434 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:19.816223 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:18:20.271509 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:20.271472 6202 generic.go:356] "Generic (PLEG): container finished" podID="dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" containerID="62766045165b4eaf44f22b983bc3d8b585d76a3109bf8bb6d94b4d3187ee0252" exitCode=0 Oct 30 17:18:20.271751 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:20.271526 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" Oct 30 17:18:20.271751 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:20.271553 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" event={"ID":"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a","Type":"ContainerDied","Data":"62766045165b4eaf44f22b983bc3d8b585d76a3109bf8bb6d94b4d3187ee0252"} Oct 30 17:18:20.271751 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:20.271610 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8" event={"ID":"dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a","Type":"ContainerDied","Data":"7e55c2a1d44d5631d87d7e10b0bcb7a97d75fba270d5a50b9e72f8f5fa3c9c7c"} Oct 30 17:18:20.271751 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:20.271632 6202 scope.go:122] "RemoveContainer" containerID="62766045165b4eaf44f22b983bc3d8b585d76a3109bf8bb6d94b4d3187ee0252" Oct 30 17:18:20.282327 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:20.282304 6202 scope.go:122] "RemoveContainer" containerID="529709dde15bdd2c696871fd51f5c2e62bf14f13c662a87fcc8b61f2db84a52e" Oct 30 17:18:20.291145 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:20.291122 6202 scope.go:122] "RemoveContainer" containerID="62766045165b4eaf44f22b983bc3d8b585d76a3109bf8bb6d94b4d3187ee0252" Oct 30 17:18:20.291417 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:18:20.291400 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62766045165b4eaf44f22b983bc3d8b585d76a3109bf8bb6d94b4d3187ee0252\": container with ID starting with 62766045165b4eaf44f22b983bc3d8b585d76a3109bf8bb6d94b4d3187ee0252 not found: ID does not exist" containerID="62766045165b4eaf44f22b983bc3d8b585d76a3109bf8bb6d94b4d3187ee0252" Oct 30 17:18:20.291469 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:20.291427 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62766045165b4eaf44f22b983bc3d8b585d76a3109bf8bb6d94b4d3187ee0252"} err="failed to get container status \"62766045165b4eaf44f22b983bc3d8b585d76a3109bf8bb6d94b4d3187ee0252\": rpc error: code = NotFound desc = could not find container \"62766045165b4eaf44f22b983bc3d8b585d76a3109bf8bb6d94b4d3187ee0252\": container with ID starting with 62766045165b4eaf44f22b983bc3d8b585d76a3109bf8bb6d94b4d3187ee0252 not found: ID does not exist" Oct 30 17:18:20.291469 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:20.291446 6202 scope.go:122] "RemoveContainer" containerID="529709dde15bdd2c696871fd51f5c2e62bf14f13c662a87fcc8b61f2db84a52e" Oct 30 17:18:20.291713 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:18:20.291692 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529709dde15bdd2c696871fd51f5c2e62bf14f13c662a87fcc8b61f2db84a52e\": container with ID starting with 529709dde15bdd2c696871fd51f5c2e62bf14f13c662a87fcc8b61f2db84a52e not found: ID does not exist" containerID="529709dde15bdd2c696871fd51f5c2e62bf14f13c662a87fcc8b61f2db84a52e" Oct 30 17:18:20.291788 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:20.291719 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529709dde15bdd2c696871fd51f5c2e62bf14f13c662a87fcc8b61f2db84a52e"} err="failed to get container status \"529709dde15bdd2c696871fd51f5c2e62bf14f13c662a87fcc8b61f2db84a52e\": rpc error: code = NotFound desc = could not find container \"529709dde15bdd2c696871fd51f5c2e62bf14f13c662a87fcc8b61f2db84a52e\": container with ID starting with 529709dde15bdd2c696871fd51f5c2e62bf14f13c662a87fcc8b61f2db84a52e not found: ID does not exist" Oct 30 17:18:20.326676 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:20.326651 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8"] Oct 30 17:18:20.339790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:20.339743 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-cs9m8"] Oct 30 17:18:20.408498 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:20.408465 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" path="/var/lib/kubelet/pods/dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a/volumes" Oct 30 17:18:21.115170 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:21.115135 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:21.296161 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:21.296124 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:21.489619 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:21.489514 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:21.718560 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:21.718530 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:22.819339 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:22.819296 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:28.773343 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:28.773302 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:34.770003 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:34.769961 6202 scope.go:122] "RemoveContainer" containerID="1f9e893d4bd5d69916944522b7d992d1f2d9646fe783639145fa0222defd40bf" Oct 30 17:18:34.779376 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:34.779352 6202 scope.go:122] "RemoveContainer" containerID="2d4b5495fc88679cc7b6683ccb7a7aaecda9b31cfe91af22a84b6e0806173a9b" Oct 30 17:18:37.028836 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:37.028735 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8"] Oct 30 17:18:37.041959 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:37.041930 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63fd6c00-7mfd8"] Oct 30 17:18:38.408282 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:38.408240 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f03c8fbb-68d9-49ec-8d1f-2e7affec093d" path="/var/lib/kubelet/pods/f03c8fbb-68d9-49ec-8d1f-2e7affec093d/volumes" Oct 30 17:18:43.161538 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:43.161501 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:44.029433 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:44.029391 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp"] Oct 30 17:18:44.043038 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:44.043003 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e38ffa-flbnp"] Oct 30 17:18:44.044344 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:44.044310 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7"] Oct 30 17:18:44.045524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:44.045502 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44d721b6-h7dw7"] Oct 30 17:18:44.068008 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:18:44.067970 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:18:44.408696 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:44.408648 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f07f3a4-50a6-4dcf-9528-bc03555ce3d4" path="/var/lib/kubelet/pods/1f07f3a4-50a6-4dcf-9528-bc03555ce3d4/volumes" Oct 30 17:18:44.409292 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:44.409267 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c93bf2-386e-4a57-b52d-5c12f0da7671" path="/var/lib/kubelet/pods/c9c93bf2-386e-4a57-b52d-5c12f0da7671/volumes" Oct 30 17:18:44.546422 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:44.546386 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:44.660841 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:44.660727 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:44.772380 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:18:44.772317 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:09.100180 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100139 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn"] Oct 30 17:19:09.100605 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100539 6202 cpu_manager.go:414] "containerMap: removing container" podUID="5b428320-6e0b-463f-be2e-74ec2cfda977" containerName="payload-pre-processing" Oct 30 17:19:09.100605 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100553 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b428320-6e0b-463f-be2e-74ec2cfda977" containerName="payload-pre-processing" Oct 30 17:19:09.100605 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100567 6202 cpu_manager.go:414] "containerMap: removing container" podUID="dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" containerName="istio-proxy" Oct 30 17:19:09.100605 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100573 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" containerName="istio-proxy" Oct 30 17:19:09.100605 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100584 6202 cpu_manager.go:414] "containerMap: removing container" podUID="dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5" containerName="payload-pre-processing" Oct 30 17:19:09.100605 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100589 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5" containerName="payload-pre-processing" Oct 30 17:19:09.100605 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100597 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a46e1ced-667f-4218-9fe5-4dafa958b55e" containerName="payload-processing" Oct 30 17:19:09.100605 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100601 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a46e1ced-667f-4218-9fe5-4dafa958b55e" containerName="payload-processing" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100620 6202 cpu_manager.go:414] "containerMap: removing container" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100625 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100637 6202 cpu_manager.go:414] "containerMap: removing container" podUID="19c858ea-1556-446d-ac02-5c3b2cc6f709" containerName="payload-processing" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100641 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c858ea-1556-446d-ac02-5c3b2cc6f709" containerName="payload-processing" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100647 6202 cpu_manager.go:414] "containerMap: removing container" podUID="ac56359c-6664-4198-88ef-2208c355cd11" containerName="istio-proxy" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100651 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac56359c-6664-4198-88ef-2208c355cd11" containerName="istio-proxy" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100658 6202 cpu_manager.go:414] "containerMap: removing container" podUID="ac56359c-6664-4198-88ef-2208c355cd11" containerName="istio-proxy" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100662 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac56359c-6664-4198-88ef-2208c355cd11" containerName="istio-proxy" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100669 6202 cpu_manager.go:414] "containerMap: removing container" podUID="dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" containerName="istio-proxy" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100673 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" containerName="istio-proxy" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100679 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9f10fc70-1bf2-4cca-ace8-47fd5063eada" containerName="revoke-keys" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100683 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f10fc70-1bf2-4cca-ace8-47fd5063eada" containerName="revoke-keys" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100747 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c858ea-1556-446d-ac02-5c3b2cc6f709" containerName="payload-processing" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100757 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac56359c-6664-4198-88ef-2208c355cd11" containerName="istio-proxy" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100789 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac56359c-6664-4198-88ef-2208c355cd11" containerName="istio-proxy" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100800 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" containerName="istio-proxy" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100807 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f10fc70-1bf2-4cca-ace8-47fd5063eada" containerName="revoke-keys" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100814 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b428320-6e0b-463f-be2e-74ec2cfda977" containerName="payload-pre-processing" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100820 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3d175f-3803-4ef3-b5bc-2b3c55d88a7a" containerName="istio-proxy" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100826 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfbacf8f-cd50-4ed5-a2ed-e7ad2a9a18a5" containerName="payload-pre-processing" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100832 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a46e1ced-667f-4218-9fe5-4dafa958b55e" containerName="payload-processing" Oct 30 17:19:09.100913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.100840 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cbb0ec9-d73e-4b74-bce1-15b919acb9b4" containerName="main" Oct 30 17:19:09.104159 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.104140 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.132741 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.132705 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn"] Oct 30 17:19:09.157127 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.157095 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.157287 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.157133 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.157287 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.157151 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.157287 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.157266 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.157503 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.157299 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.157503 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.157324 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.157503 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.157417 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.157503 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.157458 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvbjv\" (UniqueName: \"kubernetes.io/projected/27b9c1c4-c3da-4caf-a594-b391b48cc93a-kube-api-access-xvbjv\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.157746 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.157507 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.258726 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.258688 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvbjv\" (UniqueName: \"kubernetes.io/projected/27b9c1c4-c3da-4caf-a594-b391b48cc93a-kube-api-access-xvbjv\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.258929 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.258736 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.258929 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.258761 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.258929 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.258803 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.258929 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.258828 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.258929 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.258871 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.258929 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.258898 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.258929 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.258923 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.259306 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.258992 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.259380 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.259319 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.259380 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.259339 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.259489 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.259402 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.259489 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.259468 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.259728 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.259710 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.261360 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.261328 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.261580 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.261563 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.283127 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.283083 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.283333 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.283317 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvbjv\" (UniqueName: \"kubernetes.io/projected/27b9c1c4-c3da-4caf-a594-b391b48cc93a-kube-api-access-xvbjv\") pod \"maas-default-gateway-openshift-default-76f4876667-ffkqn\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.417957 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.417878 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:09.726112 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.726079 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn"] Oct 30 17:19:09.726296 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:19:09.726275 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b9c1c4_c3da_4caf_a594_b391b48cc93a.slice/crio-d72848ce6c064a7adace945d06aaee9cfd3e16ac9f6936c4ff2c99b7d346de67 WatchSource:0}: Error finding container d72848ce6c064a7adace945d06aaee9cfd3e16ac9f6936c4ff2c99b7d346de67: Status 404 returned error can't find the container with id d72848ce6c064a7adace945d06aaee9cfd3e16ac9f6936c4ff2c99b7d346de67 Oct 30 17:19:09.728351 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.728319 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:19:09.728434 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.728383 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:19:09.728434 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:09.728413 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:19:10.570402 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:10.570366 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" event={"ID":"27b9c1c4-c3da-4caf-a594-b391b48cc93a","Type":"ContainerStarted","Data":"95d5b5c0030e559bb54ce0010cfec3dfd5b13e18e15e5f364b44c9adefdcba16"} Oct 30 17:19:10.570945 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:10.570418 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" event={"ID":"27b9c1c4-c3da-4caf-a594-b391b48cc93a","Type":"ContainerStarted","Data":"d72848ce6c064a7adace945d06aaee9cfd3e16ac9f6936c4ff2c99b7d346de67"} Oct 30 17:19:10.630892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:10.630826 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" podStartSLOduration=1.630808219 podStartE2EDuration="1.630808219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:19:10.629354736 +0000 UTC m=+3464.339809481" watchObservedRunningTime="2026-10-30 17:19:10.630808219 +0000 UTC m=+3464.341262967" Oct 30 17:19:11.418491 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:11.418453 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:11.420159 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:11.420126 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-ffkqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused" start-of-body= Oct 30 17:19:11.420318 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:11.420178 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" podUID="27b9c1c4-c3da-4caf-a594-b391b48cc93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused" Oct 30 17:19:12.319979 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:12.319946 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:12.419057 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:12.419023 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-ffkqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused" start-of-body= Oct 30 17:19:12.419239 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:12.419078 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" podUID="27b9c1c4-c3da-4caf-a594-b391b48cc93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused" Oct 30 17:19:13.419211 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:13.419175 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-ffkqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused" start-of-body= Oct 30 17:19:13.419600 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:13.419229 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" podUID="27b9c1c4-c3da-4caf-a594-b391b48cc93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused" Oct 30 17:19:14.089160 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:19:14.089125 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:19:14.418533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:14.418442 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-ffkqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused" start-of-body= Oct 30 17:19:14.418533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:14.418504 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" podUID="27b9c1c4-c3da-4caf-a594-b391b48cc93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused" Oct 30 17:19:14.819261 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:14.819157 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:15.419028 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:15.418991 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-ffkqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused" start-of-body= Oct 30 17:19:15.419292 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:15.419049 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" podUID="27b9c1c4-c3da-4caf-a594-b391b48cc93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused" Oct 30 17:19:16.418245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:16.418205 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-ffkqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused" start-of-body= Oct 30 17:19:16.418735 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:16.418266 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" podUID="27b9c1c4-c3da-4caf-a594-b391b48cc93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused" Oct 30 17:19:18.062529 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:18.062498 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:18.063068 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:18.062895 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:18.063809 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:18.063792 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:19:34.834829 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:34.834797 6202 scope.go:122] "RemoveContainer" containerID="ad8df48270bc962a37e0181022ff5fe82e26babb2efed5189c271fb67463a728" Oct 30 17:19:34.844041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:34.844015 6202 scope.go:122] "RemoveContainer" containerID="13e15c67845abf5ace4890ba9f9dc9cc5278afec306dd72e600a07e805f5a5c1" Oct 30 17:19:34.852691 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:34.852672 6202 scope.go:122] "RemoveContainer" containerID="ec093f90000083dcaf68d34f7c4270e6b43175de8a1b3bb20dd54d399bb6b0b7" Oct 30 17:19:34.860822 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:34.860803 6202 scope.go:122] "RemoveContainer" containerID="f0f41d55174bc283484e836248bacca23701ef53eb7b40105ca811094102e8de" Oct 30 17:19:43.275822 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:43.275783 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:44.067190 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:19:44.067150 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:19:52.742027 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:52.741986 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r"] Oct 30 17:19:52.747383 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:52.747363 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r" Oct 30 17:19:52.768061 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:52.768022 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r"] Oct 30 17:19:52.768457 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:52.768427 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53241a31-ced2-49d4-8309-05ac9ea4f166-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-hg28r\" (UID: \"53241a31-ced2-49d4-8309-05ac9ea4f166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r" Oct 30 17:19:52.768612 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:52.768554 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53241a31-ced2-49d4-8309-05ac9ea4f166-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-hg28r\" (UID: \"53241a31-ced2-49d4-8309-05ac9ea4f166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r" Oct 30 17:19:52.768691 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:52.768640 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9btx\" (UniqueName: \"kubernetes.io/projected/53241a31-ced2-49d4-8309-05ac9ea4f166-kube-api-access-b9btx\") pod \"maas-api-revoke-sub-c1e271a9-hg28r\" (UID: \"53241a31-ced2-49d4-8309-05ac9ea4f166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r" Oct 30 17:19:52.779663 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:52.779638 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:19:52.779872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:52.779641 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:19:52.779872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:52.779709 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:19:52.779872 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:52.779789 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:19:52.869907 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:52.869866 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9btx\" (UniqueName: \"kubernetes.io/projected/53241a31-ced2-49d4-8309-05ac9ea4f166-kube-api-access-b9btx\") pod \"maas-api-revoke-sub-c1e271a9-hg28r\" (UID: \"53241a31-ced2-49d4-8309-05ac9ea4f166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r" Oct 30 17:19:52.870109 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:52.870004 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53241a31-ced2-49d4-8309-05ac9ea4f166-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-hg28r\" (UID: \"53241a31-ced2-49d4-8309-05ac9ea4f166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r" Oct 30 17:19:52.870109 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:52.870089 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53241a31-ced2-49d4-8309-05ac9ea4f166-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-hg28r\" (UID: \"53241a31-ced2-49d4-8309-05ac9ea4f166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r" Oct 30 17:19:52.870681 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:52.870655 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53241a31-ced2-49d4-8309-05ac9ea4f166-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-hg28r\" (UID: \"53241a31-ced2-49d4-8309-05ac9ea4f166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r" Oct 30 17:19:52.872719 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:52.872683 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53241a31-ced2-49d4-8309-05ac9ea4f166-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-hg28r\" (UID: \"53241a31-ced2-49d4-8309-05ac9ea4f166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r" Oct 30 17:19:52.894839 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:52.894813 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9btx\" (UniqueName: \"kubernetes.io/projected/53241a31-ced2-49d4-8309-05ac9ea4f166-kube-api-access-b9btx\") pod \"maas-api-revoke-sub-c1e271a9-hg28r\" (UID: \"53241a31-ced2-49d4-8309-05ac9ea4f166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r" Oct 30 17:19:53.060421 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:53.060333 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r" Oct 30 17:19:53.352070 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:19:53.352039 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53241a31_ced2_49d4_8309_05ac9ea4f166.slice/crio-8b0b88257dfcf03c1972090eab25633c1c8c9574d9b31392ae1972269c58b091 WatchSource:0}: Error finding container 8b0b88257dfcf03c1972090eab25633c1c8c9574d9b31392ae1972269c58b091: Status 404 returned error can't find the container with id 8b0b88257dfcf03c1972090eab25633c1c8c9574d9b31392ae1972269c58b091 Oct 30 17:19:53.352070 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:53.352041 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r"] Oct 30 17:19:53.827478 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:53.827433 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r" event={"ID":"53241a31-ced2-49d4-8309-05ac9ea4f166","Type":"ContainerStarted","Data":"2d38cc7b237faae474341e05210902953bbaff7ce9f7715d3dff74c5b3eb7b95"} Oct 30 17:19:53.827945 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:53.827491 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r" event={"ID":"53241a31-ced2-49d4-8309-05ac9ea4f166","Type":"ContainerStarted","Data":"8b0b88257dfcf03c1972090eab25633c1c8c9574d9b31392ae1972269c58b091"} Oct 30 17:19:53.870740 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:53.870681 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r" podStartSLOduration=1.8706672690000001 podStartE2EDuration="1.870667269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:19:53.869805526 +0000 UTC m=+3507.580260462" watchObservedRunningTime="2026-10-30 17:19:53.870667269 +0000 UTC m=+3507.581122014" Oct 30 17:19:54.834433 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:54.834399 6202 generic.go:356] "Generic (PLEG): container finished" podID="53241a31-ced2-49d4-8309-05ac9ea4f166" containerID="2d38cc7b237faae474341e05210902953bbaff7ce9f7715d3dff74c5b3eb7b95" exitCode=0 Oct 30 17:19:54.834901 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:54.834489 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r" event={"ID":"53241a31-ced2-49d4-8309-05ac9ea4f166","Type":"ContainerDied","Data":"2d38cc7b237faae474341e05210902953bbaff7ce9f7715d3dff74c5b3eb7b95"} Oct 30 17:19:55.973675 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:55.973649 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r" Oct 30 17:19:55.998939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:55.998907 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53241a31-ced2-49d4-8309-05ac9ea4f166-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53241a31-ced2-49d4-8309-05ac9ea4f166-maas-api-xks-ca\") pod \"53241a31-ced2-49d4-8309-05ac9ea4f166\" (UID: \"53241a31-ced2-49d4-8309-05ac9ea4f166\") " Oct 30 17:19:55.999083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:55.999026 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53241a31-ced2-49d4-8309-05ac9ea4f166-kube-api-access-b9btx\" (UniqueName: \"kubernetes.io/projected/53241a31-ced2-49d4-8309-05ac9ea4f166-kube-api-access-b9btx\") pod \"53241a31-ced2-49d4-8309-05ac9ea4f166\" (UID: \"53241a31-ced2-49d4-8309-05ac9ea4f166\") " Oct 30 17:19:55.999083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:55.999073 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53241a31-ced2-49d4-8309-05ac9ea4f166-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53241a31-ced2-49d4-8309-05ac9ea4f166-maas-api-openshift-ca\") pod \"53241a31-ced2-49d4-8309-05ac9ea4f166\" (UID: \"53241a31-ced2-49d4-8309-05ac9ea4f166\") " Oct 30 17:19:55.999468 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:55.999440 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53241a31-ced2-49d4-8309-05ac9ea4f166-maas-api-openshift-ca" pod "53241a31-ced2-49d4-8309-05ac9ea4f166" (UID: "53241a31-ced2-49d4-8309-05ac9ea4f166"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:19:56.001188 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:56.001165 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53241a31-ced2-49d4-8309-05ac9ea4f166-kube-api-access-b9btx" pod "53241a31-ced2-49d4-8309-05ac9ea4f166" (UID: "53241a31-ced2-49d4-8309-05ac9ea4f166"). InnerVolumeSpecName "kube-api-access-b9btx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:19:56.001285 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:56.001227 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53241a31-ced2-49d4-8309-05ac9ea4f166-maas-api-xks-ca" pod "53241a31-ced2-49d4-8309-05ac9ea4f166" (UID: "53241a31-ced2-49d4-8309-05ac9ea4f166"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:19:56.100276 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:56.100194 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53241a31-ced2-49d4-8309-05ac9ea4f166-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:19:56.100276 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:56.100222 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9btx\" (UniqueName: \"kubernetes.io/projected/53241a31-ced2-49d4-8309-05ac9ea4f166-kube-api-access-b9btx\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:19:56.100276 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:56.100232 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53241a31-ced2-49d4-8309-05ac9ea4f166-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:19:56.851015 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:56.850974 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r" event={"ID":"53241a31-ced2-49d4-8309-05ac9ea4f166","Type":"ContainerDied","Data":"8b0b88257dfcf03c1972090eab25633c1c8c9574d9b31392ae1972269c58b091"} Oct 30 17:19:56.851202 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:56.851033 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b0b88257dfcf03c1972090eab25633c1c8c9574d9b31392ae1972269c58b091" Oct 30 17:19:56.851202 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:19:56.850990 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r" Oct 30 17:20:09.268329 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.268235 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6"] Oct 30 17:20:09.268912 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.268666 6202 cpu_manager.go:414] "containerMap: removing container" podUID="53241a31-ced2-49d4-8309-05ac9ea4f166" containerName="revoke-keys" Oct 30 17:20:09.268912 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.268680 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53241a31-ced2-49d4-8309-05ac9ea4f166" containerName="revoke-keys" Oct 30 17:20:09.268912 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.268799 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="53241a31-ced2-49d4-8309-05ac9ea4f166" containerName="revoke-keys" Oct 30 17:20:09.272044 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.272020 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.305203 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.305158 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6"] Oct 30 17:20:09.327681 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.327645 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.327983 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.327960 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.328181 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.328146 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.328395 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.328375 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.328613 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.328594 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.328874 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.328837 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.329086 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.329060 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.329256 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.329162 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a034df-8dc0-489b-a623-4bb9e730e6ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.329256 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.329204 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkpw8\" (UniqueName: \"kubernetes.io/projected/04a034df-8dc0-489b-a623-4bb9e730e6ad-kube-api-access-mkpw8\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.430500 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.430460 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.430715 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.430514 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.430715 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.430544 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.430715 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.430572 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.430715 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.430621 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a034df-8dc0-489b-a623-4bb9e730e6ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.430715 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.430656 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkpw8\" (UniqueName: \"kubernetes.io/projected/04a034df-8dc0-489b-a623-4bb9e730e6ad-kube-api-access-mkpw8\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.430715 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.430695 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.431168 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.430739 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.431168 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.431022 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.431168 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.431040 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.431168 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.431053 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.431396 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.431378 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.431637 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.431608 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.431716 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.431614 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a034df-8dc0-489b-a623-4bb9e730e6ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.433583 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.433557 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.433669 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.433590 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.455389 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.455357 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkpw8\" (UniqueName: \"kubernetes.io/projected/04a034df-8dc0-489b-a623-4bb9e730e6ad-kube-api-access-mkpw8\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.455532 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.455361 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5mcb6\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.590256 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.589970 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:09.965135 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.965034 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6"] Oct 30 17:20:09.974159 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:20:09.974116 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a034df_8dc0_489b_a623_4bb9e730e6ad.slice/crio-f1386fed6fda77cd93ff0cf0378dfc1dde4dd2d34020bafe0d19af5fb598297a WatchSource:0}: Error finding container f1386fed6fda77cd93ff0cf0378dfc1dde4dd2d34020bafe0d19af5fb598297a: Status 404 returned error can't find the container with id f1386fed6fda77cd93ff0cf0378dfc1dde4dd2d34020bafe0d19af5fb598297a Oct 30 17:20:09.977945 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.977891 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:20:09.978202 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.978175 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:20:09.978406 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:09.978367 6202 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:20:10.936586 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:10.936544 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" event={"ID":"04a034df-8dc0-489b-a623-4bb9e730e6ad","Type":"ContainerStarted","Data":"7c24ac9e5690ea0e3856fa3b3dfc782f4c6563444bcda55e43fcb5a57a9c0276"} Oct 30 17:20:10.937024 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:10.936600 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" event={"ID":"04a034df-8dc0-489b-a623-4bb9e730e6ad","Type":"ContainerStarted","Data":"f1386fed6fda77cd93ff0cf0378dfc1dde4dd2d34020bafe0d19af5fb598297a"} Oct 30 17:20:11.016914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:11.016860 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" podStartSLOduration=2.0168462639999998 podStartE2EDuration="2.016846264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:11.002160196 +0000 UTC m=+3524.712614940" watchObservedRunningTime="2026-10-30 17:20:11.016846264 +0000 UTC m=+3524.727301002" Oct 30 17:20:11.590785 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:11.590729 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:11.592398 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:11.592372 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5mcb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body= Oct 30 17:20:11.592540 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:11.592418 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" podUID="04a034df-8dc0-489b-a623-4bb9e730e6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" Oct 30 17:20:12.591744 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:12.591706 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5mcb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body= Oct 30 17:20:12.592193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:12.591797 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" podUID="04a034df-8dc0-489b-a623-4bb9e730e6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" Oct 30 17:20:13.590928 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:13.590896 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5mcb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body= Oct 30 17:20:13.591121 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:13.590953 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" podUID="04a034df-8dc0-489b-a623-4bb9e730e6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" Oct 30 17:20:14.090680 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:20:14.090644 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:20:14.591203 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:14.591171 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5mcb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body= Oct 30 17:20:14.591395 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:14.591224 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" podUID="04a034df-8dc0-489b-a623-4bb9e730e6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" Oct 30 17:20:15.591493 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:15.591455 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5mcb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body= Oct 30 17:20:15.591957 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:15.591514 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" podUID="04a034df-8dc0-489b-a623-4bb9e730e6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" Oct 30 17:20:16.591612 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:16.591576 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5mcb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body= Oct 30 17:20:16.592051 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:16.591626 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" podUID="04a034df-8dc0-489b-a623-4bb9e730e6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" Oct 30 17:20:17.591229 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:17.591189 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5mcb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body= Oct 30 17:20:17.591421 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:17.591266 6202 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" podUID="04a034df-8dc0-489b-a623-4bb9e730e6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" Oct 30 17:20:18.595138 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:18.595108 6202 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:18.595558 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:18.595377 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:18.596103 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:18.596081 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:20:19.566061 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:19.566026 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:19.742093 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:19.742062 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:34.899864 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:34.899831 6202 scope.go:122] "RemoveContainer" containerID="f381bea4acb2b9b99d6c395b94315cc539ee5ad209bba1eee1addf424e4c86d8" Oct 30 17:20:34.909298 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:34.909276 6202 scope.go:122] "RemoveContainer" containerID="855e06add1bf915a460a51ea58d0b9ff41f972ca5081e9041280c86d607d223b" Oct 30 17:20:34.917741 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:34.917720 6202 scope.go:122] "RemoveContainer" containerID="da5a898c5fa3a3298e58c47a64614a52b7660c05eecb6b943a1f027ce338ad59" Oct 30 17:20:34.926286 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:34.926264 6202 scope.go:122] "RemoveContainer" containerID="9fa7ab85b2b0ba87688e0f74b148559e5ee037496851db80276c48341ad1bae2" Oct 30 17:20:34.934576 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:34.934545 6202 scope.go:122] "RemoveContainer" containerID="04c57318e3c5036fbac4debfd93f9b64e27d22929f2394f2013097ccbe6cf692" Oct 30 17:20:34.942553 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:34.942535 6202 scope.go:122] "RemoveContainer" containerID="aaaaab53f3a5e64e01df7036cc81e706050cd6d3e1f4d060b495d64a2772addd" Oct 30 17:20:43.429461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:20:43.429417 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:44.067160 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:20:44.067121 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:21:14.067501 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:21:14.067458 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:21:34.981475 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:21:34.981442 6202 scope.go:122] "RemoveContainer" containerID="5a88dc218b0d16e9409f1344679215996b639e59bbd23e175f167b53e4a21a4f" Oct 30 17:21:34.990868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:21:34.990845 6202 scope.go:122] "RemoveContainer" containerID="72bdcb876f0b9a6e43aa1f26c616ec3be405fcb29857145a00ed91a5cf09eb5b" Oct 30 17:21:40.031253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:21:40.031155 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx"] Oct 30 17:21:40.053967 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:21:40.048006 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5"] Oct 30 17:21:40.053967 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:21:40.050645 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de1d73b-8z7fx"] Oct 30 17:21:40.057186 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:21:40.057157 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ed6312-nd7s5"] Oct 30 17:21:40.421818 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:21:40.421159 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57982247-907d-43dd-9274-a28b925bcf5b" path="/var/lib/kubelet/pods/57982247-907d-43dd-9274-a28b925bcf5b/volumes" Oct 30 17:21:40.421818 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:21:40.421751 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1687a6a-5aa8-4770-85c4-bfd703bb96b8" path="/var/lib/kubelet/pods/a1687a6a-5aa8-4770-85c4-bfd703bb96b8/volumes" Oct 30 17:21:43.524606 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:21:43.524565 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:44.066926 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:21:44.066887 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:21:46.036415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:21:46.036379 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn"] Oct 30 17:21:46.049972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:21:46.049936 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc"] Oct 30 17:21:46.051113 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:21:46.051101 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de92707-885gn"] Oct 30 17:21:46.052178 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:21:46.052160 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed70e1c-k5wdc"] Oct 30 17:21:46.408431 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:21:46.408402 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="635a2ae3-b676-441a-81f8-8037caefe261" path="/var/lib/kubelet/pods/635a2ae3-b676-441a-81f8-8037caefe261/volumes" Oct 30 17:21:46.408857 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:21:46.408843 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9f595f1-0e66-4066-af92-0ded0ada9ac1" path="/var/lib/kubelet/pods/a9f595f1-0e66-4066-af92-0ded0ada9ac1/volumes" Oct 30 17:21:59.836357 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:21:59.836303 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:00.014720 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:00.014679 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:08.029790 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:08.029737 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l"] Oct 30 17:22:08.043286 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:08.043253 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk"] Oct 30 17:22:08.044291 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:08.044280 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef032b4-2sd9l"] Oct 30 17:22:08.045352 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:08.045309 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91156806-gkcjk"] Oct 30 17:22:08.408821 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:08.408785 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76f830ec-ce77-4535-8d17-a6d64e901053" path="/var/lib/kubelet/pods/76f830ec-ce77-4535-8d17-a6d64e901053/volumes" Oct 30 17:22:08.409194 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:08.409181 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="823d03c5-f042-4c15-8e73-d57c779e8a9a" path="/var/lib/kubelet/pods/823d03c5-f042-4c15-8e73-d57c779e8a9a/volumes" Oct 30 17:22:10.506370 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:10.506335 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:14.067091 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:22:14.067058 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:22:29.767145 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:29.767109 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:35.017568 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:35.017533 6202 scope.go:122] "RemoveContainer" containerID="73ba203fb90fcf1efff3f89accc1127be6470fd992675bc5e1f838a45e2c9532" Oct 30 17:22:35.026429 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:35.026406 6202 scope.go:122] "RemoveContainer" containerID="3d4e638b6fb26d2c811f8437d9a1ca607c26982a5f5d3eee08b05b24dfdaff01" Oct 30 17:22:35.030507 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:35.030478 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg"] Oct 30 17:22:35.035198 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:35.035180 6202 scope.go:122] "RemoveContainer" containerID="7e0e59b9fe0042ef53e0c448cc7e3118281be6dfa9ddec9f9ca7e87b4efccee1" Oct 30 17:22:35.043326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:35.043296 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d33b897-8hqvg"] Oct 30 17:22:35.043703 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:35.043679 6202 scope.go:122] "RemoveContainer" containerID="fb9c21c7bb078610a88e23ded98f7a6a4c5d0331c950a22b9087850c3d7dbfda" Oct 30 17:22:35.051728 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:35.051709 6202 scope.go:122] "RemoveContainer" containerID="f4f34b03deb10988c817c6c71da7528283c7f238c91fd666915298ba4850aea7" Oct 30 17:22:35.059577 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:35.059554 6202 scope.go:122] "RemoveContainer" containerID="c3f6e942e1d519eff4860ebe3fd8959b4f53c98db3cafdaa1930239a0a33a660" Oct 30 17:22:35.070284 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:35.070263 6202 scope.go:122] "RemoveContainer" containerID="b8b5416a466ab3c2f9f260e36f6baf1b36d05da1a679f87a60c518513e9be9f6" Oct 30 17:22:35.078585 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:35.078561 6202 scope.go:122] "RemoveContainer" containerID="8413d9e67328febb578ac9921cd19768e611308fb23476121f8e9db6b60d9c09" Oct 30 17:22:36.409426 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:36.409394 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c2d2f9-1429-4441-88f2-78b89feabbee" path="/var/lib/kubelet/pods/a3c2d2f9-1429-4441-88f2-78b89feabbee/volumes" Oct 30 17:22:43.621617 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:43.621572 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:44.066212 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:22:44.066130 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:22:55.030020 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:55.029982 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh"] Oct 30 17:22:55.043898 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:55.043860 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5190542-kjvsh"] Oct 30 17:22:56.408668 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:22:56.408631 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f10fc70-1bf2-4cca-ace8-47fd5063eada" path="/var/lib/kubelet/pods/9f10fc70-1bf2-4cca-ace8-47fd5063eada/volumes" Oct 30 17:23:14.067076 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:23:14.067030 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:23:35.125900 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:23:35.125866 6202 scope.go:122] "RemoveContainer" containerID="f267735ff757c5b712f19418671adfb1f1007e408b068dc89f077e695d920d47" Oct 30 17:23:35.134733 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:23:35.134711 6202 scope.go:122] "RemoveContainer" containerID="f21995a9a5abd0fb09efc815d3f43dad289b981dcf7783fa5d90347a2cf8e0f0" Oct 30 17:23:35.146036 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:23:35.146011 6202 scope.go:122] "RemoveContainer" containerID="1554465d4e37aca41f321d597cfaf0eaf746142ea85fc4dfca6e848bb228a0e5" Oct 30 17:23:35.153978 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:23:35.153876 6202 scope.go:122] "RemoveContainer" containerID="09f70a0b42f9f400409753b7c3f5643f9e5494fc0fd82337fc1e9a1497d213aa" Oct 30 17:23:35.161091 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:23:35.161072 6202 scope.go:122] "RemoveContainer" containerID="71473a71dafd4e04c390038d2599446d1bab0df6e60c56f2f3b94467fb05a556" Oct 30 17:23:35.168289 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:23:35.168271 6202 scope.go:122] "RemoveContainer" containerID="3188b4d58a9d8439a45a93d0a07cabdb6db911183e052d205d9d116ce73d4468" Oct 30 17:23:35.175664 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:23:35.175646 6202 scope.go:122] "RemoveContainer" containerID="bd99352b1c8ebee123bd4ecd3d69e825debfc360d7b0e19b07154166dfa1c25b" Oct 30 17:23:35.182921 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:23:35.182905 6202 scope.go:122] "RemoveContainer" containerID="2398d113d49d69c48014065a60e4cfe8e1816f9abaa7474a6396e16ca24a0c6c" Oct 30 17:23:35.190110 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:23:35.190093 6202 scope.go:122] "RemoveContainer" containerID="53eba797a2a163e4a954974c1ef60a3903a914adb4f064239f8077d6ee1bb3c4" Oct 30 17:23:35.198244 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:23:35.198223 6202 scope.go:122] "RemoveContainer" containerID="6f6dd964ac58f7a59e62f43c4ebb86cbe33edbe86a8ffe034239757da41885bd" Oct 30 17:23:35.205705 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:23:35.205689 6202 scope.go:122] "RemoveContainer" containerID="8d9bac77c7e666b0903b8aeb85684c1afb33df28bc320ddc530c9ca98cc3aeac" Oct 30 17:23:43.699289 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:23:43.699250 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:44.067239 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:23:44.067152 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:23:52.801112 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:23:52.801065 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:07.350607 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:07.350567 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:09.721286 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:09.721235 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:10.197508 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:10.197473 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:10.452652 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:10.452554 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:13.916955 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:13.916907 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:14.030704 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:14.030665 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:14.067150 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:24:14.067115 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:24:15.380202 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:15.380169 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:15.532238 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:15.532191 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:15.653963 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:15.653859 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:17.270349 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:17.270292 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:18.803472 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:18.803423 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:20.721408 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:20.721373 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:21.828193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:21.828160 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:22.237018 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:22.236938 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:22.551397 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:22.551325 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:23.948315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:23.948266 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:25.700566 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:25.700532 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:27.376898 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:27.376862 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:28.937337 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:28.937307 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:29.048723 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:29.048686 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:29.596322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:29.596293 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:29.708239 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:29.708188 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:29.826197 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:29.826164 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:30.650753 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:30.650706 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:31.221928 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:31.221898 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:32.494230 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:32.494195 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:34.287987 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:34.287955 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:35.257108 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:35.257077 6202 scope.go:122] "RemoveContainer" containerID="58b1b0a0eda9cac728d84716a448d200055d01c3e5689bc7baef18b265b4549f" Oct 30 17:24:35.862071 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:35.862028 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:37.481340 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:37.481300 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:39.010190 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:39.010155 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:40.689176 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:40.689143 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:42.197416 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:42.197363 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:43.770710 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:43.770676 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:43.877331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:43.877302 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:44.067945 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:24:44.067855 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:24:44.184955 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:44.184918 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:45.484158 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:45.484126 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:47.055857 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:47.055826 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:48.762222 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:48.762181 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:50.351672 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:50.351633 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:51.927373 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:51.927344 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:53.447417 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:53.447372 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:54.991370 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:54.991325 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:56.028128 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:56.028096 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r"] Oct 30 17:24:56.041068 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:56.041042 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hg28r"] Oct 30 17:24:56.408674 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:56.408641 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53241a31-ced2-49d4-8309-05ac9ea4f166" path="/var/lib/kubelet/pods/53241a31-ced2-49d4-8309-05ac9ea4f166/volumes" Oct 30 17:24:56.576493 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:56.576462 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:58.179248 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:58.179215 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:59.867481 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:24:59.867449 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:01.531743 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:01.531693 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:03.068384 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:03.068350 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:04.036238 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:04.036203 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:04.942201 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:04.942170 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:06.629174 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:06.629131 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:08.295271 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:08.295240 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:09.080115 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:09.080083 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:09.847652 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:09.847608 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:11.486804 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:11.486750 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:13.050561 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:13.050532 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:14.067758 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:25:14.067720 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:25:14.532088 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:14.532057 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:14.721144 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:14.721111 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:14.830130 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:14.830090 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:16.257922 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:16.257885 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:17.604016 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:17.603982 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:18.061200 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:18.061108 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:18.175415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:18.175380 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:28.725871 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:28.725833 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:34.657080 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:34.657047 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:34.769461 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:34.769433 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:35.066476 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:35.066372 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:38.135067 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:38.135018 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:38.245040 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:38.244997 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:43.973521 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:43.973450 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:44.066537 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:25:44.066497 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:25:44.300270 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:44.300182 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:45.351293 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:45.351260 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:47.477616 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:47.477575 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:53.080269 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:25:53.080231 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:01.861199 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:01.861159 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq"] Oct 30 17:26:01.861735 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:01.861383 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq" podUID="07e83efa-0d59-4133-8588-133fe009c226" containerName="manager" containerID="cri-o://7b9e583263389fb1d89e7b419a2437c0d36cf8e04cc74a6924370766396ffb05" gracePeriod=10 Oct 30 17:26:02.122196 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:02.122138 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq" Oct 30 17:26:02.234924 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:02.234895 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e83efa-0d59-4133-8588-133fe009c226-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07e83efa-0d59-4133-8588-133fe009c226-extensions-socket-volume\") pod \"07e83efa-0d59-4133-8588-133fe009c226\" (UID: \"07e83efa-0d59-4133-8588-133fe009c226\") " Oct 30 17:26:02.234924 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:02.234936 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e83efa-0d59-4133-8588-133fe009c226-kube-api-access-zrwgk\" (UniqueName: \"kubernetes.io/projected/07e83efa-0d59-4133-8588-133fe009c226-kube-api-access-zrwgk\") pod \"07e83efa-0d59-4133-8588-133fe009c226\" (UID: \"07e83efa-0d59-4133-8588-133fe009c226\") " Oct 30 17:26:02.235317 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:02.235295 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e83efa-0d59-4133-8588-133fe009c226-extensions-socket-volume" pod "07e83efa-0d59-4133-8588-133fe009c226" (UID: "07e83efa-0d59-4133-8588-133fe009c226"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:02.237344 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:02.237317 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e83efa-0d59-4133-8588-133fe009c226-kube-api-access-zrwgk" pod "07e83efa-0d59-4133-8588-133fe009c226" (UID: "07e83efa-0d59-4133-8588-133fe009c226"). InnerVolumeSpecName "kube-api-access-zrwgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:02.335839 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:02.335801 6202 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07e83efa-0d59-4133-8588-133fe009c226-extensions-socket-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:26:02.335839 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:02.335833 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrwgk\" (UniqueName: \"kubernetes.io/projected/07e83efa-0d59-4133-8588-133fe009c226-kube-api-access-zrwgk\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:26:02.890353 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:02.890317 6202 generic.go:356] "Generic (PLEG): container finished" podID="07e83efa-0d59-4133-8588-133fe009c226" containerID="7b9e583263389fb1d89e7b419a2437c0d36cf8e04cc74a6924370766396ffb05" exitCode=0 Oct 30 17:26:02.890868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:02.890367 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq" Oct 30 17:26:02.890868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:02.890374 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq" event={"ID":"07e83efa-0d59-4133-8588-133fe009c226","Type":"ContainerDied","Data":"7b9e583263389fb1d89e7b419a2437c0d36cf8e04cc74a6924370766396ffb05"} Oct 30 17:26:02.890868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:02.890432 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq" event={"ID":"07e83efa-0d59-4133-8588-133fe009c226","Type":"ContainerDied","Data":"eb8abb116c953db112b49fc2f35b1f75f9b00eed97410c34a3af20f3fe5e989f"} Oct 30 17:26:02.890868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:02.890455 6202 scope.go:122] "RemoveContainer" containerID="7b9e583263389fb1d89e7b419a2437c0d36cf8e04cc74a6924370766396ffb05" Oct 30 17:26:02.900047 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:02.900031 6202 scope.go:122] "RemoveContainer" containerID="7b9e583263389fb1d89e7b419a2437c0d36cf8e04cc74a6924370766396ffb05" Oct 30 17:26:02.900309 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:26:02.900289 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9e583263389fb1d89e7b419a2437c0d36cf8e04cc74a6924370766396ffb05\": container with ID starting with 7b9e583263389fb1d89e7b419a2437c0d36cf8e04cc74a6924370766396ffb05 not found: ID does not exist" containerID="7b9e583263389fb1d89e7b419a2437c0d36cf8e04cc74a6924370766396ffb05" Oct 30 17:26:02.900367 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:02.900318 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9e583263389fb1d89e7b419a2437c0d36cf8e04cc74a6924370766396ffb05"} err="failed to get container status \"7b9e583263389fb1d89e7b419a2437c0d36cf8e04cc74a6924370766396ffb05\": rpc error: code = NotFound desc = could not find container \"7b9e583263389fb1d89e7b419a2437c0d36cf8e04cc74a6924370766396ffb05\": container with ID starting with 7b9e583263389fb1d89e7b419a2437c0d36cf8e04cc74a6924370766396ffb05 not found: ID does not exist" Oct 30 17:26:02.959212 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:02.959181 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq"] Oct 30 17:26:02.972530 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:02.972503 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qh4lq"] Oct 30 17:26:04.407933 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:04.407888 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07e83efa-0d59-4133-8588-133fe009c226" path="/var/lib/kubelet/pods/07e83efa-0d59-4133-8588-133fe009c226/volumes" Oct 30 17:26:14.067660 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:26:14.067615 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:26:35.284054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:35.284018 6202 scope.go:122] "RemoveContainer" containerID="2d38cc7b237faae474341e05210902953bbaff7ce9f7715d3dff74c5b3eb7b95" Oct 30 17:26:44.067057 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:26:44.067015 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:26:44.103741 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:44.103711 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:44.430986 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:26:44.430954 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:14.067343 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:27:14.067300 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:27:37.418415 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.418323 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp"] Oct 30 17:27:37.419004 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.418981 6202 cpu_manager.go:414] "containerMap: removing container" podUID="07e83efa-0d59-4133-8588-133fe009c226" containerName="manager" Oct 30 17:27:37.419104 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.419006 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e83efa-0d59-4133-8588-133fe009c226" containerName="manager" Oct 30 17:27:37.419166 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.419101 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e83efa-0d59-4133-8588-133fe009c226" containerName="manager" Oct 30 17:27:37.422090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.422070 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp" Oct 30 17:27:37.433786 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.433750 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a30a2bea-6b7f-497d-a77c-5f7007cc7076-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-ff2wp\" (UID: \"a30a2bea-6b7f-497d-a77c-5f7007cc7076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp" Oct 30 17:27:37.433884 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.433804 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6bkt\" (UniqueName: \"kubernetes.io/projected/a30a2bea-6b7f-497d-a77c-5f7007cc7076-kube-api-access-j6bkt\") pod \"maas-api-revoke-sub-d576e68b-ff2wp\" (UID: \"a30a2bea-6b7f-497d-a77c-5f7007cc7076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp" Oct 30 17:27:37.433931 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.433886 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a30a2bea-6b7f-497d-a77c-5f7007cc7076-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-ff2wp\" (UID: \"a30a2bea-6b7f-497d-a77c-5f7007cc7076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp" Oct 30 17:27:37.438135 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.438115 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:27:37.451379 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.451355 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:27:37.451379 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.451355 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:27:37.451625 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.451607 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:27:37.452786 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.452755 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp"] Oct 30 17:27:37.535286 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.535252 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a30a2bea-6b7f-497d-a77c-5f7007cc7076-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-ff2wp\" (UID: \"a30a2bea-6b7f-497d-a77c-5f7007cc7076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp" Oct 30 17:27:37.535286 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.535288 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6bkt\" (UniqueName: \"kubernetes.io/projected/a30a2bea-6b7f-497d-a77c-5f7007cc7076-kube-api-access-j6bkt\") pod \"maas-api-revoke-sub-d576e68b-ff2wp\" (UID: \"a30a2bea-6b7f-497d-a77c-5f7007cc7076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp" Oct 30 17:27:37.535521 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.535322 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a30a2bea-6b7f-497d-a77c-5f7007cc7076-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-ff2wp\" (UID: \"a30a2bea-6b7f-497d-a77c-5f7007cc7076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp" Oct 30 17:27:37.535943 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.535921 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a30a2bea-6b7f-497d-a77c-5f7007cc7076-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-ff2wp\" (UID: \"a30a2bea-6b7f-497d-a77c-5f7007cc7076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp" Oct 30 17:27:37.537821 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.537801 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a30a2bea-6b7f-497d-a77c-5f7007cc7076-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-ff2wp\" (UID: \"a30a2bea-6b7f-497d-a77c-5f7007cc7076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp" Oct 30 17:27:37.559686 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.559664 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6bkt\" (UniqueName: \"kubernetes.io/projected/a30a2bea-6b7f-497d-a77c-5f7007cc7076-kube-api-access-j6bkt\") pod \"maas-api-revoke-sub-d576e68b-ff2wp\" (UID: \"a30a2bea-6b7f-497d-a77c-5f7007cc7076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp" Oct 30 17:27:37.735118 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:37.735038 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp" Oct 30 17:27:38.027438 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:38.027405 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp"] Oct 30 17:27:38.028430 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:27:38.028401 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30a2bea_6b7f_497d_a77c_5f7007cc7076.slice/crio-4966e6148f3ad0253703881c889e8c834c07f65caf3a38975c28922448b96cc9 WatchSource:0}: Error finding container 4966e6148f3ad0253703881c889e8c834c07f65caf3a38975c28922448b96cc9: Status 404 returned error can't find the container with id 4966e6148f3ad0253703881c889e8c834c07f65caf3a38975c28922448b96cc9 Oct 30 17:27:38.330568 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:38.330535 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp" event={"ID":"a30a2bea-6b7f-497d-a77c-5f7007cc7076","Type":"ContainerStarted","Data":"4966e6148f3ad0253703881c889e8c834c07f65caf3a38975c28922448b96cc9"} Oct 30 17:27:39.337041 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:39.337009 6202 generic.go:356] "Generic (PLEG): container finished" podID="a30a2bea-6b7f-497d-a77c-5f7007cc7076" containerID="ca5ca7d9ad1d7b38b48eb006896f54e6f02b089e8e7db85356a68cd1a8ec04f1" exitCode=0 Oct 30 17:27:39.337448 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:39.337095 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp" event={"ID":"a30a2bea-6b7f-497d-a77c-5f7007cc7076","Type":"ContainerDied","Data":"ca5ca7d9ad1d7b38b48eb006896f54e6f02b089e8e7db85356a68cd1a8ec04f1"} Oct 30 17:27:40.473942 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:40.473915 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp" Oct 30 17:27:40.565558 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:40.565522 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a30a2bea-6b7f-497d-a77c-5f7007cc7076-kube-api-access-j6bkt\" (UniqueName: \"kubernetes.io/projected/a30a2bea-6b7f-497d-a77c-5f7007cc7076-kube-api-access-j6bkt\") pod \"a30a2bea-6b7f-497d-a77c-5f7007cc7076\" (UID: \"a30a2bea-6b7f-497d-a77c-5f7007cc7076\") " Oct 30 17:27:40.565716 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:40.565593 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a30a2bea-6b7f-497d-a77c-5f7007cc7076-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a30a2bea-6b7f-497d-a77c-5f7007cc7076-maas-api-xks-ca\") pod \"a30a2bea-6b7f-497d-a77c-5f7007cc7076\" (UID: \"a30a2bea-6b7f-497d-a77c-5f7007cc7076\") " Oct 30 17:27:40.565716 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:40.565686 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a30a2bea-6b7f-497d-a77c-5f7007cc7076-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a30a2bea-6b7f-497d-a77c-5f7007cc7076-maas-api-openshift-ca\") pod \"a30a2bea-6b7f-497d-a77c-5f7007cc7076\" (UID: \"a30a2bea-6b7f-497d-a77c-5f7007cc7076\") " Oct 30 17:27:40.566089 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:40.566066 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a30a2bea-6b7f-497d-a77c-5f7007cc7076-maas-api-openshift-ca" pod "a30a2bea-6b7f-497d-a77c-5f7007cc7076" (UID: "a30a2bea-6b7f-497d-a77c-5f7007cc7076"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:40.567847 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:40.567821 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a30a2bea-6b7f-497d-a77c-5f7007cc7076-maas-api-xks-ca" pod "a30a2bea-6b7f-497d-a77c-5f7007cc7076" (UID: "a30a2bea-6b7f-497d-a77c-5f7007cc7076"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:40.567964 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:40.567903 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a30a2bea-6b7f-497d-a77c-5f7007cc7076-kube-api-access-j6bkt" pod "a30a2bea-6b7f-497d-a77c-5f7007cc7076" (UID: "a30a2bea-6b7f-497d-a77c-5f7007cc7076"). InnerVolumeSpecName "kube-api-access-j6bkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:40.666382 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:40.666314 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a30a2bea-6b7f-497d-a77c-5f7007cc7076-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:27:40.666382 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:40.666342 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6bkt\" (UniqueName: \"kubernetes.io/projected/a30a2bea-6b7f-497d-a77c-5f7007cc7076-kube-api-access-j6bkt\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:27:40.666382 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:40.666352 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a30a2bea-6b7f-497d-a77c-5f7007cc7076-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:27:41.346985 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:41.346939 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp" event={"ID":"a30a2bea-6b7f-497d-a77c-5f7007cc7076","Type":"ContainerDied","Data":"4966e6148f3ad0253703881c889e8c834c07f65caf3a38975c28922448b96cc9"} Oct 30 17:27:41.347177 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:41.347003 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4966e6148f3ad0253703881c889e8c834c07f65caf3a38975c28922448b96cc9" Oct 30 17:27:41.347177 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:41.346960 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp" Oct 30 17:27:44.066677 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:27:44.066635 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:27:44.261692 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:44.261654 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:54.468690 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.468652 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489"] Oct 30 17:27:54.469243 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.469222 6202 cpu_manager.go:414] "containerMap: removing container" podUID="a30a2bea-6b7f-497d-a77c-5f7007cc7076" containerName="revoke-keys" Oct 30 17:27:54.469346 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.469246 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a30a2bea-6b7f-497d-a77c-5f7007cc7076" containerName="revoke-keys" Oct 30 17:27:54.469413 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.469353 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="a30a2bea-6b7f-497d-a77c-5f7007cc7076" containerName="revoke-keys" Oct 30 17:27:54.472709 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.472688 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489" Oct 30 17:27:54.492089 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.492061 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489"] Oct 30 17:27:54.504178 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.504149 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:27:54.504178 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.504156 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:27:54.504331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.504192 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:27:54.504331 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.504286 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:27:54.626817 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.626754 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10789212-2d42-4796-b759-7dd7ac372f19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bg489\" (UID: \"10789212-2d42-4796-b759-7dd7ac372f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489" Oct 30 17:27:54.626996 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.626889 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10789212-2d42-4796-b759-7dd7ac372f19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bg489\" (UID: \"10789212-2d42-4796-b759-7dd7ac372f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489" Oct 30 17:27:54.626996 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.626928 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6fkq\" (UniqueName: \"kubernetes.io/projected/10789212-2d42-4796-b759-7dd7ac372f19-kube-api-access-t6fkq\") pod \"maas-api-revoke-sub-d2bc93bc-bg489\" (UID: \"10789212-2d42-4796-b759-7dd7ac372f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489" Oct 30 17:27:54.728291 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.728223 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6fkq\" (UniqueName: \"kubernetes.io/projected/10789212-2d42-4796-b759-7dd7ac372f19-kube-api-access-t6fkq\") pod \"maas-api-revoke-sub-d2bc93bc-bg489\" (UID: \"10789212-2d42-4796-b759-7dd7ac372f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489" Oct 30 17:27:54.728442 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.728311 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10789212-2d42-4796-b759-7dd7ac372f19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bg489\" (UID: \"10789212-2d42-4796-b759-7dd7ac372f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489" Oct 30 17:27:54.728442 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.728374 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10789212-2d42-4796-b759-7dd7ac372f19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bg489\" (UID: \"10789212-2d42-4796-b759-7dd7ac372f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489" Oct 30 17:27:54.729007 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.728980 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10789212-2d42-4796-b759-7dd7ac372f19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bg489\" (UID: \"10789212-2d42-4796-b759-7dd7ac372f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489" Oct 30 17:27:54.730811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.730787 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10789212-2d42-4796-b759-7dd7ac372f19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bg489\" (UID: \"10789212-2d42-4796-b759-7dd7ac372f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489" Oct 30 17:27:54.752590 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.752565 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6fkq\" (UniqueName: \"kubernetes.io/projected/10789212-2d42-4796-b759-7dd7ac372f19-kube-api-access-t6fkq\") pod \"maas-api-revoke-sub-d2bc93bc-bg489\" (UID: \"10789212-2d42-4796-b759-7dd7ac372f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489" Oct 30 17:27:54.784564 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:54.784540 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489" Oct 30 17:27:55.076216 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:55.076188 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489"] Oct 30 17:27:55.076845 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:27:55.076818 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10789212_2d42_4796_b759_7dd7ac372f19.slice/crio-7b50d27568ed90ea67f10bd36c6fcaa4e28eecc7c9acb8882eb85ca0c22353e1 WatchSource:0}: Error finding container 7b50d27568ed90ea67f10bd36c6fcaa4e28eecc7c9acb8882eb85ca0c22353e1: Status 404 returned error can't find the container with id 7b50d27568ed90ea67f10bd36c6fcaa4e28eecc7c9acb8882eb85ca0c22353e1 Oct 30 17:27:55.414235 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:55.414198 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489" event={"ID":"10789212-2d42-4796-b759-7dd7ac372f19","Type":"ContainerStarted","Data":"a9c8bcd201aac8a5fbf7fce30a20c779185ac6d83c5f607081ffaa6a19bdddef"} Oct 30 17:27:55.414404 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:55.414267 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489" event={"ID":"10789212-2d42-4796-b759-7dd7ac372f19","Type":"ContainerStarted","Data":"7b50d27568ed90ea67f10bd36c6fcaa4e28eecc7c9acb8882eb85ca0c22353e1"} Oct 30 17:27:55.455958 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:55.455900 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489" podStartSLOduration=1.455888469 podStartE2EDuration="1.455888469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:55.455192342 +0000 UTC m=+3989.165647089" watchObservedRunningTime="2026-10-30 17:27:55.455888469 +0000 UTC m=+3989.166343207" Oct 30 17:27:56.419541 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:56.419505 6202 generic.go:356] "Generic (PLEG): container finished" podID="10789212-2d42-4796-b759-7dd7ac372f19" containerID="a9c8bcd201aac8a5fbf7fce30a20c779185ac6d83c5f607081ffaa6a19bdddef" exitCode=0 Oct 30 17:27:56.419996 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:56.419579 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489" event={"ID":"10789212-2d42-4796-b759-7dd7ac372f19","Type":"ContainerDied","Data":"a9c8bcd201aac8a5fbf7fce30a20c779185ac6d83c5f607081ffaa6a19bdddef"} Oct 30 17:27:57.554469 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:57.554445 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489" Oct 30 17:27:57.652894 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:57.652864 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10789212-2d42-4796-b759-7dd7ac372f19-kube-api-access-t6fkq\" (UniqueName: \"kubernetes.io/projected/10789212-2d42-4796-b759-7dd7ac372f19-kube-api-access-t6fkq\") pod \"10789212-2d42-4796-b759-7dd7ac372f19\" (UID: \"10789212-2d42-4796-b759-7dd7ac372f19\") " Oct 30 17:27:57.653074 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:57.652936 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10789212-2d42-4796-b759-7dd7ac372f19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10789212-2d42-4796-b759-7dd7ac372f19-maas-api-xks-ca\") pod \"10789212-2d42-4796-b759-7dd7ac372f19\" (UID: \"10789212-2d42-4796-b759-7dd7ac372f19\") " Oct 30 17:27:57.653074 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:57.652997 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10789212-2d42-4796-b759-7dd7ac372f19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10789212-2d42-4796-b759-7dd7ac372f19-maas-api-openshift-ca\") pod \"10789212-2d42-4796-b759-7dd7ac372f19\" (UID: \"10789212-2d42-4796-b759-7dd7ac372f19\") " Oct 30 17:27:57.653389 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:57.653355 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10789212-2d42-4796-b759-7dd7ac372f19-maas-api-openshift-ca" pod "10789212-2d42-4796-b759-7dd7ac372f19" (UID: "10789212-2d42-4796-b759-7dd7ac372f19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:57.655202 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:57.655177 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10789212-2d42-4796-b759-7dd7ac372f19-maas-api-xks-ca" pod "10789212-2d42-4796-b759-7dd7ac372f19" (UID: "10789212-2d42-4796-b759-7dd7ac372f19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:57.655202 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:57.655191 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10789212-2d42-4796-b759-7dd7ac372f19-kube-api-access-t6fkq" pod "10789212-2d42-4796-b759-7dd7ac372f19" (UID: "10789212-2d42-4796-b759-7dd7ac372f19"). InnerVolumeSpecName "kube-api-access-t6fkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:57.753703 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:57.753622 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6fkq\" (UniqueName: \"kubernetes.io/projected/10789212-2d42-4796-b759-7dd7ac372f19-kube-api-access-t6fkq\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:27:57.753703 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:57.753653 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10789212-2d42-4796-b759-7dd7ac372f19-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:27:57.753703 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:57.753662 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10789212-2d42-4796-b759-7dd7ac372f19-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:27:58.428779 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:58.428738 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489" Oct 30 17:27:58.428779 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:58.428755 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489" event={"ID":"10789212-2d42-4796-b759-7dd7ac372f19","Type":"ContainerDied","Data":"7b50d27568ed90ea67f10bd36c6fcaa4e28eecc7c9acb8882eb85ca0c22353e1"} Oct 30 17:27:58.428959 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:27:58.428805 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b50d27568ed90ea67f10bd36c6fcaa4e28eecc7c9acb8882eb85ca0c22353e1" Oct 30 17:28:14.066842 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:28:14.066801 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:28:26.237194 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:26.237156 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:27.822755 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:27.822719 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:30.451383 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:30.451334 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:30.566637 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:30.566597 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:31.941097 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:31.941069 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:32.059562 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:32.059525 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:32.177181 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:32.177149 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:33.575311 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:33.575278 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:35.276295 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:35.276257 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:36.882050 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:36.882012 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:38.667476 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:38.667444 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:40.258887 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:40.258853 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:41.972187 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:41.972149 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:43.694294 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:43.694262 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:44.067624 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:28:44.067539 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:28:44.334059 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:44.334025 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:45.403125 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:45.403086 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:45.525496 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:45.525462 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:46.101381 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:46.101346 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:46.233572 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:46.233535 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:47.117462 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:47.117408 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:47.630884 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:47.630854 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:48.910034 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:48.909997 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:50.590832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:50.590793 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:52.426870 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:52.426839 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:54.054445 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:54.054407 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:55.585277 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:55.585239 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:57.188643 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:57.188607 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:58.753988 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:28:58.753953 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:00.433518 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:00.433480 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:02.073389 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:02.073354 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:03.628121 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:03.628080 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:05.296369 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:05.296334 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:06.965013 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:06.964978 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:07.160365 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.160330 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7"] Oct 30 17:29:07.160857 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.160838 6202 cpu_manager.go:414] "containerMap: removing container" podUID="10789212-2d42-4796-b759-7dd7ac372f19" containerName="revoke-keys" Oct 30 17:29:07.160857 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.160858 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10789212-2d42-4796-b759-7dd7ac372f19" containerName="revoke-keys" Oct 30 17:29:07.160974 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.160944 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="10789212-2d42-4796-b759-7dd7ac372f19" containerName="revoke-keys" Oct 30 17:29:07.163753 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.163738 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7" Oct 30 17:29:07.180880 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.180820 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:29:07.180880 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.180858 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:29:07.180880 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.180861 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:29:07.181072 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.180990 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:29:07.206757 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.206725 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7"] Oct 30 17:29:07.211153 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.211131 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zksx8\" (UniqueName: \"kubernetes.io/projected/0f563d4f-8cb0-403a-a084-a7c57d16d796-kube-api-access-zksx8\") pod \"maas-api-revoke-sub-175680ee-hlpk7\" (UID: \"0f563d4f-8cb0-403a-a084-a7c57d16d796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7" Oct 30 17:29:07.211320 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.211164 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f563d4f-8cb0-403a-a084-a7c57d16d796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-175680ee-hlpk7\" (UID: \"0f563d4f-8cb0-403a-a084-a7c57d16d796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7" Oct 30 17:29:07.211320 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.211264 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f563d4f-8cb0-403a-a084-a7c57d16d796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-175680ee-hlpk7\" (UID: \"0f563d4f-8cb0-403a-a084-a7c57d16d796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7" Oct 30 17:29:07.312541 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.312455 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f563d4f-8cb0-403a-a084-a7c57d16d796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-175680ee-hlpk7\" (UID: \"0f563d4f-8cb0-403a-a084-a7c57d16d796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7" Oct 30 17:29:07.312541 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.312513 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zksx8\" (UniqueName: \"kubernetes.io/projected/0f563d4f-8cb0-403a-a084-a7c57d16d796-kube-api-access-zksx8\") pod \"maas-api-revoke-sub-175680ee-hlpk7\" (UID: \"0f563d4f-8cb0-403a-a084-a7c57d16d796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7" Oct 30 17:29:07.312541 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.312538 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f563d4f-8cb0-403a-a084-a7c57d16d796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-175680ee-hlpk7\" (UID: \"0f563d4f-8cb0-403a-a084-a7c57d16d796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7" Oct 30 17:29:07.313290 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.313269 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f563d4f-8cb0-403a-a084-a7c57d16d796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-175680ee-hlpk7\" (UID: \"0f563d4f-8cb0-403a-a084-a7c57d16d796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7" Oct 30 17:29:07.314942 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.314923 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f563d4f-8cb0-403a-a084-a7c57d16d796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-175680ee-hlpk7\" (UID: \"0f563d4f-8cb0-403a-a084-a7c57d16d796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7" Oct 30 17:29:07.336914 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.336883 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zksx8\" (UniqueName: \"kubernetes.io/projected/0f563d4f-8cb0-403a-a084-a7c57d16d796-kube-api-access-zksx8\") pod \"maas-api-revoke-sub-175680ee-hlpk7\" (UID: \"0f563d4f-8cb0-403a-a084-a7c57d16d796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7" Oct 30 17:29:07.475442 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.475413 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7" Oct 30 17:29:07.770756 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:07.769898 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7"] Oct 30 17:29:08.553139 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:08.553099 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:08.741371 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:08.741285 6202 generic.go:356] "Generic (PLEG): container finished" podID="0f563d4f-8cb0-403a-a084-a7c57d16d796" containerID="43a296a4ae1b86b70d07018f0d97cfc44184d86c4ee3a040d887c2fd7da81753" exitCode=0 Oct 30 17:29:08.741533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:08.741376 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7" event={"ID":"0f563d4f-8cb0-403a-a084-a7c57d16d796","Type":"ContainerDied","Data":"43a296a4ae1b86b70d07018f0d97cfc44184d86c4ee3a040d887c2fd7da81753"} Oct 30 17:29:08.741533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:08.741430 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7" event={"ID":"0f563d4f-8cb0-403a-a084-a7c57d16d796","Type":"ContainerStarted","Data":"2d42a603e50b16cb1e9a9a61b0a0ee2b77dd820ec116aa1e5ad30ffc958317c9"} Oct 30 17:29:09.878706 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:09.878680 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7" Oct 30 17:29:09.936386 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:09.936357 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f563d4f-8cb0-403a-a084-a7c57d16d796-kube-api-access-zksx8\" (UniqueName: \"kubernetes.io/projected/0f563d4f-8cb0-403a-a084-a7c57d16d796-kube-api-access-zksx8\") pod \"0f563d4f-8cb0-403a-a084-a7c57d16d796\" (UID: \"0f563d4f-8cb0-403a-a084-a7c57d16d796\") " Oct 30 17:29:09.936542 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:09.936422 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f563d4f-8cb0-403a-a084-a7c57d16d796-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f563d4f-8cb0-403a-a084-a7c57d16d796-maas-api-xks-ca\") pod \"0f563d4f-8cb0-403a-a084-a7c57d16d796\" (UID: \"0f563d4f-8cb0-403a-a084-a7c57d16d796\") " Oct 30 17:29:09.936542 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:09.936455 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f563d4f-8cb0-403a-a084-a7c57d16d796-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f563d4f-8cb0-403a-a084-a7c57d16d796-maas-api-openshift-ca\") pod \"0f563d4f-8cb0-403a-a084-a7c57d16d796\" (UID: \"0f563d4f-8cb0-403a-a084-a7c57d16d796\") " Oct 30 17:29:09.936891 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:09.936862 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f563d4f-8cb0-403a-a084-a7c57d16d796-maas-api-openshift-ca" pod "0f563d4f-8cb0-403a-a084-a7c57d16d796" (UID: "0f563d4f-8cb0-403a-a084-a7c57d16d796"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:09.938589 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:09.938562 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f563d4f-8cb0-403a-a084-a7c57d16d796-kube-api-access-zksx8" pod "0f563d4f-8cb0-403a-a084-a7c57d16d796" (UID: "0f563d4f-8cb0-403a-a084-a7c57d16d796"). InnerVolumeSpecName "kube-api-access-zksx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:09.938589 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:09.938575 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f563d4f-8cb0-403a-a084-a7c57d16d796-maas-api-xks-ca" pod "0f563d4f-8cb0-403a-a084-a7c57d16d796" (UID: "0f563d4f-8cb0-403a-a084-a7c57d16d796"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:29:10.038163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:10.038103 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zksx8\" (UniqueName: \"kubernetes.io/projected/0f563d4f-8cb0-403a-a084-a7c57d16d796-kube-api-access-zksx8\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:29:10.038163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:10.038125 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f563d4f-8cb0-403a-a084-a7c57d16d796-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:29:10.038163 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:10.038136 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f563d4f-8cb0-403a-a084-a7c57d16d796-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:29:10.240396 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:10.240361 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:10.753349 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:10.753273 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7" Oct 30 17:29:10.753349 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:10.753315 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7" event={"ID":"0f563d4f-8cb0-403a-a084-a7c57d16d796","Type":"ContainerDied","Data":"2d42a603e50b16cb1e9a9a61b0a0ee2b77dd820ec116aa1e5ad30ffc958317c9"} Oct 30 17:29:10.753599 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:10.753362 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d42a603e50b16cb1e9a9a61b0a0ee2b77dd820ec116aa1e5ad30ffc958317c9" Oct 30 17:29:11.763356 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:11.763321 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:13.279408 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:13.279377 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:14.066843 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:29:14.066806 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:29:15.049033 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:15.049001 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:16.625499 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:16.625460 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:18.103030 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:18.102998 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:18.271782 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:18.271741 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:19.990661 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:19.990627 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:20.984082 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:20.984046 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:21.110576 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:21.110546 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:21.511258 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:21.511219 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:23.117994 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:23.117948 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:23.231561 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:23.231521 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:24.658974 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:24.658940 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:26.209097 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:26.209055 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:27.862259 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:27.862229 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:28.246718 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.246624 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p"] Oct 30 17:29:28.247245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.247222 6202 cpu_manager.go:414] "containerMap: removing container" podUID="0f563d4f-8cb0-403a-a084-a7c57d16d796" containerName="revoke-keys" Oct 30 17:29:28.247354 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.247246 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f563d4f-8cb0-403a-a084-a7c57d16d796" containerName="revoke-keys" Oct 30 17:29:28.247419 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.247362 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f563d4f-8cb0-403a-a084-a7c57d16d796" containerName="revoke-keys" Oct 30 17:29:28.250832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.250810 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p" Oct 30 17:29:28.269322 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.269294 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:29:28.269436 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.269325 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:29:28.283569 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.283546 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p"] Oct 30 17:29:28.334385 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.334360 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:29:28.334518 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.334384 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:29:28.420746 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.420718 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672389be-47a5-4683-8df9-08953df7ce4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p\" (UID: \"672389be-47a5-4683-8df9-08953df7ce4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p" Oct 30 17:29:28.420900 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.420833 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672389be-47a5-4683-8df9-08953df7ce4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p\" (UID: \"672389be-47a5-4683-8df9-08953df7ce4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p" Oct 30 17:29:28.420946 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.420910 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcpks\" (UniqueName: \"kubernetes.io/projected/672389be-47a5-4683-8df9-08953df7ce4f-kube-api-access-jcpks\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p\" (UID: \"672389be-47a5-4683-8df9-08953df7ce4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p" Oct 30 17:29:28.521500 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.521427 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcpks\" (UniqueName: \"kubernetes.io/projected/672389be-47a5-4683-8df9-08953df7ce4f-kube-api-access-jcpks\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p\" (UID: \"672389be-47a5-4683-8df9-08953df7ce4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p" Oct 30 17:29:28.521644 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.521495 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672389be-47a5-4683-8df9-08953df7ce4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p\" (UID: \"672389be-47a5-4683-8df9-08953df7ce4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p" Oct 30 17:29:28.521644 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.521559 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672389be-47a5-4683-8df9-08953df7ce4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p\" (UID: \"672389be-47a5-4683-8df9-08953df7ce4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p" Oct 30 17:29:28.522185 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.522164 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672389be-47a5-4683-8df9-08953df7ce4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p\" (UID: \"672389be-47a5-4683-8df9-08953df7ce4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p" Oct 30 17:29:28.524034 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.524012 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672389be-47a5-4683-8df9-08953df7ce4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p\" (UID: \"672389be-47a5-4683-8df9-08953df7ce4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p" Oct 30 17:29:28.545711 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.545688 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcpks\" (UniqueName: \"kubernetes.io/projected/672389be-47a5-4683-8df9-08953df7ce4f-kube-api-access-jcpks\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p\" (UID: \"672389be-47a5-4683-8df9-08953df7ce4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p" Oct 30 17:29:28.563427 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.563404 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p" Oct 30 17:29:28.857679 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:28.857575 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p"] Oct 30 17:29:28.858168 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:29:28.858138 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672389be_47a5_4683_8df9_08953df7ce4f.slice/crio-e3c06e07c377eeffb5aa6ffd3d2d9546e3ffc47a8d95f39ea99451c2832b3fc5 WatchSource:0}: Error finding container e3c06e07c377eeffb5aa6ffd3d2d9546e3ffc47a8d95f39ea99451c2832b3fc5: Status 404 returned error can't find the container with id e3c06e07c377eeffb5aa6ffd3d2d9546e3ffc47a8d95f39ea99451c2832b3fc5 Oct 30 17:29:29.495679 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:29.495600 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:29.837583 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:29.837551 6202 generic.go:356] "Generic (PLEG): container finished" podID="672389be-47a5-4683-8df9-08953df7ce4f" containerID="4fce91254af1a03a1092e90cbd3e1c480d2cf9b5527e96ec38023384bfe34bd9" exitCode=0 Oct 30 17:29:29.837793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:29.837603 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p" event={"ID":"672389be-47a5-4683-8df9-08953df7ce4f","Type":"ContainerDied","Data":"4fce91254af1a03a1092e90cbd3e1c480d2cf9b5527e96ec38023384bfe34bd9"} Oct 30 17:29:29.837793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:29.837639 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p" event={"ID":"672389be-47a5-4683-8df9-08953df7ce4f","Type":"ContainerStarted","Data":"e3c06e07c377eeffb5aa6ffd3d2d9546e3ffc47a8d95f39ea99451c2832b3fc5"} Oct 30 17:29:30.965920 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:30.965888 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p" Oct 30 17:29:31.043823 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:31.043794 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/672389be-47a5-4683-8df9-08953df7ce4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672389be-47a5-4683-8df9-08953df7ce4f-maas-api-openshift-ca\") pod \"672389be-47a5-4683-8df9-08953df7ce4f\" (UID: \"672389be-47a5-4683-8df9-08953df7ce4f\") " Oct 30 17:29:31.044054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:31.043843 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/672389be-47a5-4683-8df9-08953df7ce4f-kube-api-access-jcpks\" (UniqueName: \"kubernetes.io/projected/672389be-47a5-4683-8df9-08953df7ce4f-kube-api-access-jcpks\") pod \"672389be-47a5-4683-8df9-08953df7ce4f\" (UID: \"672389be-47a5-4683-8df9-08953df7ce4f\") " Oct 30 17:29:31.044054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:31.043940 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/672389be-47a5-4683-8df9-08953df7ce4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672389be-47a5-4683-8df9-08953df7ce4f-maas-api-xks-ca\") pod \"672389be-47a5-4683-8df9-08953df7ce4f\" (UID: \"672389be-47a5-4683-8df9-08953df7ce4f\") " Oct 30 17:29:31.044201 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:31.044173 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/672389be-47a5-4683-8df9-08953df7ce4f-maas-api-openshift-ca" pod "672389be-47a5-4683-8df9-08953df7ce4f" (UID: "672389be-47a5-4683-8df9-08953df7ce4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:31.044728 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:31.044703 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:31.046079 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:31.046056 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/672389be-47a5-4683-8df9-08953df7ce4f-maas-api-xks-ca" pod "672389be-47a5-4683-8df9-08953df7ce4f" (UID: "672389be-47a5-4683-8df9-08953df7ce4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:29:31.046175 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:31.046139 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672389be-47a5-4683-8df9-08953df7ce4f-kube-api-access-jcpks" pod "672389be-47a5-4683-8df9-08953df7ce4f" (UID: "672389be-47a5-4683-8df9-08953df7ce4f"). InnerVolumeSpecName "kube-api-access-jcpks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:31.144994 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:31.144913 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672389be-47a5-4683-8df9-08953df7ce4f-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:29:31.144994 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:31.144942 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcpks\" (UniqueName: \"kubernetes.io/projected/672389be-47a5-4683-8df9-08953df7ce4f-kube-api-access-jcpks\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:29:31.144994 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:31.144954 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672389be-47a5-4683-8df9-08953df7ce4f-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:29:31.848306 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:31.848263 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p" event={"ID":"672389be-47a5-4683-8df9-08953df7ce4f","Type":"ContainerDied","Data":"e3c06e07c377eeffb5aa6ffd3d2d9546e3ffc47a8d95f39ea99451c2832b3fc5"} Oct 30 17:29:31.848306 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:31.848284 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p" Oct 30 17:29:31.848533 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:31.848319 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3c06e07c377eeffb5aa6ffd3d2d9546e3ffc47a8d95f39ea99451c2832b3fc5" Oct 30 17:29:32.597056 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:32.597018 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:34.250101 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:34.250068 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:35.859680 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:35.859630 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:37.445197 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:37.445164 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:38.773679 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:38.773636 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:38.886930 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:38.886898 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:39.052431 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:39.052336 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:40.319328 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:40.319296 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:40.745228 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:40.745188 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:40.860179 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:40.860147 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p"] Oct 30 17:29:40.867114 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:40.867085 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:40.876887 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:40.873826 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e2e8d2-jmq2p"] Oct 30 17:29:41.560613 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:41.560574 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:42.408679 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:42.408644 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="672389be-47a5-4683-8df9-08953df7ce4f" path="/var/lib/kubelet/pods/672389be-47a5-4683-8df9-08953df7ce4f/volumes" Oct 30 17:29:44.066718 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:29:44.066678 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:29:44.439390 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:44.439346 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:47.592059 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:47.592021 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:47.706511 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:47.706475 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:48.012369 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:48.012273 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:53.843573 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:53.843526 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:59.584991 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:29:59.584955 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:00.140327 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:00.140287 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96"] Oct 30 17:30:00.140759 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:00.140740 6202 cpu_manager.go:414] "containerMap: removing container" podUID="672389be-47a5-4683-8df9-08953df7ce4f" containerName="revoke-keys" Oct 30 17:30:00.140878 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:00.140760 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="672389be-47a5-4683-8df9-08953df7ce4f" containerName="revoke-keys" Oct 30 17:30:00.140947 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:00.140877 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="672389be-47a5-4683-8df9-08953df7ce4f" containerName="revoke-keys" Oct 30 17:30:00.144356 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:00.144336 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" Oct 30 17:30:00.161691 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:00.161667 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:30:00.163067 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:00.163043 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96"] Oct 30 17:30:00.174743 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:00.174720 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:30:00.174869 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:00.174843 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:30:00.200169 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:00.200140 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/225f5c3b-1476-4047-99e5-8f0ef1b2403a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889690-ztt96\" (UID: \"225f5c3b-1476-4047-99e5-8f0ef1b2403a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" Oct 30 17:30:00.200279 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:00.200187 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw6vd\" (UniqueName: \"kubernetes.io/projected/225f5c3b-1476-4047-99e5-8f0ef1b2403a-kube-api-access-lw6vd\") pod \"maas-api-key-cleanup-29889690-ztt96\" (UID: \"225f5c3b-1476-4047-99e5-8f0ef1b2403a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" Oct 30 17:30:00.301174 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:00.301147 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/225f5c3b-1476-4047-99e5-8f0ef1b2403a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889690-ztt96\" (UID: \"225f5c3b-1476-4047-99e5-8f0ef1b2403a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" Oct 30 17:30:00.301323 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:00.301194 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw6vd\" (UniqueName: \"kubernetes.io/projected/225f5c3b-1476-4047-99e5-8f0ef1b2403a-kube-api-access-lw6vd\") pod \"maas-api-key-cleanup-29889690-ztt96\" (UID: \"225f5c3b-1476-4047-99e5-8f0ef1b2403a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" Oct 30 17:30:00.301752 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:00.301731 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/225f5c3b-1476-4047-99e5-8f0ef1b2403a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889690-ztt96\" (UID: \"225f5c3b-1476-4047-99e5-8f0ef1b2403a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" Oct 30 17:30:00.325313 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:00.325283 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw6vd\" (UniqueName: \"kubernetes.io/projected/225f5c3b-1476-4047-99e5-8f0ef1b2403a-kube-api-access-lw6vd\") pod \"maas-api-key-cleanup-29889690-ztt96\" (UID: \"225f5c3b-1476-4047-99e5-8f0ef1b2403a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" Oct 30 17:30:00.456692 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:00.456623 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" Oct 30 17:30:00.746806 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:00.746758 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96"] Oct 30 17:30:00.747739 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:30:00.747712 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225f5c3b_1476_4047_99e5_8f0ef1b2403a.slice/crio-67c8dc5d2c9114594bee023aefdb9376de49cc8fc1309f9d437ec516be45ea0f WatchSource:0}: Error finding container 67c8dc5d2c9114594bee023aefdb9376de49cc8fc1309f9d437ec516be45ea0f: Status 404 returned error can't find the container with id 67c8dc5d2c9114594bee023aefdb9376de49cc8fc1309f9d437ec516be45ea0f Oct 30 17:30:00.809046 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:00.809016 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:01.010729 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:01.010657 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" event={"ID":"225f5c3b-1476-4047-99e5-8f0ef1b2403a","Type":"ContainerStarted","Data":"67c8dc5d2c9114594bee023aefdb9376de49cc8fc1309f9d437ec516be45ea0f"} Oct 30 17:30:02.016643 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:02.016554 6202 generic.go:356] "Generic (PLEG): container finished" podID="225f5c3b-1476-4047-99e5-8f0ef1b2403a" containerID="a5c9e6c816538afff37550162d6ad448fcbd50aa8e7c19be13bc8e31712bb3c9" exitCode=60 Oct 30 17:30:02.017064 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:02.016646 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" event={"ID":"225f5c3b-1476-4047-99e5-8f0ef1b2403a","Type":"ContainerDied","Data":"a5c9e6c816538afff37550162d6ad448fcbd50aa8e7c19be13bc8e31712bb3c9"} Oct 30 17:30:02.017064 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:02.016848 6202 scope.go:122] "RemoveContainer" containerID="a5c9e6c816538afff37550162d6ad448fcbd50aa8e7c19be13bc8e31712bb3c9" Oct 30 17:30:03.022427 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:03.022393 6202 generic.go:356] "Generic (PLEG): container finished" podID="225f5c3b-1476-4047-99e5-8f0ef1b2403a" containerID="32755f6ed5c382bb93acf28ab56f78913e93612099416839f4f64fab3ce48361" exitCode=60 Oct 30 17:30:03.022918 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:03.022484 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" event={"ID":"225f5c3b-1476-4047-99e5-8f0ef1b2403a","Type":"ContainerDied","Data":"32755f6ed5c382bb93acf28ab56f78913e93612099416839f4f64fab3ce48361"} Oct 30 17:30:03.022918 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:03.022541 6202 scope.go:122] "RemoveContainer" containerID="a5c9e6c816538afff37550162d6ad448fcbd50aa8e7c19be13bc8e31712bb3c9" Oct 30 17:30:03.022918 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:03.022871 6202 scope.go:122] "RemoveContainer" containerID="32755f6ed5c382bb93acf28ab56f78913e93612099416839f4f64fab3ce48361" Oct 30 17:30:03.023183 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:30:03.023149 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889690-ztt96_odh-ai-gateway-infra(225f5c3b-1476-4047-99e5-8f0ef1b2403a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" podUID="225f5c3b-1476-4047-99e5-8f0ef1b2403a" Oct 30 17:30:04.029103 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:04.029076 6202 scope.go:122] "RemoveContainer" containerID="32755f6ed5c382bb93acf28ab56f78913e93612099416839f4f64fab3ce48361" Oct 30 17:30:04.029498 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:30:04.029301 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889690-ztt96_odh-ai-gateway-infra(225f5c3b-1476-4047-99e5-8f0ef1b2403a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" podUID="225f5c3b-1476-4047-99e5-8f0ef1b2403a" Oct 30 17:30:04.727226 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:04.727190 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:04.850277 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:04.850240 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:06.662972 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:06.662936 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:14.068607 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:30:14.068547 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:30:14.103426 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:14.103383 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:14.403532 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:14.403504 6202 scope.go:122] "RemoveContainer" containerID="32755f6ed5c382bb93acf28ab56f78913e93612099416839f4f64fab3ce48361" Oct 30 17:30:15.079721 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:15.079687 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" event={"ID":"225f5c3b-1476-4047-99e5-8f0ef1b2403a","Type":"ContainerStarted","Data":"5161c21c7e83b7196b2866be7478608157b2e30a64118d3b2160af112e5e06c4"} Oct 30 17:30:15.122912 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:15.122863 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" podStartSLOduration=15.122848819 podStartE2EDuration="15.122848819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:15.121206222 +0000 UTC m=+4128.831660968" watchObservedRunningTime="2026-10-30 17:30:15.122848819 +0000 UTC m=+4128.833303565" Oct 30 17:30:16.086138 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:16.086102 6202 generic.go:356] "Generic (PLEG): container finished" podID="225f5c3b-1476-4047-99e5-8f0ef1b2403a" containerID="5161c21c7e83b7196b2866be7478608157b2e30a64118d3b2160af112e5e06c4" exitCode=60 Oct 30 17:30:16.086631 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:16.086172 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" event={"ID":"225f5c3b-1476-4047-99e5-8f0ef1b2403a","Type":"ContainerDied","Data":"5161c21c7e83b7196b2866be7478608157b2e30a64118d3b2160af112e5e06c4"} Oct 30 17:30:16.086631 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:16.086245 6202 scope.go:122] "RemoveContainer" containerID="32755f6ed5c382bb93acf28ab56f78913e93612099416839f4f64fab3ce48361" Oct 30 17:30:16.086631 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:16.086558 6202 scope.go:122] "RemoveContainer" containerID="5161c21c7e83b7196b2866be7478608157b2e30a64118d3b2160af112e5e06c4" Oct 30 17:30:16.086874 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:30:16.086851 6202 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889690-ztt96_odh-ai-gateway-infra(225f5c3b-1476-4047-99e5-8f0ef1b2403a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" podUID="225f5c3b-1476-4047-99e5-8f0ef1b2403a" Oct 30 17:30:16.137301 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:16.137266 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96"] Oct 30 17:30:17.243530 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:17.243498 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" Oct 30 17:30:17.378516 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:17.378415 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/225f5c3b-1476-4047-99e5-8f0ef1b2403a-kube-api-access-lw6vd\" (UniqueName: \"kubernetes.io/projected/225f5c3b-1476-4047-99e5-8f0ef1b2403a-kube-api-access-lw6vd\") pod \"225f5c3b-1476-4047-99e5-8f0ef1b2403a\" (UID: \"225f5c3b-1476-4047-99e5-8f0ef1b2403a\") " Oct 30 17:30:17.378705 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:17.378526 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/225f5c3b-1476-4047-99e5-8f0ef1b2403a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/225f5c3b-1476-4047-99e5-8f0ef1b2403a-maas-api-service-ca\") pod \"225f5c3b-1476-4047-99e5-8f0ef1b2403a\" (UID: \"225f5c3b-1476-4047-99e5-8f0ef1b2403a\") " Oct 30 17:30:17.379236 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:17.379193 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225f5c3b-1476-4047-99e5-8f0ef1b2403a-maas-api-service-ca" pod "225f5c3b-1476-4047-99e5-8f0ef1b2403a" (UID: "225f5c3b-1476-4047-99e5-8f0ef1b2403a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:17.381597 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:17.381569 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225f5c3b-1476-4047-99e5-8f0ef1b2403a-kube-api-access-lw6vd" pod "225f5c3b-1476-4047-99e5-8f0ef1b2403a" (UID: "225f5c3b-1476-4047-99e5-8f0ef1b2403a"). InnerVolumeSpecName "kube-api-access-lw6vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:17.479628 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:17.479576 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/225f5c3b-1476-4047-99e5-8f0ef1b2403a-maas-api-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:30:17.479628 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:17.479618 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw6vd\" (UniqueName: \"kubernetes.io/projected/225f5c3b-1476-4047-99e5-8f0ef1b2403a-kube-api-access-lw6vd\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:30:18.112791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:18.110165 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" event={"ID":"225f5c3b-1476-4047-99e5-8f0ef1b2403a","Type":"ContainerDied","Data":"67c8dc5d2c9114594bee023aefdb9376de49cc8fc1309f9d437ec516be45ea0f"} Oct 30 17:30:18.112791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:18.110240 6202 scope.go:122] "RemoveContainer" containerID="5161c21c7e83b7196b2866be7478608157b2e30a64118d3b2160af112e5e06c4" Oct 30 17:30:18.112791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:18.110353 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96" Oct 30 17:30:18.182800 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:18.182158 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96"] Oct 30 17:30:18.198203 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:18.198131 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-ztt96"] Oct 30 17:30:18.414839 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:18.414352 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="225f5c3b-1476-4047-99e5-8f0ef1b2403a" path="/var/lib/kubelet/pods/225f5c3b-1476-4047-99e5-8f0ef1b2403a/volumes" Oct 30 17:30:18.699844 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:18.699371 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:19.108793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:19.107875 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-ffkqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:30:19.108793 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:19.107950 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" podUID="27b9c1c4-c3da-4caf-a594-b391b48cc93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:30:19.658791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:19.652884 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5mcb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:30:19.658791 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:19.652956 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" podUID="04a034df-8dc0-489b-a623-4bb9e730e6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:30:34.108544 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:34.107950 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-ffkqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:30:34.108544 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:34.108033 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" podUID="27b9c1c4-c3da-4caf-a594-b391b48cc93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:30:34.644792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:34.641873 6202 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5mcb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:30:34.644792 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:34.641995 6202 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" podUID="04a034df-8dc0-489b-a623-4bb9e730e6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:30:35.967503 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:35.967471 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz"] Oct 30 17:30:35.967948 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:35.967875 6202 cpu_manager.go:414] "containerMap: removing container" podUID="225f5c3b-1476-4047-99e5-8f0ef1b2403a" containerName="cleanup" Oct 30 17:30:35.967948 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:35.967888 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="225f5c3b-1476-4047-99e5-8f0ef1b2403a" containerName="cleanup" Oct 30 17:30:35.967948 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:35.967897 6202 cpu_manager.go:414] "containerMap: removing container" podUID="225f5c3b-1476-4047-99e5-8f0ef1b2403a" containerName="cleanup" Oct 30 17:30:35.967948 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:35.967902 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="225f5c3b-1476-4047-99e5-8f0ef1b2403a" containerName="cleanup" Oct 30 17:30:35.968102 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:35.967978 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="225f5c3b-1476-4047-99e5-8f0ef1b2403a" containerName="cleanup" Oct 30 17:30:35.968102 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:35.967986 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="225f5c3b-1476-4047-99e5-8f0ef1b2403a" containerName="cleanup" Oct 30 17:30:35.970608 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:35.970591 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz" Oct 30 17:30:35.986912 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:35.986888 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:30:35.999517 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:35.999493 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:30:36.001166 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:36.001124 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz"] Oct 30 17:30:36.040558 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:36.040535 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:30:36.040696 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:36.040644 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:30:36.110783 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:36.110744 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145ed570-31e7-420a-bafa-eff4e2b08ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-jvcxz\" (UID: \"145ed570-31e7-420a-bafa-eff4e2b08ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz" Oct 30 17:30:36.110955 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:36.110817 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145ed570-31e7-420a-bafa-eff4e2b08ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-jvcxz\" (UID: \"145ed570-31e7-420a-bafa-eff4e2b08ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz" Oct 30 17:30:36.110955 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:36.110924 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9xkb\" (UniqueName: \"kubernetes.io/projected/145ed570-31e7-420a-bafa-eff4e2b08ab1-kube-api-access-c9xkb\") pod \"maas-api-revoke-sub-452b6d7b-jvcxz\" (UID: \"145ed570-31e7-420a-bafa-eff4e2b08ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz" Oct 30 17:30:36.212716 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:36.212449 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9xkb\" (UniqueName: \"kubernetes.io/projected/145ed570-31e7-420a-bafa-eff4e2b08ab1-kube-api-access-c9xkb\") pod \"maas-api-revoke-sub-452b6d7b-jvcxz\" (UID: \"145ed570-31e7-420a-bafa-eff4e2b08ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz" Oct 30 17:30:36.212716 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:36.212585 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145ed570-31e7-420a-bafa-eff4e2b08ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-jvcxz\" (UID: \"145ed570-31e7-420a-bafa-eff4e2b08ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz" Oct 30 17:30:36.212716 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:36.212631 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145ed570-31e7-420a-bafa-eff4e2b08ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-jvcxz\" (UID: \"145ed570-31e7-420a-bafa-eff4e2b08ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz" Oct 30 17:30:36.219721 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:36.213567 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145ed570-31e7-420a-bafa-eff4e2b08ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-jvcxz\" (UID: \"145ed570-31e7-420a-bafa-eff4e2b08ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz" Oct 30 17:30:36.219721 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:36.216638 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145ed570-31e7-420a-bafa-eff4e2b08ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-jvcxz\" (UID: \"145ed570-31e7-420a-bafa-eff4e2b08ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz" Oct 30 17:30:36.238121 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:36.238090 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9xkb\" (UniqueName: \"kubernetes.io/projected/145ed570-31e7-420a-bafa-eff4e2b08ab1-kube-api-access-c9xkb\") pod \"maas-api-revoke-sub-452b6d7b-jvcxz\" (UID: \"145ed570-31e7-420a-bafa-eff4e2b08ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz" Oct 30 17:30:36.281747 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:36.281711 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz" Oct 30 17:30:36.573398 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:36.573364 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz"] Oct 30 17:30:36.575485 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:30:36.575453 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145ed570_31e7_420a_bafa_eff4e2b08ab1.slice/crio-bc7288f307764b066d518c7afa3c609c12a1db7efa0db6253e783c9755afef06 WatchSource:0}: Error finding container bc7288f307764b066d518c7afa3c609c12a1db7efa0db6253e783c9755afef06: Status 404 returned error can't find the container with id bc7288f307764b066d518c7afa3c609c12a1db7efa0db6253e783c9755afef06 Oct 30 17:30:37.226048 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:37.226009 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz" event={"ID":"145ed570-31e7-420a-bafa-eff4e2b08ab1","Type":"ContainerStarted","Data":"ba0aa73165d070b96e7694e5a5f250cf58382fdce15027b142c1fddaaad3d462"} Oct 30 17:30:37.226534 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:37.226061 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz" event={"ID":"145ed570-31e7-420a-bafa-eff4e2b08ab1","Type":"ContainerStarted","Data":"bc7288f307764b066d518c7afa3c609c12a1db7efa0db6253e783c9755afef06"} Oct 30 17:30:37.270945 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:37.270902 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz" podStartSLOduration=2.270887179 podStartE2EDuration="2.270887179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:37.269287188 +0000 UTC m=+4150.979741930" watchObservedRunningTime="2026-10-30 17:30:37.270887179 +0000 UTC m=+4150.981342023" Oct 30 17:30:38.231702 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:38.231663 6202 generic.go:356] "Generic (PLEG): container finished" podID="145ed570-31e7-420a-bafa-eff4e2b08ab1" containerID="ba0aa73165d070b96e7694e5a5f250cf58382fdce15027b142c1fddaaad3d462" exitCode=0 Oct 30 17:30:38.232196 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:38.231744 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz" event={"ID":"145ed570-31e7-420a-bafa-eff4e2b08ab1","Type":"ContainerDied","Data":"ba0aa73165d070b96e7694e5a5f250cf58382fdce15027b142c1fddaaad3d462"} Oct 30 17:30:39.366002 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:39.365976 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz" Oct 30 17:30:39.446555 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:39.446518 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145ed570-31e7-420a-bafa-eff4e2b08ab1-kube-api-access-c9xkb\" (UniqueName: \"kubernetes.io/projected/145ed570-31e7-420a-bafa-eff4e2b08ab1-kube-api-access-c9xkb\") pod \"145ed570-31e7-420a-bafa-eff4e2b08ab1\" (UID: \"145ed570-31e7-420a-bafa-eff4e2b08ab1\") " Oct 30 17:30:39.446755 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:39.446572 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/145ed570-31e7-420a-bafa-eff4e2b08ab1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145ed570-31e7-420a-bafa-eff4e2b08ab1-maas-api-xks-ca\") pod \"145ed570-31e7-420a-bafa-eff4e2b08ab1\" (UID: \"145ed570-31e7-420a-bafa-eff4e2b08ab1\") " Oct 30 17:30:39.446755 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:39.446686 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/145ed570-31e7-420a-bafa-eff4e2b08ab1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145ed570-31e7-420a-bafa-eff4e2b08ab1-maas-api-openshift-ca\") pod \"145ed570-31e7-420a-bafa-eff4e2b08ab1\" (UID: \"145ed570-31e7-420a-bafa-eff4e2b08ab1\") " Oct 30 17:30:39.447138 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:39.447112 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145ed570-31e7-420a-bafa-eff4e2b08ab1-maas-api-openshift-ca" pod "145ed570-31e7-420a-bafa-eff4e2b08ab1" (UID: "145ed570-31e7-420a-bafa-eff4e2b08ab1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:39.448739 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:39.448718 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145ed570-31e7-420a-bafa-eff4e2b08ab1-maas-api-xks-ca" pod "145ed570-31e7-420a-bafa-eff4e2b08ab1" (UID: "145ed570-31e7-420a-bafa-eff4e2b08ab1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:30:39.448867 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:39.448841 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145ed570-31e7-420a-bafa-eff4e2b08ab1-kube-api-access-c9xkb" pod "145ed570-31e7-420a-bafa-eff4e2b08ab1" (UID: "145ed570-31e7-420a-bafa-eff4e2b08ab1"). InnerVolumeSpecName "kube-api-access-c9xkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:39.548361 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:39.548283 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9xkb\" (UniqueName: \"kubernetes.io/projected/145ed570-31e7-420a-bafa-eff4e2b08ab1-kube-api-access-c9xkb\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:30:39.548361 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:39.548313 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/145ed570-31e7-420a-bafa-eff4e2b08ab1-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:30:39.548361 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:39.548322 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/145ed570-31e7-420a-bafa-eff4e2b08ab1-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:30:40.243032 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:40.243000 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz" Oct 30 17:30:40.243193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:40.242999 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz" event={"ID":"145ed570-31e7-420a-bafa-eff4e2b08ab1","Type":"ContainerDied","Data":"bc7288f307764b066d518c7afa3c609c12a1db7efa0db6253e783c9755afef06"} Oct 30 17:30:40.243193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:40.243127 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc7288f307764b066d518c7afa3c609c12a1db7efa0db6253e783c9755afef06" Oct 30 17:30:44.066463 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:30:44.066422 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:30:44.576010 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:44.575967 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:47.298742 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:47.298705 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:50.877715 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:50.877677 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:54.519001 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.518965 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc"] Oct 30 17:30:54.519417 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.519323 6202 cpu_manager.go:414] "containerMap: removing container" podUID="225f5c3b-1476-4047-99e5-8f0ef1b2403a" containerName="cleanup" Oct 30 17:30:54.519417 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.519333 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="225f5c3b-1476-4047-99e5-8f0ef1b2403a" containerName="cleanup" Oct 30 17:30:54.519417 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.519361 6202 cpu_manager.go:414] "containerMap: removing container" podUID="145ed570-31e7-420a-bafa-eff4e2b08ab1" containerName="revoke-keys" Oct 30 17:30:54.519417 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.519366 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="145ed570-31e7-420a-bafa-eff4e2b08ab1" containerName="revoke-keys" Oct 30 17:30:54.519417 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.519420 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="225f5c3b-1476-4047-99e5-8f0ef1b2403a" containerName="cleanup" Oct 30 17:30:54.519610 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.519434 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="145ed570-31e7-420a-bafa-eff4e2b08ab1" containerName="revoke-keys" Oct 30 17:30:54.521695 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.521674 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc" Oct 30 17:30:54.532254 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.532232 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-7qgkc\" (UID: \"9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc" Oct 30 17:30:54.532352 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.532262 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxldw\" (UniqueName: \"kubernetes.io/projected/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-kube-api-access-mxldw\") pod \"maas-api-revoke-sub-3e31a506-7qgkc\" (UID: \"9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc" Oct 30 17:30:54.532403 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.532363 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-7qgkc\" (UID: \"9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc" Oct 30 17:30:54.547997 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.547970 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:30:54.548116 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.548104 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:30:54.548188 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.548147 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:30:54.548253 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.548224 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:30:54.549647 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.549624 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc"] Oct 30 17:30:54.633418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.633384 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-7qgkc\" (UID: \"9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc" Oct 30 17:30:54.633591 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.633467 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-7qgkc\" (UID: \"9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc" Oct 30 17:30:54.633591 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.633495 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxldw\" (UniqueName: \"kubernetes.io/projected/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-kube-api-access-mxldw\") pod \"maas-api-revoke-sub-3e31a506-7qgkc\" (UID: \"9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc" Oct 30 17:30:54.634024 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.634002 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-7qgkc\" (UID: \"9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc" Oct 30 17:30:54.635934 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.635904 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-7qgkc\" (UID: \"9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc" Oct 30 17:30:54.656326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.656298 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxldw\" (UniqueName: \"kubernetes.io/projected/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-kube-api-access-mxldw\") pod \"maas-api-revoke-sub-3e31a506-7qgkc\" (UID: \"9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc" Oct 30 17:30:54.833129 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:54.833091 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc" Oct 30 17:30:55.152079 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:55.152047 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc"] Oct 30 17:30:55.154164 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:30:55.154138 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7dd8fb_2be3_4c77_bed0_ed7c26ccebc4.slice/crio-dbf6e80d470abccdb133402ff6b2b7728cd67977009e4fe1d3a19573540a5f40 WatchSource:0}: Error finding container dbf6e80d470abccdb133402ff6b2b7728cd67977009e4fe1d3a19573540a5f40: Status 404 returned error can't find the container with id dbf6e80d470abccdb133402ff6b2b7728cd67977009e4fe1d3a19573540a5f40 Oct 30 17:30:55.326942 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:55.326898 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc" event={"ID":"9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4","Type":"ContainerStarted","Data":"dbf6e80d470abccdb133402ff6b2b7728cd67977009e4fe1d3a19573540a5f40"} Oct 30 17:30:56.333388 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:56.333353 6202 generic.go:356] "Generic (PLEG): container finished" podID="9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4" containerID="15fe8a0549f38bf06b424dc6ba8e10ca2bd6f529d0517b5ca573d20a7bcd526c" exitCode=0 Oct 30 17:30:56.333832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:56.333443 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc" event={"ID":"9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4","Type":"ContainerDied","Data":"15fe8a0549f38bf06b424dc6ba8e10ca2bd6f529d0517b5ca573d20a7bcd526c"} Oct 30 17:30:57.470641 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:57.470616 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc" Oct 30 17:30:57.595947 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:57.595869 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-maas-api-openshift-ca\") pod \"9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4\" (UID: \"9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4\") " Oct 30 17:30:57.595947 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:57.595936 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-kube-api-access-mxldw\" (UniqueName: \"kubernetes.io/projected/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-kube-api-access-mxldw\") pod \"9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4\" (UID: \"9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4\") " Oct 30 17:30:57.596142 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:57.595979 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-maas-api-xks-ca\") pod \"9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4\" (UID: \"9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4\") " Oct 30 17:30:57.596256 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:57.596231 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-maas-api-openshift-ca" pod "9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4" (UID: "9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:57.598127 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:57.598096 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-maas-api-xks-ca" pod "9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4" (UID: "9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:30:57.598233 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:57.598195 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-kube-api-access-mxldw" pod "9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4" (UID: "9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4"). InnerVolumeSpecName "kube-api-access-mxldw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:57.696811 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:57.696747 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:30:57.696960 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:57.696814 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxldw\" (UniqueName: \"kubernetes.io/projected/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-kube-api-access-mxldw\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:30:57.696960 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:57.696831 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:30:58.343432 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:58.343397 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc" event={"ID":"9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4","Type":"ContainerDied","Data":"dbf6e80d470abccdb133402ff6b2b7728cd67977009e4fe1d3a19573540a5f40"} Oct 30 17:30:58.343432 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:58.343438 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc" Oct 30 17:30:58.343656 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:30:58.343443 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbf6e80d470abccdb133402ff6b2b7728cd67977009e4fe1d3a19573540a5f40" Oct 30 17:31:14.066874 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:31:14.066831 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:31:19.627513 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.627476 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm"] Oct 30 17:31:19.627964 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.627883 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4" containerName="revoke-keys" Oct 30 17:31:19.627964 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.627894 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4" containerName="revoke-keys" Oct 30 17:31:19.628064 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.627973 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4" containerName="revoke-keys" Oct 30 17:31:19.631311 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.631288 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm" Oct 30 17:31:19.646479 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.646452 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:31:19.657933 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.657880 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:31:19.657933 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.657911 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:31:19.658167 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.657947 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:31:19.660640 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.660616 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm"] Oct 30 17:31:19.765278 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.765240 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc9vp\" (UniqueName: \"kubernetes.io/projected/c93793c1-875c-4c81-a727-e462823b1d35-kube-api-access-kc9vp\") pod \"maas-api-revoke-sub-fef6b280-cb7nm\" (UID: \"c93793c1-875c-4c81-a727-e462823b1d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm" Oct 30 17:31:19.765479 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.765294 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93793c1-875c-4c81-a727-e462823b1d35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef6b280-cb7nm\" (UID: \"c93793c1-875c-4c81-a727-e462823b1d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm" Oct 30 17:31:19.765479 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.765373 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93793c1-875c-4c81-a727-e462823b1d35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef6b280-cb7nm\" (UID: \"c93793c1-875c-4c81-a727-e462823b1d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm" Oct 30 17:31:19.828321 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.828281 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:19.866864 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.866828 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93793c1-875c-4c81-a727-e462823b1d35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef6b280-cb7nm\" (UID: \"c93793c1-875c-4c81-a727-e462823b1d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm" Oct 30 17:31:19.867064 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.866954 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc9vp\" (UniqueName: \"kubernetes.io/projected/c93793c1-875c-4c81-a727-e462823b1d35-kube-api-access-kc9vp\") pod \"maas-api-revoke-sub-fef6b280-cb7nm\" (UID: \"c93793c1-875c-4c81-a727-e462823b1d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm" Oct 30 17:31:19.867064 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.866993 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93793c1-875c-4c81-a727-e462823b1d35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef6b280-cb7nm\" (UID: \"c93793c1-875c-4c81-a727-e462823b1d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm" Oct 30 17:31:19.867593 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.867571 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93793c1-875c-4c81-a727-e462823b1d35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef6b280-cb7nm\" (UID: \"c93793c1-875c-4c81-a727-e462823b1d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm" Oct 30 17:31:19.869464 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.869444 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93793c1-875c-4c81-a727-e462823b1d35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef6b280-cb7nm\" (UID: \"c93793c1-875c-4c81-a727-e462823b1d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm" Oct 30 17:31:19.890271 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.890201 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc9vp\" (UniqueName: \"kubernetes.io/projected/c93793c1-875c-4c81-a727-e462823b1d35-kube-api-access-kc9vp\") pod \"maas-api-revoke-sub-fef6b280-cb7nm\" (UID: \"c93793c1-875c-4c81-a727-e462823b1d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm" Oct 30 17:31:19.944424 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:19.944385 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm" Oct 30 17:31:20.240197 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:20.240162 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm"] Oct 30 17:31:20.242351 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:31:20.242307 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93793c1_875c_4c81_a727_e462823b1d35.slice/crio-d12e7fcf08ce913b11e1088af9a8e0bec2baa2aa910f6eb392b198929c7b26e4 WatchSource:0}: Error finding container d12e7fcf08ce913b11e1088af9a8e0bec2baa2aa910f6eb392b198929c7b26e4: Status 404 returned error can't find the container with id d12e7fcf08ce913b11e1088af9a8e0bec2baa2aa910f6eb392b198929c7b26e4 Oct 30 17:31:20.463355 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:20.463317 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm" event={"ID":"c93793c1-875c-4c81-a727-e462823b1d35","Type":"ContainerStarted","Data":"d12e7fcf08ce913b11e1088af9a8e0bec2baa2aa910f6eb392b198929c7b26e4"} Oct 30 17:31:21.472918 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:21.472879 6202 generic.go:356] "Generic (PLEG): container finished" podID="c93793c1-875c-4c81-a727-e462823b1d35" containerID="6eff159dbb913eb20273a0653e60fa855d17d6f96339a1eb41a8e2701839f7b9" exitCode=0 Oct 30 17:31:21.473378 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:21.472971 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm" event={"ID":"c93793c1-875c-4c81-a727-e462823b1d35","Type":"ContainerDied","Data":"6eff159dbb913eb20273a0653e60fa855d17d6f96339a1eb41a8e2701839f7b9"} Oct 30 17:31:22.618491 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:22.618463 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm" Oct 30 17:31:22.793178 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:22.793081 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c93793c1-875c-4c81-a727-e462823b1d35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93793c1-875c-4c81-a727-e462823b1d35-maas-api-openshift-ca\") pod \"c93793c1-875c-4c81-a727-e462823b1d35\" (UID: \"c93793c1-875c-4c81-a727-e462823b1d35\") " Oct 30 17:31:22.793178 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:22.793156 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c93793c1-875c-4c81-a727-e462823b1d35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93793c1-875c-4c81-a727-e462823b1d35-maas-api-xks-ca\") pod \"c93793c1-875c-4c81-a727-e462823b1d35\" (UID: \"c93793c1-875c-4c81-a727-e462823b1d35\") " Oct 30 17:31:22.793443 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:22.793185 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c93793c1-875c-4c81-a727-e462823b1d35-kube-api-access-kc9vp\" (UniqueName: \"kubernetes.io/projected/c93793c1-875c-4c81-a727-e462823b1d35-kube-api-access-kc9vp\") pod \"c93793c1-875c-4c81-a727-e462823b1d35\" (UID: \"c93793c1-875c-4c81-a727-e462823b1d35\") " Oct 30 17:31:22.793528 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:22.793502 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c93793c1-875c-4c81-a727-e462823b1d35-maas-api-openshift-ca" pod "c93793c1-875c-4c81-a727-e462823b1d35" (UID: "c93793c1-875c-4c81-a727-e462823b1d35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:31:22.795433 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:22.795408 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c93793c1-875c-4c81-a727-e462823b1d35-maas-api-xks-ca" pod "c93793c1-875c-4c81-a727-e462823b1d35" (UID: "c93793c1-875c-4c81-a727-e462823b1d35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:31:22.795433 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:22.795416 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c93793c1-875c-4c81-a727-e462823b1d35-kube-api-access-kc9vp" pod "c93793c1-875c-4c81-a727-e462823b1d35" (UID: "c93793c1-875c-4c81-a727-e462823b1d35"). InnerVolumeSpecName "kube-api-access-kc9vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:31:22.893897 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:22.893857 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93793c1-875c-4c81-a727-e462823b1d35-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:31:22.893897 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:22.893888 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93793c1-875c-4c81-a727-e462823b1d35-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:31:22.893897 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:22.893899 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc9vp\" (UniqueName: \"kubernetes.io/projected/c93793c1-875c-4c81-a727-e462823b1d35-kube-api-access-kc9vp\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:31:23.483714 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:23.483676 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm" event={"ID":"c93793c1-875c-4c81-a727-e462823b1d35","Type":"ContainerDied","Data":"d12e7fcf08ce913b11e1088af9a8e0bec2baa2aa910f6eb392b198929c7b26e4"} Oct 30 17:31:23.483714 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:23.483707 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm" Oct 30 17:31:23.484035 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:23.483727 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d12e7fcf08ce913b11e1088af9a8e0bec2baa2aa910f6eb392b198929c7b26e4" Oct 30 17:31:32.038774 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.038738 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7"] Oct 30 17:31:32.039230 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.039211 6202 cpu_manager.go:414] "containerMap: removing container" podUID="c93793c1-875c-4c81-a727-e462823b1d35" containerName="revoke-keys" Oct 30 17:31:32.039289 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.039236 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c93793c1-875c-4c81-a727-e462823b1d35" containerName="revoke-keys" Oct 30 17:31:32.039377 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.039363 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="c93793c1-875c-4c81-a727-e462823b1d35" containerName="revoke-keys" Oct 30 17:31:32.043054 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.043035 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" Oct 30 17:31:32.074650 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.074605 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:31:32.074847 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.074675 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:31:32.076457 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.076428 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7"] Oct 30 17:31:32.078176 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.078153 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 17:31:32.086268 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.086233 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttdp7\" (UniqueName: \"kubernetes.io/projected/322a46d1-b63d-4776-bbbc-57428a70c4c9-kube-api-access-ttdp7\") pod \"maas-api-5c48877c75-dzzx7\" (UID: \"322a46d1-b63d-4776-bbbc-57428a70c4c9\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" Oct 30 17:31:32.086413 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.086287 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/322a46d1-b63d-4776-bbbc-57428a70c4c9-openshift-service-ca\") pod \"maas-api-5c48877c75-dzzx7\" (UID: \"322a46d1-b63d-4776-bbbc-57428a70c4c9\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" Oct 30 17:31:32.086470 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.086431 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/322a46d1-b63d-4776-bbbc-57428a70c4c9-metrics-certs\") pod \"maas-api-5c48877c75-dzzx7\" (UID: \"322a46d1-b63d-4776-bbbc-57428a70c4c9\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" Oct 30 17:31:32.086635 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.086613 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/322a46d1-b63d-4776-bbbc-57428a70c4c9-maas-api-tls\") pod \"maas-api-5c48877c75-dzzx7\" (UID: \"322a46d1-b63d-4776-bbbc-57428a70c4c9\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" Oct 30 17:31:32.086743 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.086634 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 17:31:32.086743 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.086654 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5jjc6\"" Oct 30 17:31:32.187493 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.187455 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/322a46d1-b63d-4776-bbbc-57428a70c4c9-maas-api-tls\") pod \"maas-api-5c48877c75-dzzx7\" (UID: \"322a46d1-b63d-4776-bbbc-57428a70c4c9\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" Oct 30 17:31:32.187493 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.187498 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttdp7\" (UniqueName: \"kubernetes.io/projected/322a46d1-b63d-4776-bbbc-57428a70c4c9-kube-api-access-ttdp7\") pod \"maas-api-5c48877c75-dzzx7\" (UID: \"322a46d1-b63d-4776-bbbc-57428a70c4c9\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" Oct 30 17:31:32.187801 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.187518 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/322a46d1-b63d-4776-bbbc-57428a70c4c9-openshift-service-ca\") pod \"maas-api-5c48877c75-dzzx7\" (UID: \"322a46d1-b63d-4776-bbbc-57428a70c4c9\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" Oct 30 17:31:32.187801 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.187545 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/322a46d1-b63d-4776-bbbc-57428a70c4c9-metrics-certs\") pod \"maas-api-5c48877c75-dzzx7\" (UID: \"322a46d1-b63d-4776-bbbc-57428a70c4c9\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" Oct 30 17:31:32.188321 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.188295 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/322a46d1-b63d-4776-bbbc-57428a70c4c9-openshift-service-ca\") pod \"maas-api-5c48877c75-dzzx7\" (UID: \"322a46d1-b63d-4776-bbbc-57428a70c4c9\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" Oct 30 17:31:32.190154 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.190124 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/322a46d1-b63d-4776-bbbc-57428a70c4c9-maas-api-tls\") pod \"maas-api-5c48877c75-dzzx7\" (UID: \"322a46d1-b63d-4776-bbbc-57428a70c4c9\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" Oct 30 17:31:32.190270 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.190195 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/322a46d1-b63d-4776-bbbc-57428a70c4c9-metrics-certs\") pod \"maas-api-5c48877c75-dzzx7\" (UID: \"322a46d1-b63d-4776-bbbc-57428a70c4c9\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" Oct 30 17:31:32.210180 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.210149 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttdp7\" (UniqueName: \"kubernetes.io/projected/322a46d1-b63d-4776-bbbc-57428a70c4c9-kube-api-access-ttdp7\") pod \"maas-api-5c48877c75-dzzx7\" (UID: \"322a46d1-b63d-4776-bbbc-57428a70c4c9\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" Oct 30 17:31:32.357075 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.357037 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" Oct 30 17:31:32.649980 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.649951 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7"] Oct 30 17:31:32.651462 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:31:32.651431 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322a46d1_b63d_4776_bbbc_57428a70c4c9.slice/crio-384e5be4fe4ab7f91d403c35e1622e86cf2c141750ea34db0391c0ab150fa487 WatchSource:0}: Error finding container 384e5be4fe4ab7f91d403c35e1622e86cf2c141750ea34db0391c0ab150fa487: Status 404 returned error can't find the container with id 384e5be4fe4ab7f91d403c35e1622e86cf2c141750ea34db0391c0ab150fa487 Oct 30 17:31:32.652817 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:32.652801 6202 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:31:33.534694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:33.534654 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" event={"ID":"322a46d1-b63d-4776-bbbc-57428a70c4c9","Type":"ContainerStarted","Data":"b1b30477b696c47c1e84fc2d5e8f15e2b48b0203f913f545e423f67d1f89abb9"} Oct 30 17:31:33.535160 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:33.534715 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" event={"ID":"322a46d1-b63d-4776-bbbc-57428a70c4c9","Type":"ContainerStarted","Data":"384e5be4fe4ab7f91d403c35e1622e86cf2c141750ea34db0391c0ab150fa487"} Oct 30 17:31:33.535160 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:33.534741 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" Oct 30 17:31:33.577316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:33.577263 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" podStartSLOduration=1.232543804 podStartE2EDuration="1.577248908s" totalImagesPullingTime="344.705104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:33.576931014 +0000 UTC m=+4207.287385757" watchObservedRunningTime="2026-10-30 17:31:33.577248908 +0000 UTC m=+4207.287703651" Oct 30 17:31:39.544370 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:39.544342 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" Oct 30 17:31:44.067617 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:31:44.067578 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:31:44.730193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:44.730154 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:47.674320 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:47.674271 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7"] Oct 30 17:31:47.675047 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:47.674547 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" podUID="322a46d1-b63d-4776-bbbc-57428a70c4c9" containerName="maas-api" containerID="cri-o://b1b30477b696c47c1e84fc2d5e8f15e2b48b0203f913f545e423f67d1f89abb9" gracePeriod=30 Oct 30 17:31:47.925760 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:47.925699 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" Oct 30 17:31:47.969589 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:47.969551 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/322a46d1-b63d-4776-bbbc-57428a70c4c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/322a46d1-b63d-4776-bbbc-57428a70c4c9-openshift-service-ca\") pod \"322a46d1-b63d-4776-bbbc-57428a70c4c9\" (UID: \"322a46d1-b63d-4776-bbbc-57428a70c4c9\") " Oct 30 17:31:47.969808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:47.969697 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/322a46d1-b63d-4776-bbbc-57428a70c4c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/322a46d1-b63d-4776-bbbc-57428a70c4c9-metrics-certs\") pod \"322a46d1-b63d-4776-bbbc-57428a70c4c9\" (UID: \"322a46d1-b63d-4776-bbbc-57428a70c4c9\") " Oct 30 17:31:47.969808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:47.969738 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322a46d1-b63d-4776-bbbc-57428a70c4c9-kube-api-access-ttdp7\" (UniqueName: \"kubernetes.io/projected/322a46d1-b63d-4776-bbbc-57428a70c4c9-kube-api-access-ttdp7\") pod \"322a46d1-b63d-4776-bbbc-57428a70c4c9\" (UID: \"322a46d1-b63d-4776-bbbc-57428a70c4c9\") " Oct 30 17:31:47.969808 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:47.969800 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/322a46d1-b63d-4776-bbbc-57428a70c4c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/322a46d1-b63d-4776-bbbc-57428a70c4c9-maas-api-tls\") pod \"322a46d1-b63d-4776-bbbc-57428a70c4c9\" (UID: \"322a46d1-b63d-4776-bbbc-57428a70c4c9\") " Oct 30 17:31:47.969999 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:47.969975 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322a46d1-b63d-4776-bbbc-57428a70c4c9-openshift-service-ca" pod "322a46d1-b63d-4776-bbbc-57428a70c4c9" (UID: "322a46d1-b63d-4776-bbbc-57428a70c4c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:31:47.970205 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:47.970180 6202 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/322a46d1-b63d-4776-bbbc-57428a70c4c9-openshift-service-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:31:47.972319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:47.972288 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322a46d1-b63d-4776-bbbc-57428a70c4c9-kube-api-access-ttdp7" pod "322a46d1-b63d-4776-bbbc-57428a70c4c9" (UID: "322a46d1-b63d-4776-bbbc-57428a70c4c9"). InnerVolumeSpecName "kube-api-access-ttdp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:31:47.972471 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:47.972342 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322a46d1-b63d-4776-bbbc-57428a70c4c9-metrics-certs" pod "322a46d1-b63d-4776-bbbc-57428a70c4c9" (UID: "322a46d1-b63d-4776-bbbc-57428a70c4c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:31:47.972540 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:47.972480 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322a46d1-b63d-4776-bbbc-57428a70c4c9-maas-api-tls" pod "322a46d1-b63d-4776-bbbc-57428a70c4c9" (UID: "322a46d1-b63d-4776-bbbc-57428a70c4c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:31:48.071552 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:48.071515 6202 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/322a46d1-b63d-4776-bbbc-57428a70c4c9-metrics-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:31:48.071552 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:48.071546 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttdp7\" (UniqueName: \"kubernetes.io/projected/322a46d1-b63d-4776-bbbc-57428a70c4c9-kube-api-access-ttdp7\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:31:48.071552 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:48.071557 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/322a46d1-b63d-4776-bbbc-57428a70c4c9-maas-api-tls\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:31:48.630509 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:48.630470 6202 generic.go:356] "Generic (PLEG): container finished" podID="322a46d1-b63d-4776-bbbc-57428a70c4c9" containerID="b1b30477b696c47c1e84fc2d5e8f15e2b48b0203f913f545e423f67d1f89abb9" exitCode=0 Oct 30 17:31:48.630509 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:48.630513 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" event={"ID":"322a46d1-b63d-4776-bbbc-57428a70c4c9","Type":"ContainerDied","Data":"b1b30477b696c47c1e84fc2d5e8f15e2b48b0203f913f545e423f67d1f89abb9"} Oct 30 17:31:48.630788 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:48.630554 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" event={"ID":"322a46d1-b63d-4776-bbbc-57428a70c4c9","Type":"ContainerDied","Data":"384e5be4fe4ab7f91d403c35e1622e86cf2c141750ea34db0391c0ab150fa487"} Oct 30 17:31:48.630788 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:48.630577 6202 scope.go:122] "RemoveContainer" containerID="b1b30477b696c47c1e84fc2d5e8f15e2b48b0203f913f545e423f67d1f89abb9" Oct 30 17:31:48.630788 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:48.630514 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7" Oct 30 17:31:48.639530 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:48.639508 6202 scope.go:122] "RemoveContainer" containerID="b1b30477b696c47c1e84fc2d5e8f15e2b48b0203f913f545e423f67d1f89abb9" Oct 30 17:31:48.639852 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:31:48.639823 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b30477b696c47c1e84fc2d5e8f15e2b48b0203f913f545e423f67d1f89abb9\": container with ID starting with b1b30477b696c47c1e84fc2d5e8f15e2b48b0203f913f545e423f67d1f89abb9 not found: ID does not exist" containerID="b1b30477b696c47c1e84fc2d5e8f15e2b48b0203f913f545e423f67d1f89abb9" Oct 30 17:31:48.639974 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:48.639860 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b30477b696c47c1e84fc2d5e8f15e2b48b0203f913f545e423f67d1f89abb9"} err="failed to get container status \"b1b30477b696c47c1e84fc2d5e8f15e2b48b0203f913f545e423f67d1f89abb9\": rpc error: code = NotFound desc = could not find container \"b1b30477b696c47c1e84fc2d5e8f15e2b48b0203f913f545e423f67d1f89abb9\": container with ID starting with b1b30477b696c47c1e84fc2d5e8f15e2b48b0203f913f545e423f67d1f89abb9 not found: ID does not exist" Oct 30 17:31:48.683927 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:48.683833 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7"] Oct 30 17:31:48.696188 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:48.696146 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-dzzx7"] Oct 30 17:31:50.414674 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:50.414620 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="322a46d1-b63d-4776-bbbc-57428a70c4c9" path="/var/lib/kubelet/pods/322a46d1-b63d-4776-bbbc-57428a70c4c9/volumes" Oct 30 17:31:55.737671 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.737636 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm"] Oct 30 17:31:55.738133 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.738103 6202 cpu_manager.go:414] "containerMap: removing container" podUID="322a46d1-b63d-4776-bbbc-57428a70c4c9" containerName="maas-api" Oct 30 17:31:55.738133 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.738118 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322a46d1-b63d-4776-bbbc-57428a70c4c9" containerName="maas-api" Oct 30 17:31:55.738221 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.738206 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="322a46d1-b63d-4776-bbbc-57428a70c4c9" containerName="maas-api" Oct 30 17:31:55.741521 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.741501 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm" Oct 30 17:31:55.759147 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.759060 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:31:55.759296 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.759206 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:31:55.771001 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.770972 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:31:55.771147 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.771019 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:31:55.772712 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.772693 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm"] Oct 30 17:31:55.860522 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.860469 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h985g\" (UniqueName: \"kubernetes.io/projected/14f3e4af-fd5a-4a61-9e4b-e521f3287456-kube-api-access-h985g\") pod \"maas-api-revoke-sub-f03629db-rk2gm\" (UID: \"14f3e4af-fd5a-4a61-9e4b-e521f3287456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm" Oct 30 17:31:55.860744 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.860603 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f3e4af-fd5a-4a61-9e4b-e521f3287456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-rk2gm\" (UID: \"14f3e4af-fd5a-4a61-9e4b-e521f3287456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm" Oct 30 17:31:55.860744 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.860652 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f3e4af-fd5a-4a61-9e4b-e521f3287456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-rk2gm\" (UID: \"14f3e4af-fd5a-4a61-9e4b-e521f3287456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm" Oct 30 17:31:55.962275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.962231 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h985g\" (UniqueName: \"kubernetes.io/projected/14f3e4af-fd5a-4a61-9e4b-e521f3287456-kube-api-access-h985g\") pod \"maas-api-revoke-sub-f03629db-rk2gm\" (UID: \"14f3e4af-fd5a-4a61-9e4b-e521f3287456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm" Oct 30 17:31:55.962509 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.962331 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f3e4af-fd5a-4a61-9e4b-e521f3287456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-rk2gm\" (UID: \"14f3e4af-fd5a-4a61-9e4b-e521f3287456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm" Oct 30 17:31:55.962509 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.962368 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f3e4af-fd5a-4a61-9e4b-e521f3287456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-rk2gm\" (UID: \"14f3e4af-fd5a-4a61-9e4b-e521f3287456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm" Oct 30 17:31:55.963151 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.963126 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f3e4af-fd5a-4a61-9e4b-e521f3287456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-rk2gm\" (UID: \"14f3e4af-fd5a-4a61-9e4b-e521f3287456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm" Oct 30 17:31:55.965004 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.964983 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f3e4af-fd5a-4a61-9e4b-e521f3287456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-rk2gm\" (UID: \"14f3e4af-fd5a-4a61-9e4b-e521f3287456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm" Oct 30 17:31:55.985912 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:55.985878 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h985g\" (UniqueName: \"kubernetes.io/projected/14f3e4af-fd5a-4a61-9e4b-e521f3287456-kube-api-access-h985g\") pod \"maas-api-revoke-sub-f03629db-rk2gm\" (UID: \"14f3e4af-fd5a-4a61-9e4b-e521f3287456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm" Oct 30 17:31:56.054723 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:56.054636 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm" Oct 30 17:31:56.352794 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:56.352746 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm"] Oct 30 17:31:56.355273 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:31:56.355243 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f3e4af_fd5a_4a61_9e4b_e521f3287456.slice/crio-93c72d00f9d72e458f29f500f7556fcf203d8e793b02b48c11e0b1423f55ee36 WatchSource:0}: Error finding container 93c72d00f9d72e458f29f500f7556fcf203d8e793b02b48c11e0b1423f55ee36: Status 404 returned error can't find the container with id 93c72d00f9d72e458f29f500f7556fcf203d8e793b02b48c11e0b1423f55ee36 Oct 30 17:31:56.698802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:56.698749 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm" event={"ID":"14f3e4af-fd5a-4a61-9e4b-e521f3287456","Type":"ContainerStarted","Data":"07a05667cad7e7a2d0e5656dffd6de93f021fe22c49acd733fabc5397d09d5df"} Oct 30 17:31:56.699039 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:56.698819 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm" event={"ID":"14f3e4af-fd5a-4a61-9e4b-e521f3287456","Type":"ContainerStarted","Data":"93c72d00f9d72e458f29f500f7556fcf203d8e793b02b48c11e0b1423f55ee36"} Oct 30 17:31:56.739474 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:56.739426 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm" podStartSLOduration=1.7394142110000002 podStartE2EDuration="1.739414211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:56.738070057 +0000 UTC m=+4230.448524801" watchObservedRunningTime="2026-10-30 17:31:56.739414211 +0000 UTC m=+4230.449868987" Oct 30 17:31:57.705510 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:57.705464 6202 generic.go:356] "Generic (PLEG): container finished" podID="14f3e4af-fd5a-4a61-9e4b-e521f3287456" containerID="07a05667cad7e7a2d0e5656dffd6de93f021fe22c49acd733fabc5397d09d5df" exitCode=0 Oct 30 17:31:57.705735 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:57.705530 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm" event={"ID":"14f3e4af-fd5a-4a61-9e4b-e521f3287456","Type":"ContainerDied","Data":"07a05667cad7e7a2d0e5656dffd6de93f021fe22c49acd733fabc5397d09d5df"} Oct 30 17:31:58.841394 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:58.841361 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm" Oct 30 17:31:58.890712 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:58.890677 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14f3e4af-fd5a-4a61-9e4b-e521f3287456-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f3e4af-fd5a-4a61-9e4b-e521f3287456-maas-api-openshift-ca\") pod \"14f3e4af-fd5a-4a61-9e4b-e521f3287456\" (UID: \"14f3e4af-fd5a-4a61-9e4b-e521f3287456\") " Oct 30 17:31:58.890944 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:58.890729 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f3e4af-fd5a-4a61-9e4b-e521f3287456-kube-api-access-h985g\" (UniqueName: \"kubernetes.io/projected/14f3e4af-fd5a-4a61-9e4b-e521f3287456-kube-api-access-h985g\") pod \"14f3e4af-fd5a-4a61-9e4b-e521f3287456\" (UID: \"14f3e4af-fd5a-4a61-9e4b-e521f3287456\") " Oct 30 17:31:58.890944 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:58.890851 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14f3e4af-fd5a-4a61-9e4b-e521f3287456-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f3e4af-fd5a-4a61-9e4b-e521f3287456-maas-api-xks-ca\") pod \"14f3e4af-fd5a-4a61-9e4b-e521f3287456\" (UID: \"14f3e4af-fd5a-4a61-9e4b-e521f3287456\") " Oct 30 17:31:58.891171 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:58.891131 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f3e4af-fd5a-4a61-9e4b-e521f3287456-maas-api-openshift-ca" pod "14f3e4af-fd5a-4a61-9e4b-e521f3287456" (UID: "14f3e4af-fd5a-4a61-9e4b-e521f3287456"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:31:58.891275 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:58.891256 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f3e4af-fd5a-4a61-9e4b-e521f3287456-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:31:58.893408 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:58.893384 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14f3e4af-fd5a-4a61-9e4b-e521f3287456-maas-api-xks-ca" pod "14f3e4af-fd5a-4a61-9e4b-e521f3287456" (UID: "14f3e4af-fd5a-4a61-9e4b-e521f3287456"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:31:58.893634 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:58.893608 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f3e4af-fd5a-4a61-9e4b-e521f3287456-kube-api-access-h985g" pod "14f3e4af-fd5a-4a61-9e4b-e521f3287456" (UID: "14f3e4af-fd5a-4a61-9e4b-e521f3287456"). InnerVolumeSpecName "kube-api-access-h985g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:31:58.992254 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:58.992158 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f3e4af-fd5a-4a61-9e4b-e521f3287456-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:31:58.992254 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:58.992202 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h985g\" (UniqueName: \"kubernetes.io/projected/14f3e4af-fd5a-4a61-9e4b-e521f3287456-kube-api-access-h985g\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:31:59.715624 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:59.715592 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm" Oct 30 17:31:59.715844 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:59.715594 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm" event={"ID":"14f3e4af-fd5a-4a61-9e4b-e521f3287456","Type":"ContainerDied","Data":"93c72d00f9d72e458f29f500f7556fcf203d8e793b02b48c11e0b1423f55ee36"} Oct 30 17:31:59.715844 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:31:59.715723 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93c72d00f9d72e458f29f500f7556fcf203d8e793b02b48c11e0b1423f55ee36" Oct 30 17:32:13.591418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.591369 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz"] Oct 30 17:32:13.591989 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.591899 6202 cpu_manager.go:414] "containerMap: removing container" podUID="14f3e4af-fd5a-4a61-9e4b-e521f3287456" containerName="revoke-keys" Oct 30 17:32:13.591989 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.591932 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f3e4af-fd5a-4a61-9e4b-e521f3287456" containerName="revoke-keys" Oct 30 17:32:13.592230 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.592041 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f3e4af-fd5a-4a61-9e4b-e521f3287456" containerName="revoke-keys" Oct 30 17:32:13.598354 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.598335 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz" Oct 30 17:32:13.614641 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.614600 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz"] Oct 30 17:32:13.625082 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.625054 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:32:13.625082 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.625053 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:32:13.625315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.625109 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:32:13.625315 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.625186 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:32:13.757324 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.757278 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f6b47b-677e-416a-a1d4-c65c81916e00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-x58sz\" (UID: \"07f6b47b-677e-416a-a1d4-c65c81916e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz" Oct 30 17:32:13.757324 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.757324 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwcxp\" (UniqueName: \"kubernetes.io/projected/07f6b47b-677e-416a-a1d4-c65c81916e00-kube-api-access-kwcxp\") pod \"maas-api-revoke-sub-0165d7d2-x58sz\" (UID: \"07f6b47b-677e-416a-a1d4-c65c81916e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz" Oct 30 17:32:13.757623 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.757469 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f6b47b-677e-416a-a1d4-c65c81916e00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-x58sz\" (UID: \"07f6b47b-677e-416a-a1d4-c65c81916e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz" Oct 30 17:32:13.858912 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.858833 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f6b47b-677e-416a-a1d4-c65c81916e00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-x58sz\" (UID: \"07f6b47b-677e-416a-a1d4-c65c81916e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz" Oct 30 17:32:13.858912 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.858877 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwcxp\" (UniqueName: \"kubernetes.io/projected/07f6b47b-677e-416a-a1d4-c65c81916e00-kube-api-access-kwcxp\") pod \"maas-api-revoke-sub-0165d7d2-x58sz\" (UID: \"07f6b47b-677e-416a-a1d4-c65c81916e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz" Oct 30 17:32:13.859096 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.858924 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f6b47b-677e-416a-a1d4-c65c81916e00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-x58sz\" (UID: \"07f6b47b-677e-416a-a1d4-c65c81916e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz" Oct 30 17:32:13.859555 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.859530 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f6b47b-677e-416a-a1d4-c65c81916e00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-x58sz\" (UID: \"07f6b47b-677e-416a-a1d4-c65c81916e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz" Oct 30 17:32:13.861316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.861287 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f6b47b-677e-416a-a1d4-c65c81916e00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-x58sz\" (UID: \"07f6b47b-677e-416a-a1d4-c65c81916e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz" Oct 30 17:32:13.882280 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.882252 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwcxp\" (UniqueName: \"kubernetes.io/projected/07f6b47b-677e-416a-a1d4-c65c81916e00-kube-api-access-kwcxp\") pod \"maas-api-revoke-sub-0165d7d2-x58sz\" (UID: \"07f6b47b-677e-416a-a1d4-c65c81916e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz" Oct 30 17:32:13.912165 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:13.912143 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz" Oct 30 17:32:14.066199 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:32:14.066162 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:32:14.201953 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:14.201202 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz"] Oct 30 17:32:14.203864 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:32:14.203833 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f6b47b_677e_416a_a1d4_c65c81916e00.slice/crio-bb42f0c587c4537e37d3a81b942d5fbf32c6fdf76635a9de203475c53ae387c2 WatchSource:0}: Error finding container bb42f0c587c4537e37d3a81b942d5fbf32c6fdf76635a9de203475c53ae387c2: Status 404 returned error can't find the container with id bb42f0c587c4537e37d3a81b942d5fbf32c6fdf76635a9de203475c53ae387c2 Oct 30 17:32:14.803823 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:14.803785 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz" event={"ID":"07f6b47b-677e-416a-a1d4-c65c81916e00","Type":"ContainerStarted","Data":"3c2cea5a90d30b7cf11d64cb9f17e4c45df22bf329f73bad42c9badff4a9fdf5"} Oct 30 17:32:14.804281 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:14.803837 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz" event={"ID":"07f6b47b-677e-416a-a1d4-c65c81916e00","Type":"ContainerStarted","Data":"bb42f0c587c4537e37d3a81b942d5fbf32c6fdf76635a9de203475c53ae387c2"} Oct 30 17:32:14.843354 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:14.843299 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz" podStartSLOduration=1.8432857679999999 podStartE2EDuration="1.843285768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:14.843102406 +0000 UTC m=+4248.553557152" watchObservedRunningTime="2026-10-30 17:32:14.843285768 +0000 UTC m=+4248.553740511" Oct 30 17:32:15.809699 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:15.809649 6202 generic.go:356] "Generic (PLEG): container finished" podID="07f6b47b-677e-416a-a1d4-c65c81916e00" containerID="3c2cea5a90d30b7cf11d64cb9f17e4c45df22bf329f73bad42c9badff4a9fdf5" exitCode=0 Oct 30 17:32:15.809699 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:15.809691 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz" event={"ID":"07f6b47b-677e-416a-a1d4-c65c81916e00","Type":"ContainerDied","Data":"3c2cea5a90d30b7cf11d64cb9f17e4c45df22bf329f73bad42c9badff4a9fdf5"} Oct 30 17:32:16.947048 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:16.947021 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz" Oct 30 17:32:17.089390 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:17.089360 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07f6b47b-677e-416a-a1d4-c65c81916e00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f6b47b-677e-416a-a1d4-c65c81916e00-maas-api-xks-ca\") pod \"07f6b47b-677e-416a-a1d4-c65c81916e00\" (UID: \"07f6b47b-677e-416a-a1d4-c65c81916e00\") " Oct 30 17:32:17.089577 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:17.089415 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f6b47b-677e-416a-a1d4-c65c81916e00-kube-api-access-kwcxp\" (UniqueName: \"kubernetes.io/projected/07f6b47b-677e-416a-a1d4-c65c81916e00-kube-api-access-kwcxp\") pod \"07f6b47b-677e-416a-a1d4-c65c81916e00\" (UID: \"07f6b47b-677e-416a-a1d4-c65c81916e00\") " Oct 30 17:32:17.089577 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:17.089447 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07f6b47b-677e-416a-a1d4-c65c81916e00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f6b47b-677e-416a-a1d4-c65c81916e00-maas-api-openshift-ca\") pod \"07f6b47b-677e-416a-a1d4-c65c81916e00\" (UID: \"07f6b47b-677e-416a-a1d4-c65c81916e00\") " Oct 30 17:32:17.089892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:17.089868 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f6b47b-677e-416a-a1d4-c65c81916e00-maas-api-openshift-ca" pod "07f6b47b-677e-416a-a1d4-c65c81916e00" (UID: "07f6b47b-677e-416a-a1d4-c65c81916e00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:32:17.091597 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:17.091571 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f6b47b-677e-416a-a1d4-c65c81916e00-maas-api-xks-ca" pod "07f6b47b-677e-416a-a1d4-c65c81916e00" (UID: "07f6b47b-677e-416a-a1d4-c65c81916e00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:32:17.091704 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:17.091654 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f6b47b-677e-416a-a1d4-c65c81916e00-kube-api-access-kwcxp" pod "07f6b47b-677e-416a-a1d4-c65c81916e00" (UID: "07f6b47b-677e-416a-a1d4-c65c81916e00"). InnerVolumeSpecName "kube-api-access-kwcxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:17.191166 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:17.191128 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f6b47b-677e-416a-a1d4-c65c81916e00-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:32:17.191166 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:17.191158 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwcxp\" (UniqueName: \"kubernetes.io/projected/07f6b47b-677e-416a-a1d4-c65c81916e00-kube-api-access-kwcxp\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:32:17.191166 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:17.191168 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f6b47b-677e-416a-a1d4-c65c81916e00-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:32:17.820220 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:17.820187 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz" event={"ID":"07f6b47b-677e-416a-a1d4-c65c81916e00","Type":"ContainerDied","Data":"bb42f0c587c4537e37d3a81b942d5fbf32c6fdf76635a9de203475c53ae387c2"} Oct 30 17:32:17.820411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:17.820242 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb42f0c587c4537e37d3a81b942d5fbf32c6fdf76635a9de203475c53ae387c2" Oct 30 17:32:17.820411 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:17.820202 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz" Oct 30 17:32:18.101418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:18.101380 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:24.529010 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.528972 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z"] Oct 30 17:32:24.529627 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.529404 6202 cpu_manager.go:414] "containerMap: removing container" podUID="07f6b47b-677e-416a-a1d4-c65c81916e00" containerName="revoke-keys" Oct 30 17:32:24.529627 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.529426 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f6b47b-677e-416a-a1d4-c65c81916e00" containerName="revoke-keys" Oct 30 17:32:24.529627 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.529528 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f6b47b-677e-416a-a1d4-c65c81916e00" containerName="revoke-keys" Oct 30 17:32:24.533023 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.533004 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z" Oct 30 17:32:24.550466 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.550435 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:32:24.559295 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.559262 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9940dbed-c8d8-40d4-8baf-c1ed02b71622-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-7dt2z\" (UID: \"9940dbed-c8d8-40d4-8baf-c1ed02b71622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z" Oct 30 17:32:24.559446 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.559426 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9940dbed-c8d8-40d4-8baf-c1ed02b71622-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-7dt2z\" (UID: \"9940dbed-c8d8-40d4-8baf-c1ed02b71622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z" Oct 30 17:32:24.559522 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.559467 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-564s5\" (UniqueName: \"kubernetes.io/projected/9940dbed-c8d8-40d4-8baf-c1ed02b71622-kube-api-access-564s5\") pod \"maas-api-revoke-sub-6b19338f-7dt2z\" (UID: \"9940dbed-c8d8-40d4-8baf-c1ed02b71622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z" Oct 30 17:32:24.559522 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.559490 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:32:24.559632 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.559523 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:32:24.559632 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.559595 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:32:24.562048 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.562025 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z"] Oct 30 17:32:24.660221 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.660183 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-564s5\" (UniqueName: \"kubernetes.io/projected/9940dbed-c8d8-40d4-8baf-c1ed02b71622-kube-api-access-564s5\") pod \"maas-api-revoke-sub-6b19338f-7dt2z\" (UID: \"9940dbed-c8d8-40d4-8baf-c1ed02b71622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z" Oct 30 17:32:24.660426 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.660249 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9940dbed-c8d8-40d4-8baf-c1ed02b71622-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-7dt2z\" (UID: \"9940dbed-c8d8-40d4-8baf-c1ed02b71622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z" Oct 30 17:32:24.660426 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.660324 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9940dbed-c8d8-40d4-8baf-c1ed02b71622-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-7dt2z\" (UID: \"9940dbed-c8d8-40d4-8baf-c1ed02b71622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z" Oct 30 17:32:24.661015 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.660988 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9940dbed-c8d8-40d4-8baf-c1ed02b71622-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-7dt2z\" (UID: \"9940dbed-c8d8-40d4-8baf-c1ed02b71622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z" Oct 30 17:32:24.662717 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.662694 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9940dbed-c8d8-40d4-8baf-c1ed02b71622-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-7dt2z\" (UID: \"9940dbed-c8d8-40d4-8baf-c1ed02b71622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z" Oct 30 17:32:24.683780 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.683743 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-564s5\" (UniqueName: \"kubernetes.io/projected/9940dbed-c8d8-40d4-8baf-c1ed02b71622-kube-api-access-564s5\") pod \"maas-api-revoke-sub-6b19338f-7dt2z\" (UID: \"9940dbed-c8d8-40d4-8baf-c1ed02b71622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z" Oct 30 17:32:24.846937 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:24.846900 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z" Oct 30 17:32:25.139560 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:25.139524 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z"] Oct 30 17:32:25.141230 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:32:25.141200 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9940dbed_c8d8_40d4_8baf_c1ed02b71622.slice/crio-5c705ef7c2914e015744222123761d47fe1bd509adae059b3af16c68165377e5 WatchSource:0}: Error finding container 5c705ef7c2914e015744222123761d47fe1bd509adae059b3af16c68165377e5: Status 404 returned error can't find the container with id 5c705ef7c2914e015744222123761d47fe1bd509adae059b3af16c68165377e5 Oct 30 17:32:25.860837 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:25.860798 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z" event={"ID":"9940dbed-c8d8-40d4-8baf-c1ed02b71622","Type":"ContainerStarted","Data":"af0907f7c73b468b49967b76d4f987922f8eaf1c367d33fcc0dce0177837e1d1"} Oct 30 17:32:25.861330 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:25.860869 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z" event={"ID":"9940dbed-c8d8-40d4-8baf-c1ed02b71622","Type":"ContainerStarted","Data":"5c705ef7c2914e015744222123761d47fe1bd509adae059b3af16c68165377e5"} Oct 30 17:32:25.900752 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:25.900704 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z" podStartSLOduration=1.900691326 podStartE2EDuration="1.900691326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:25.900248568 +0000 UTC m=+4259.610703312" watchObservedRunningTime="2026-10-30 17:32:25.900691326 +0000 UTC m=+4259.611146070" Oct 30 17:32:26.867957 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:26.867919 6202 generic.go:356] "Generic (PLEG): container finished" podID="9940dbed-c8d8-40d4-8baf-c1ed02b71622" containerID="af0907f7c73b468b49967b76d4f987922f8eaf1c367d33fcc0dce0177837e1d1" exitCode=0 Oct 30 17:32:26.868387 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:26.867997 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z" event={"ID":"9940dbed-c8d8-40d4-8baf-c1ed02b71622","Type":"ContainerDied","Data":"af0907f7c73b468b49967b76d4f987922f8eaf1c367d33fcc0dce0177837e1d1"} Oct 30 17:32:28.008659 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:28.008632 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z" Oct 30 17:32:28.091277 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:28.091249 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9940dbed-c8d8-40d4-8baf-c1ed02b71622-kube-api-access-564s5\" (UniqueName: \"kubernetes.io/projected/9940dbed-c8d8-40d4-8baf-c1ed02b71622-kube-api-access-564s5\") pod \"9940dbed-c8d8-40d4-8baf-c1ed02b71622\" (UID: \"9940dbed-c8d8-40d4-8baf-c1ed02b71622\") " Oct 30 17:32:28.091453 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:28.091326 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9940dbed-c8d8-40d4-8baf-c1ed02b71622-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9940dbed-c8d8-40d4-8baf-c1ed02b71622-maas-api-openshift-ca\") pod \"9940dbed-c8d8-40d4-8baf-c1ed02b71622\" (UID: \"9940dbed-c8d8-40d4-8baf-c1ed02b71622\") " Oct 30 17:32:28.091499 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:28.091450 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9940dbed-c8d8-40d4-8baf-c1ed02b71622-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9940dbed-c8d8-40d4-8baf-c1ed02b71622-maas-api-xks-ca\") pod \"9940dbed-c8d8-40d4-8baf-c1ed02b71622\" (UID: \"9940dbed-c8d8-40d4-8baf-c1ed02b71622\") " Oct 30 17:32:28.091679 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:28.091657 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9940dbed-c8d8-40d4-8baf-c1ed02b71622-maas-api-openshift-ca" pod "9940dbed-c8d8-40d4-8baf-c1ed02b71622" (UID: "9940dbed-c8d8-40d4-8baf-c1ed02b71622"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:32:28.091780 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:28.091751 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9940dbed-c8d8-40d4-8baf-c1ed02b71622-maas-api-openshift-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:32:28.093583 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:28.093559 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9940dbed-c8d8-40d4-8baf-c1ed02b71622-maas-api-xks-ca" pod "9940dbed-c8d8-40d4-8baf-c1ed02b71622" (UID: "9940dbed-c8d8-40d4-8baf-c1ed02b71622"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:32:28.093657 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:28.093586 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9940dbed-c8d8-40d4-8baf-c1ed02b71622-kube-api-access-564s5" pod "9940dbed-c8d8-40d4-8baf-c1ed02b71622" (UID: "9940dbed-c8d8-40d4-8baf-c1ed02b71622"). InnerVolumeSpecName "kube-api-access-564s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:28.193118 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:28.193037 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-564s5\" (UniqueName: \"kubernetes.io/projected/9940dbed-c8d8-40d4-8baf-c1ed02b71622-kube-api-access-564s5\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:32:28.193118 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:28.193067 6202 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9940dbed-c8d8-40d4-8baf-c1ed02b71622-maas-api-xks-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:32:28.883427 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:28.883396 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z" Oct 30 17:32:28.883625 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:28.883429 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z" event={"ID":"9940dbed-c8d8-40d4-8baf-c1ed02b71622","Type":"ContainerDied","Data":"5c705ef7c2914e015744222123761d47fe1bd509adae059b3af16c68165377e5"} Oct 30 17:32:28.883625 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:28.883476 6202 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c705ef7c2914e015744222123761d47fe1bd509adae059b3af16c68165377e5" Oct 30 17:32:41.026751 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:41.026703 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp"] Oct 30 17:32:41.038323 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:41.038290 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ff2wp"] Oct 30 17:32:42.408215 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:42.408179 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a30a2bea-6b7f-497d-a77c-5f7007cc7076" path="/var/lib/kubelet/pods/a30a2bea-6b7f-497d-a77c-5f7007cc7076/volumes" Oct 30 17:32:44.067515 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:32:44.067477 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:32:44.814470 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:44.814432 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:45.138099 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:45.138036 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:58.028949 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:58.028912 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489"] Oct 30 17:32:58.040425 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:58.040397 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bg489"] Oct 30 17:32:58.407921 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:32:58.407883 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10789212-2d42-4796-b759-7dd7ac372f19" path="/var/lib/kubelet/pods/10789212-2d42-4796-b759-7dd7ac372f19/volumes" Oct 30 17:33:14.072258 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:33:14.072216 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:33:26.683527 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:26.683483 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6d48fc876f-4g6d7_1d405c38-8a59-446b-9517-6804f6e29cdc/manager/0.log" Oct 30 17:33:28.851659 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:28.851636 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cbcd98d6d-xbsxw_b6e0ab79-e56f-453b-8159-97d62ccbeb59/copy-manifests/0.log" Oct 30 17:33:28.880111 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:28.880080 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cbcd98d6d-xbsxw_b6e0ab79-e56f-453b-8159-97d62ccbeb59/manager/0.log" Oct 30 17:33:30.576498 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:30.576467 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-x58sz_07f6b47b-677e-416a-a1d4-c65c81916e00/revoke-keys/0.log" Oct 30 17:33:30.772052 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:30.772018 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-175680ee-hlpk7_0f563d4f-8cb0-403a-a084-a7c57d16d796/revoke-keys/0.log" Oct 30 17:33:30.967892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:30.967816 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-7qgkc_9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4/revoke-keys/0.log" Oct 30 17:33:31.164855 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:31.164825 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-jvcxz_145ed570-31e7-420a-bafa-eff4e2b08ab1/revoke-keys/0.log" Oct 30 17:33:31.358866 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:31.358836 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-7dt2z_9940dbed-c8d8-40d4-8baf-c1ed02b71622/revoke-keys/0.log" Oct 30 17:33:31.547306 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:31.547276 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-rk2gm_14f3e4af-fd5a-4a61-9e4b-e521f3287456/revoke-keys/0.log" Oct 30 17:33:31.733345 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:31.733271 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fef6b280-cb7nm_c93793c1-875c-4c81-a727-e462823b1d35/revoke-keys/0.log" Oct 30 17:33:33.731920 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:33.731883 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-nsfnw_118b3526-1a2c-42dc-9061-310ccd01d554/manager/0.log" Oct 30 17:33:33.942133 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:33.942102 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-nzbcq_34745160-9b04-4309-bb7b-87cb8f4aca06/manager/0.log" Oct 30 17:33:35.748272 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:35.748199 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-hz72n_f39a5526-1dde-4513-a5ad-0c29de3b0fd0/discovery/0.log" Oct 30 17:33:35.968050 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:35.968017 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54d485b749-54hv6_785b392b-2225-4f7c-97dd-74ba17073238/kube-auth-proxy/0.log" Oct 30 17:33:36.546134 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:36.546101 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-5mcb6_04a034df-8dc0-489b-a623-4bb9e730e6ad/istio-proxy/0.log" Oct 30 17:33:37.189945 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:37.189915 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-ffkqn_27b9c1c4-c3da-4caf-a594-b391b48cc93a/istio-proxy/0.log" Oct 30 17:33:37.967760 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:37.967729 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-sm8wx_f4591587-01a3-4160-a8b7-d25450be9054/istio-proxy/0.log" Oct 30 17:33:38.391436 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:38.391403 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-57rw9_8a21ffcf-b1ed-4709-934e-f88b301ea0f4/payload-pre-processing/0.log" Oct 30 17:33:44.066937 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:33:44.066899 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:33:44.939135 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:44.939097 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:46.027098 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:46.027055 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:49.774832 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:49.774797 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-tbc5m_0c7c579a-a559-4424-b23b-cc7b91f61532/main/0.log" Oct 30 17:33:49.968581 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:49.968550 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-4kdj8_6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4/main/0.log" Oct 30 17:33:50.728731 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:33:50.728700 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8_9702f62c-b2ea-47b1-9e67-d50be44e166e/main/0.log" Oct 30 17:34:10.028040 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:10.028001 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7"] Oct 30 17:34:10.039629 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:10.039600 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175680ee-hlpk7"] Oct 30 17:34:10.408211 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:10.408177 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f563d4f-8cb0-403a-a084-a7c57d16d796" path="/var/lib/kubelet/pods/0f563d4f-8cb0-403a-a084-a7c57d16d796/volumes" Oct 30 17:34:14.066660 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:34:14.066624 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:34:20.068442 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:20.068401 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:20.200725 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:20.200693 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:20.802456 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:20.802429 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6d48fc876f-4g6d7_1d405c38-8a59-446b-9517-6804f6e29cdc/manager/0.log" Oct 30 17:34:22.738026 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:22.737996 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cbcd98d6d-xbsxw_b6e0ab79-e56f-453b-8159-97d62ccbeb59/copy-manifests/0.log" Oct 30 17:34:22.771307 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:22.771271 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cbcd98d6d-xbsxw_b6e0ab79-e56f-453b-8159-97d62ccbeb59/manager/0.log" Oct 30 17:34:34.205344 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:34.205315 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-x58sz_07f6b47b-677e-416a-a1d4-c65c81916e00/revoke-keys/0.log" Oct 30 17:34:34.393340 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:34.393307 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-7qgkc_9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4/revoke-keys/0.log" Oct 30 17:34:34.575113 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:34.575084 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-jvcxz_145ed570-31e7-420a-bafa-eff4e2b08ab1/revoke-keys/0.log" Oct 30 17:34:35.365726 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:35.365694 6202 scope.go:122] "RemoveContainer" containerID="a9c8bcd201aac8a5fbf7fce30a20c779185ac6d83c5f607081ffaa6a19bdddef" Oct 30 17:34:35.374105 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:35.374085 6202 scope.go:122] "RemoveContainer" containerID="ca5ca7d9ad1d7b38b48eb006896f54e6f02b089e8e7db85356a68cd1a8ec04f1" Oct 30 17:34:44.067046 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:34:44.067006 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:34:44.766279 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:44.766251 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-7dt2z_9940dbed-c8d8-40d4-8baf-c1ed02b71622/revoke-keys/0.log" Oct 30 17:34:44.956023 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:44.955994 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-rk2gm_14f3e4af-fd5a-4a61-9e4b-e521f3287456/revoke-keys/0.log" Oct 30 17:34:45.061285 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:45.061198 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:45.155460 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:45.155430 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fef6b280-cb7nm_c93793c1-875c-4c81-a727-e462823b1d35/revoke-keys/0.log" Oct 30 17:34:45.385451 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:45.385418 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:56.751542 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:56.751507 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-nsfnw_118b3526-1a2c-42dc-9061-310ccd01d554/manager/0.log" Oct 30 17:34:56.950065 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:56.950039 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-nzbcq_34745160-9b04-4309-bb7b-87cb8f4aca06/manager/0.log" Oct 30 17:34:58.525602 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:58.525568 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-hz72n_f39a5526-1dde-4513-a5ad-0c29de3b0fd0/discovery/0.log" Oct 30 17:34:58.732126 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:58.732098 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54d485b749-54hv6_785b392b-2225-4f7c-97dd-74ba17073238/kube-auth-proxy/0.log" Oct 30 17:34:59.312223 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:59.312192 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-5mcb6_04a034df-8dc0-489b-a623-4bb9e730e6ad/istio-proxy/0.log" Oct 30 17:34:59.925418 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:34:59.925389 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-ffkqn_27b9c1c4-c3da-4caf-a594-b391b48cc93a/istio-proxy/0.log" Oct 30 17:35:00.717445 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:35:00.717413 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-sm8wx_f4591587-01a3-4160-a8b7-d25450be9054/istio-proxy/0.log" Oct 30 17:35:01.116947 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:35:01.116921 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-57rw9_8a21ffcf-b1ed-4709-934e-f88b301ea0f4/payload-pre-processing/0.log" Oct 30 17:35:02.222055 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:35:02.222008 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-tbc5m_0c7c579a-a559-4424-b23b-cc7b91f61532/main/0.log" Oct 30 17:35:02.425672 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:35:02.425614 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-4kdj8_6e7bd43f-a7fb-449d-bfe3-a07c2aff68a4/main/0.log" Oct 30 17:35:13.195601 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:35:13.195562 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-7zzz8_9702f62c-b2ea-47b1-9e67-d50be44e166e/main/0.log" Oct 30 17:35:14.066860 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:35:14.066817 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:35:35.395892 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:35:35.395856 6202 scope.go:122] "RemoveContainer" containerID="4fce91254af1a03a1092e90cbd3e1c480d2cf9b5527e96ec38023384bfe34bd9" Oct 30 17:35:35.404293 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:35:35.404272 6202 scope.go:122] "RemoveContainer" containerID="43a296a4ae1b86b70d07018f0d97cfc44184d86c4ee3a040d887c2fd7da81753" Oct 30 17:35:40.028040 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:35:40.028005 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz"] Oct 30 17:35:40.039690 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:35:40.039662 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-jvcxz"] Oct 30 17:35:40.407754 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:35:40.407716 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="145ed570-31e7-420a-bafa-eff4e2b08ab1" path="/var/lib/kubelet/pods/145ed570-31e7-420a-bafa-eff4e2b08ab1/volumes" Oct 30 17:35:44.066590 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:35:44.066553 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:35:45.170876 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:35:45.170840 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:58.026251 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:35:58.026218 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc"] Oct 30 17:35:58.037681 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:35:58.037649 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-7qgkc"] Oct 30 17:35:58.408024 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:35:58.407990 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4" path="/var/lib/kubelet/pods/9d7dd8fb-2be3-4c77-bed0-ed7c26ccebc4/volumes" Oct 30 17:35:59.921100 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:35:59.921060 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:14.066863 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:36:14.066822 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:36:23.027010 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:36:23.026967 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm"] Oct 30 17:36:23.037974 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:36:23.037941 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef6b280-cb7nm"] Oct 30 17:36:24.408585 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:36:24.408547 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c93793c1-875c-4c81-a727-e462823b1d35" path="/var/lib/kubelet/pods/c93793c1-875c-4c81-a727-e462823b1d35/volumes" Oct 30 17:36:44.067786 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:36:44.067681 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:36:45.304729 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:36:45.304692 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:59.026550 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:36:59.026502 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm"] Oct 30 17:36:59.038071 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:36:59.038033 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rk2gm"] Oct 30 17:37:00.408807 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:37:00.408748 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14f3e4af-fd5a-4a61-9e4b-e521f3287456" path="/var/lib/kubelet/pods/14f3e4af-fd5a-4a61-9e4b-e521f3287456/volumes" Oct 30 17:37:00.802407 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:37:00.802321 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:00.914505 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:37:00.914471 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:14.067107 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:37:14.067056 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:37:17.026647 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:37:17.026606 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz"] Oct 30 17:37:17.038437 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:37:17.038406 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x58sz"] Oct 30 17:37:18.408842 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:37:18.408810 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f6b47b-677e-416a-a1d4-c65c81916e00" path="/var/lib/kubelet/pods/07f6b47b-677e-416a-a1d4-c65c81916e00/volumes" Oct 30 17:37:28.028821 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:37:28.028758 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z"] Oct 30 17:37:28.040524 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:37:28.040472 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7dt2z"] Oct 30 17:37:28.407978 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:37:28.407944 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9940dbed-c8d8-40d4-8baf-c1ed02b71622" path="/var/lib/kubelet/pods/9940dbed-c8d8-40d4-8baf-c1ed02b71622/volumes" Oct 30 17:37:35.433565 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:37:35.433532 6202 scope.go:122] "RemoveContainer" containerID="6eff159dbb913eb20273a0653e60fa855d17d6f96339a1eb41a8e2701839f7b9" Oct 30 17:37:35.442214 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:37:35.442194 6202 scope.go:122] "RemoveContainer" containerID="15fe8a0549f38bf06b424dc6ba8e10ca2bd6f529d0517b5ca573d20a7bcd526c" Oct 30 17:37:35.449730 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:37:35.449712 6202 scope.go:122] "RemoveContainer" containerID="ba0aa73165d070b96e7694e5a5f250cf58382fdce15027b142c1fddaaad3d462" Oct 30 17:37:44.070302 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:37:44.070262 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:37:45.406640 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:37:45.406603 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:11.906561 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:11.906470 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6"] Oct 30 17:38:11.907168 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:11.906719 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" podUID="04a034df-8dc0-489b-a623-4bb9e730e6ad" containerName="istio-proxy" containerID="cri-o://7c24ac9e5690ea0e3856fa3b3dfc782f4c6563444bcda55e43fcb5a57a9c0276" gracePeriod=30 Oct 30 17:38:11.918599 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:11.918536 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn"] Oct 30 17:38:11.918868 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:11.918844 6202 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" podUID="27b9c1c4-c3da-4caf-a594-b391b48cc93a" containerName="istio-proxy" containerID="cri-o://95d5b5c0030e559bb54ce0010cfec3dfd5b13e18e15e5f364b44c9adefdcba16" gracePeriod=30 Oct 30 17:38:14.067644 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:38:14.067605 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:38:17.170532 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.170495 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:38:17.197630 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.197578 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:38:17.332342 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332304 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-data\") pod \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " Oct 30 17:38:17.332549 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332351 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-data\") pod \"04a034df-8dc0-489b-a623-4bb9e730e6ad\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " Oct 30 17:38:17.332549 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332371 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-workload-certs\") pod \"04a034df-8dc0-489b-a623-4bb9e730e6ad\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " Oct 30 17:38:17.332549 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332393 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-credential-socket\") pod \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " Oct 30 17:38:17.332549 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332435 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a034df-8dc0-489b-a623-4bb9e730e6ad-kube-api-access-mkpw8\" (UniqueName: \"kubernetes.io/projected/04a034df-8dc0-489b-a623-4bb9e730e6ad-kube-api-access-mkpw8\") pod \"04a034df-8dc0-489b-a623-4bb9e730e6ad\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " Oct 30 17:38:17.332549 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332454 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-workload-certs\") pod \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " Oct 30 17:38:17.332549 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332500 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-token\" (UniqueName: \"kubernetes.io/projected/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-token\") pod \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " Oct 30 17:38:17.332549 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332548 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-envoy\") pod \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " Oct 30 17:38:17.333025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332583 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-workload-socket\") pod \"04a034df-8dc0-489b-a623-4bb9e730e6ad\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " Oct 30 17:38:17.333025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332632 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-workload-socket\") pod \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " Oct 30 17:38:17.333025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332650 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-data" pod "04a034df-8dc0-489b-a623-4bb9e730e6ad" (UID: "04a034df-8dc0-489b-a623-4bb9e730e6ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.333025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332659 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-data" pod "27b9c1c4-c3da-4caf-a594-b391b48cc93a" (UID: "27b9c1c4-c3da-4caf-a594-b391b48cc93a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.333025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332667 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-workload-certs" pod "04a034df-8dc0-489b-a623-4bb9e730e6ad" (UID: "04a034df-8dc0-489b-a623-4bb9e730e6ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.333025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332667 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istiod-ca-cert\") pod \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " Oct 30 17:38:17.333025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332753 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a034df-8dc0-489b-a623-4bb9e730e6ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a034df-8dc0-489b-a623-4bb9e730e6ad-istiod-ca-cert\") pod \"04a034df-8dc0-489b-a623-4bb9e730e6ad\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " Oct 30 17:38:17.333025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332801 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-credential-socket" pod "27b9c1c4-c3da-4caf-a594-b391b48cc93a" (UID: "27b9c1c4-c3da-4caf-a594-b391b48cc93a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.333025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332843 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-podinfo\") pod \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " Oct 30 17:38:17.333025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332873 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-podinfo\") pod \"04a034df-8dc0-489b-a623-4bb9e730e6ad\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " Oct 30 17:38:17.333025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332848 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-workload-certs" pod "27b9c1c4-c3da-4caf-a594-b391b48cc93a" (UID: "27b9c1c4-c3da-4caf-a594-b391b48cc93a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.333025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332907 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27b9c1c4-c3da-4caf-a594-b391b48cc93a-kube-api-access-xvbjv\" (UniqueName: \"kubernetes.io/projected/27b9c1c4-c3da-4caf-a594-b391b48cc93a-kube-api-access-xvbjv\") pod \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\" (UID: \"27b9c1c4-c3da-4caf-a594-b391b48cc93a\") " Oct 30 17:38:17.333025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332936 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-token\" (UniqueName: \"kubernetes.io/projected/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-token\") pod \"04a034df-8dc0-489b-a623-4bb9e730e6ad\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " Oct 30 17:38:17.333025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332971 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-credential-socket\") pod \"04a034df-8dc0-489b-a623-4bb9e730e6ad\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " Oct 30 17:38:17.333025 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.333011 6202 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-envoy\") pod \"04a034df-8dc0-489b-a623-4bb9e730e6ad\" (UID: \"04a034df-8dc0-489b-a623-4bb9e730e6ad\") " Oct 30 17:38:17.334090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.333568 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.334090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.333590 6202 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-data\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.334090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.333604 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.334090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.333618 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.334090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.333634 6202 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-workload-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.334090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.332934 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-workload-socket" pod "04a034df-8dc0-489b-a623-4bb9e730e6ad" (UID: "04a034df-8dc0-489b-a623-4bb9e730e6ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.334090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.333045 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istiod-ca-cert" pod "27b9c1c4-c3da-4caf-a594-b391b48cc93a" (UID: "27b9c1c4-c3da-4caf-a594-b391b48cc93a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:17.334090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.333232 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a034df-8dc0-489b-a623-4bb9e730e6ad-istiod-ca-cert" pod "04a034df-8dc0-489b-a623-4bb9e730e6ad" (UID: "04a034df-8dc0-489b-a623-4bb9e730e6ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:17.334090 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.333320 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-workload-socket" pod "27b9c1c4-c3da-4caf-a594-b391b48cc93a" (UID: "27b9c1c4-c3da-4caf-a594-b391b48cc93a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.335639 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.335593 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-credential-socket" pod "04a034df-8dc0-489b-a623-4bb9e730e6ad" (UID: "04a034df-8dc0-489b-a623-4bb9e730e6ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.335816 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.335694 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-token" pod "04a034df-8dc0-489b-a623-4bb9e730e6ad" (UID: "04a034df-8dc0-489b-a623-4bb9e730e6ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:17.336074 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.335875 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-envoy" pod "27b9c1c4-c3da-4caf-a594-b391b48cc93a" (UID: "27b9c1c4-c3da-4caf-a594-b391b48cc93a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.336074 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.335881 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-token" pod "27b9c1c4-c3da-4caf-a594-b391b48cc93a" (UID: "27b9c1c4-c3da-4caf-a594-b391b48cc93a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:17.336224 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.336084 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-podinfo" pod "04a034df-8dc0-489b-a623-4bb9e730e6ad" (UID: "04a034df-8dc0-489b-a623-4bb9e730e6ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:38:17.336649 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.336624 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-envoy" pod "04a034df-8dc0-489b-a623-4bb9e730e6ad" (UID: "04a034df-8dc0-489b-a623-4bb9e730e6ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.336649 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.336634 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-podinfo" pod "27b9c1c4-c3da-4caf-a594-b391b48cc93a" (UID: "27b9c1c4-c3da-4caf-a594-b391b48cc93a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:38:17.336938 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.336655 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a034df-8dc0-489b-a623-4bb9e730e6ad-kube-api-access-mkpw8" pod "04a034df-8dc0-489b-a623-4bb9e730e6ad" (UID: "04a034df-8dc0-489b-a623-4bb9e730e6ad"). InnerVolumeSpecName "kube-api-access-mkpw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:17.337469 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.337449 6202 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b9c1c4-c3da-4caf-a594-b391b48cc93a-kube-api-access-xvbjv" pod "27b9c1c4-c3da-4caf-a594-b391b48cc93a" (UID: "27b9c1c4-c3da-4caf-a594-b391b48cc93a"). InnerVolumeSpecName "kube-api-access-xvbjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:17.435156 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.435120 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.435156 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.435153 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.435156 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.435163 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.435156 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.435172 6202 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9c1c4-c3da-4caf-a594-b391b48cc93a-workload-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.435156 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.435182 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.435565 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.435190 6202 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a034df-8dc0-489b-a623-4bb9e730e6ad-istiod-ca-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.435565 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.435199 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b9c1c4-c3da-4caf-a594-b391b48cc93a-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.435565 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.435206 6202 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-podinfo\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.435565 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.435218 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvbjv\" (UniqueName: \"kubernetes.io/projected/27b9c1c4-c3da-4caf-a594-b391b48cc93a-kube-api-access-xvbjv\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.435565 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.435231 6202 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.435565 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.435239 6202 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-credential-socket\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.435565 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.435248 6202 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a034df-8dc0-489b-a623-4bb9e730e6ad-istio-envoy\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.435565 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.435257 6202 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkpw8\" (UniqueName: \"kubernetes.io/projected/04a034df-8dc0-489b-a623-4bb9e730e6ad-kube-api-access-mkpw8\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.529274 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.529181 6202 generic.go:356] "Generic (PLEG): container finished" podID="04a034df-8dc0-489b-a623-4bb9e730e6ad" containerID="7c24ac9e5690ea0e3856fa3b3dfc782f4c6563444bcda55e43fcb5a57a9c0276" exitCode=0 Oct 30 17:38:17.529274 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.529231 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" Oct 30 17:38:17.529274 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.529263 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" event={"ID":"04a034df-8dc0-489b-a623-4bb9e730e6ad","Type":"ContainerDied","Data":"7c24ac9e5690ea0e3856fa3b3dfc782f4c6563444bcda55e43fcb5a57a9c0276"} Oct 30 17:38:17.529546 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.529317 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6" event={"ID":"04a034df-8dc0-489b-a623-4bb9e730e6ad","Type":"ContainerDied","Data":"f1386fed6fda77cd93ff0cf0378dfc1dde4dd2d34020bafe0d19af5fb598297a"} Oct 30 17:38:17.529546 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.529347 6202 scope.go:122] "RemoveContainer" containerID="7c24ac9e5690ea0e3856fa3b3dfc782f4c6563444bcda55e43fcb5a57a9c0276" Oct 30 17:38:17.530802 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.530757 6202 generic.go:356] "Generic (PLEG): container finished" podID="27b9c1c4-c3da-4caf-a594-b391b48cc93a" containerID="95d5b5c0030e559bb54ce0010cfec3dfd5b13e18e15e5f364b44c9adefdcba16" exitCode=0 Oct 30 17:38:17.530939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.530807 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" event={"ID":"27b9c1c4-c3da-4caf-a594-b391b48cc93a","Type":"ContainerDied","Data":"95d5b5c0030e559bb54ce0010cfec3dfd5b13e18e15e5f364b44c9adefdcba16"} Oct 30 17:38:17.530939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.530824 6202 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" Oct 30 17:38:17.530939 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.530856 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn" event={"ID":"27b9c1c4-c3da-4caf-a594-b391b48cc93a","Type":"ContainerDied","Data":"d72848ce6c064a7adace945d06aaee9cfd3e16ac9f6936c4ff2c99b7d346de67"} Oct 30 17:38:17.540566 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.540543 6202 scope.go:122] "RemoveContainer" containerID="7c24ac9e5690ea0e3856fa3b3dfc782f4c6563444bcda55e43fcb5a57a9c0276" Oct 30 17:38:17.541047 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:38:17.540980 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c24ac9e5690ea0e3856fa3b3dfc782f4c6563444bcda55e43fcb5a57a9c0276\": container with ID starting with 7c24ac9e5690ea0e3856fa3b3dfc782f4c6563444bcda55e43fcb5a57a9c0276 not found: ID does not exist" containerID="7c24ac9e5690ea0e3856fa3b3dfc782f4c6563444bcda55e43fcb5a57a9c0276" Oct 30 17:38:17.541047 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.541024 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c24ac9e5690ea0e3856fa3b3dfc782f4c6563444bcda55e43fcb5a57a9c0276"} err="failed to get container status \"7c24ac9e5690ea0e3856fa3b3dfc782f4c6563444bcda55e43fcb5a57a9c0276\": rpc error: code = NotFound desc = could not find container \"7c24ac9e5690ea0e3856fa3b3dfc782f4c6563444bcda55e43fcb5a57a9c0276\": container with ID starting with 7c24ac9e5690ea0e3856fa3b3dfc782f4c6563444bcda55e43fcb5a57a9c0276 not found: ID does not exist" Oct 30 17:38:17.541047 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.541044 6202 scope.go:122] "RemoveContainer" containerID="95d5b5c0030e559bb54ce0010cfec3dfd5b13e18e15e5f364b44c9adefdcba16" Oct 30 17:38:17.549702 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.549686 6202 scope.go:122] "RemoveContainer" containerID="95d5b5c0030e559bb54ce0010cfec3dfd5b13e18e15e5f364b44c9adefdcba16" Oct 30 17:38:17.549992 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:38:17.549973 6202 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d5b5c0030e559bb54ce0010cfec3dfd5b13e18e15e5f364b44c9adefdcba16\": container with ID starting with 95d5b5c0030e559bb54ce0010cfec3dfd5b13e18e15e5f364b44c9adefdcba16 not found: ID does not exist" containerID="95d5b5c0030e559bb54ce0010cfec3dfd5b13e18e15e5f364b44c9adefdcba16" Oct 30 17:38:17.550061 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.549999 6202 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d5b5c0030e559bb54ce0010cfec3dfd5b13e18e15e5f364b44c9adefdcba16"} err="failed to get container status \"95d5b5c0030e559bb54ce0010cfec3dfd5b13e18e15e5f364b44c9adefdcba16\": rpc error: code = NotFound desc = could not find container \"95d5b5c0030e559bb54ce0010cfec3dfd5b13e18e15e5f364b44c9adefdcba16\": container with ID starting with 95d5b5c0030e559bb54ce0010cfec3dfd5b13e18e15e5f364b44c9adefdcba16 not found: ID does not exist" Oct 30 17:38:17.580276 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.580244 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6"] Oct 30 17:38:17.591845 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.591823 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5mcb6"] Oct 30 17:38:17.619262 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.619220 6202 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn"] Oct 30 17:38:17.624258 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:17.624236 6202 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ffkqn"] Oct 30 17:38:18.409419 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:18.409384 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a034df-8dc0-489b-a623-4bb9e730e6ad" path="/var/lib/kubelet/pods/04a034df-8dc0-489b-a623-4bb9e730e6ad/volumes" Oct 30 17:38:18.409865 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:18.409847 6202 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27b9c1c4-c3da-4caf-a594-b391b48cc93a" path="/var/lib/kubelet/pods/27b9c1c4-c3da-4caf-a594-b391b48cc93a/volumes" Oct 30 17:38:33.146316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:33.146277 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:35.474618 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:35.474583 6202 scope.go:122] "RemoveContainer" containerID="3c2cea5a90d30b7cf11d64cb9f17e4c45df22bf329f73bad42c9badff4a9fdf5" Oct 30 17:38:35.485110 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:35.485092 6202 scope.go:122] "RemoveContainer" containerID="07a05667cad7e7a2d0e5656dffd6de93f021fe22c49acd733fabc5397d09d5df" Oct 30 17:38:35.492722 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:35.492702 6202 scope.go:122] "RemoveContainer" containerID="af0907f7c73b468b49967b76d4f987922f8eaf1c367d33fcc0dce0177837e1d1" Oct 30 17:38:44.068595 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:38:44.068553 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor" Oct 30 17:38:44.540656 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:44.540620 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:44.663831 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:44.663797 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:45.548469 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:45.548434 6202 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:49.070432 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:49.070398 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-k4k2w_ffdd3593-6091-4874-b55e-4049aa26ab60/global-pull-secret-syncer/0.log" Oct 30 17:38:49.825644 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:49.825610 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4n84w_32528057-07fa-4eb9-882f-a43c9cd36b31/konnectivity-agent/0.log" Oct 30 17:38:50.331974 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:50.331926 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-65.ec2.internal_829ec3bbd4f4a9d896bac875cd5fa87a/haproxy/0.log" Oct 30 17:38:56.852188 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:56.852155 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-nsfnw_118b3526-1a2c-42dc-9061-310ccd01d554/manager/0.log" Oct 30 17:38:56.969068 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:56.969025 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-nzbcq_34745160-9b04-4309-bb7b-87cb8f4aca06/manager/0.log" Oct 30 17:38:58.538850 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.538811 6202 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms"] Oct 30 17:38:58.539313 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.539296 6202 cpu_manager.go:414] "containerMap: removing container" podUID="9940dbed-c8d8-40d4-8baf-c1ed02b71622" containerName="revoke-keys" Oct 30 17:38:58.539359 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.539316 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9940dbed-c8d8-40d4-8baf-c1ed02b71622" containerName="revoke-keys" Oct 30 17:38:58.539359 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.539351 6202 cpu_manager.go:414] "containerMap: removing container" podUID="27b9c1c4-c3da-4caf-a594-b391b48cc93a" containerName="istio-proxy" Oct 30 17:38:58.539425 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.539358 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27b9c1c4-c3da-4caf-a594-b391b48cc93a" containerName="istio-proxy" Oct 30 17:38:58.539425 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.539367 6202 cpu_manager.go:414] "containerMap: removing container" podUID="04a034df-8dc0-489b-a623-4bb9e730e6ad" containerName="istio-proxy" Oct 30 17:38:58.539425 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.539374 6202 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a034df-8dc0-489b-a623-4bb9e730e6ad" containerName="istio-proxy" Oct 30 17:38:58.539527 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.539459 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="9940dbed-c8d8-40d4-8baf-c1ed02b71622" containerName="revoke-keys" Oct 30 17:38:58.539527 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.539474 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="27b9c1c4-c3da-4caf-a594-b391b48cc93a" containerName="istio-proxy" Oct 30 17:38:58.539527 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.539483 6202 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a034df-8dc0-489b-a623-4bb9e730e6ad" containerName="istio-proxy" Oct 30 17:38:58.544316 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.544295 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:38:58.562039 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.562017 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms"] Oct 30 17:38:58.649507 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.649461 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vstlf\"/\"kube-root-ca.crt\"" Oct 30 17:38:58.649670 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.649516 6202 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vstlf\"/\"openshift-service-ca.crt\"" Oct 30 17:38:58.649670 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.649630 6202 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vstlf\"/\"default-dockercfg-mtr8p\"" Oct 30 17:38:58.711494 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.711454 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1da587b2-5d26-4dff-b0ff-8695ec82603d-podres\") pod \"perf-node-gather-daemonset-tdvms\" (UID: \"1da587b2-5d26-4dff-b0ff-8695ec82603d\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:38:58.711670 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.711586 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1da587b2-5d26-4dff-b0ff-8695ec82603d-sys\") pod \"perf-node-gather-daemonset-tdvms\" (UID: \"1da587b2-5d26-4dff-b0ff-8695ec82603d\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:38:58.711670 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.711663 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1da587b2-5d26-4dff-b0ff-8695ec82603d-proc\") pod \"perf-node-gather-daemonset-tdvms\" (UID: \"1da587b2-5d26-4dff-b0ff-8695ec82603d\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:38:58.711826 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.711693 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxt8r\" (UniqueName: \"kubernetes.io/projected/1da587b2-5d26-4dff-b0ff-8695ec82603d-kube-api-access-hxt8r\") pod \"perf-node-gather-daemonset-tdvms\" (UID: \"1da587b2-5d26-4dff-b0ff-8695ec82603d\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:38:58.711826 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.711748 6202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1da587b2-5d26-4dff-b0ff-8695ec82603d-lib-modules\") pod \"perf-node-gather-daemonset-tdvms\" (UID: \"1da587b2-5d26-4dff-b0ff-8695ec82603d\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:38:58.812985 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.812901 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1da587b2-5d26-4dff-b0ff-8695ec82603d-proc\") pod \"perf-node-gather-daemonset-tdvms\" (UID: \"1da587b2-5d26-4dff-b0ff-8695ec82603d\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:38:58.812985 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.812948 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxt8r\" (UniqueName: \"kubernetes.io/projected/1da587b2-5d26-4dff-b0ff-8695ec82603d-kube-api-access-hxt8r\") pod \"perf-node-gather-daemonset-tdvms\" (UID: \"1da587b2-5d26-4dff-b0ff-8695ec82603d\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:38:58.813245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.812995 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1da587b2-5d26-4dff-b0ff-8695ec82603d-lib-modules\") pod \"perf-node-gather-daemonset-tdvms\" (UID: \"1da587b2-5d26-4dff-b0ff-8695ec82603d\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:38:58.813245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.813042 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1da587b2-5d26-4dff-b0ff-8695ec82603d-proc\") pod \"perf-node-gather-daemonset-tdvms\" (UID: \"1da587b2-5d26-4dff-b0ff-8695ec82603d\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:38:58.813245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.813079 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1da587b2-5d26-4dff-b0ff-8695ec82603d-podres\") pod \"perf-node-gather-daemonset-tdvms\" (UID: \"1da587b2-5d26-4dff-b0ff-8695ec82603d\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:38:58.813245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.813106 6202 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1da587b2-5d26-4dff-b0ff-8695ec82603d-sys\") pod \"perf-node-gather-daemonset-tdvms\" (UID: \"1da587b2-5d26-4dff-b0ff-8695ec82603d\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:38:58.813245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.813126 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1da587b2-5d26-4dff-b0ff-8695ec82603d-lib-modules\") pod \"perf-node-gather-daemonset-tdvms\" (UID: \"1da587b2-5d26-4dff-b0ff-8695ec82603d\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:38:58.813245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.813221 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1da587b2-5d26-4dff-b0ff-8695ec82603d-sys\") pod \"perf-node-gather-daemonset-tdvms\" (UID: \"1da587b2-5d26-4dff-b0ff-8695ec82603d\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:38:58.813245 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.813220 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1da587b2-5d26-4dff-b0ff-8695ec82603d-podres\") pod \"perf-node-gather-daemonset-tdvms\" (UID: \"1da587b2-5d26-4dff-b0ff-8695ec82603d\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:38:58.836558 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.836534 6202 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxt8r\" (UniqueName: \"kubernetes.io/projected/1da587b2-5d26-4dff-b0ff-8695ec82603d-kube-api-access-hxt8r\") pod \"perf-node-gather-daemonset-tdvms\" (UID: \"1da587b2-5d26-4dff-b0ff-8695ec82603d\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:38:58.856618 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:58.856598 6202 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:38:59.148222 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:59.148191 6202 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms"] Oct 30 17:38:59.150258 ip-10-0-129-65 kubenswrapper[6202]: W1030 17:38:59.150227 6202 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1da587b2_5d26_4dff_b0ff_8695ec82603d.slice/crio-aa0971a3bf016a6b0908f71da35b3ead44df8a75df21aa6a7bf49c22b60f7f84 WatchSource:0}: Error finding container aa0971a3bf016a6b0908f71da35b3ead44df8a75df21aa6a7bf49c22b60f7f84: Status 404 returned error can't find the container with id aa0971a3bf016a6b0908f71da35b3ead44df8a75df21aa6a7bf49c22b60f7f84 Oct 30 17:38:59.720694 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:59.720658 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" event={"ID":"1da587b2-5d26-4dff-b0ff-8695ec82603d","Type":"ContainerStarted","Data":"e582ebe94f71800ef2a6e2c9373e97faef83934441d8f33aa4ab4c0f588e414a"} Oct 30 17:38:59.721162 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:59.720709 6202 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" event={"ID":"1da587b2-5d26-4dff-b0ff-8695ec82603d","Type":"ContainerStarted","Data":"aa0971a3bf016a6b0908f71da35b3ead44df8a75df21aa6a7bf49c22b60f7f84"} Oct 30 17:38:59.721162 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:59.720817 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:38:59.761147 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:38:59.761095 6202 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" podStartSLOduration=1.761082398 podStartE2EDuration="1.761082398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:38:59.759985986 +0000 UTC m=+4653.470440723" watchObservedRunningTime="2026-10-30 17:38:59.761082398 +0000 UTC m=+4653.471537141" Oct 30 17:39:00.434401 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:00.434371 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dktqn_9031958e-b627-4334-9e47-b9b62862c805/node-exporter/0.log" Oct 30 17:39:00.546660 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:00.546635 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dktqn_9031958e-b627-4334-9e47-b9b62862c805/kube-rbac-proxy/0.log" Oct 30 17:39:00.658551 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:00.658527 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dktqn_9031958e-b627-4334-9e47-b9b62862c805/init-textfile/0.log" Oct 30 17:39:01.863591 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:01.863565 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45db0ed9-af63-4d3f-a393-aacbf2e6ad58/prometheus/0.log" Oct 30 17:39:01.981144 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:01.981103 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45db0ed9-af63-4d3f-a393-aacbf2e6ad58/config-reloader/0.log" Oct 30 17:39:02.102264 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:02.102232 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45db0ed9-af63-4d3f-a393-aacbf2e6ad58/thanos-sidecar/0.log" Oct 30 17:39:02.217677 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:02.217601 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45db0ed9-af63-4d3f-a393-aacbf2e6ad58/kube-rbac-proxy-web/0.log" Oct 30 17:39:02.331826 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:02.331798 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45db0ed9-af63-4d3f-a393-aacbf2e6ad58/kube-rbac-proxy/0.log" Oct 30 17:39:02.448326 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:02.448280 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45db0ed9-af63-4d3f-a393-aacbf2e6ad58/kube-rbac-proxy-thanos/0.log" Oct 30 17:39:02.562012 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:02.561943 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45db0ed9-af63-4d3f-a393-aacbf2e6ad58/init-config-reloader/0.log" Oct 30 17:39:02.686109 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:02.686078 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf8hl_0dcde09b-ef11-4bb9-810b-883b0ee4cc18/prometheus-operator/0.log" Oct 30 17:39:02.794913 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:02.794886 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf8hl_0dcde09b-ef11-4bb9-810b-883b0ee4cc18/kube-rbac-proxy/0.log" Oct 30 17:39:03.031302 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:03.031274 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b5445f54d-42vtq_32474e2a-2206-4b53-a41e-6eaab7de8285/telemeter-client/0.log" Oct 30 17:39:03.145421 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:03.145401 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b5445f54d-42vtq_32474e2a-2206-4b53-a41e-6eaab7de8285/reload/0.log" Oct 30 17:39:03.260083 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:03.260054 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b5445f54d-42vtq_32474e2a-2206-4b53-a41e-6eaab7de8285/kube-rbac-proxy/0.log" Oct 30 17:39:03.382319 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:03.382279 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59c44d56bd-vwqs7_85bfc58a-5f99-41ad-a651-05ba06bc2be1/thanos-query/0.log" Oct 30 17:39:03.495144 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:03.495122 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59c44d56bd-vwqs7_85bfc58a-5f99-41ad-a651-05ba06bc2be1/kube-rbac-proxy-web/0.log" Oct 30 17:39:03.608193 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:03.608165 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59c44d56bd-vwqs7_85bfc58a-5f99-41ad-a651-05ba06bc2be1/kube-rbac-proxy/0.log" Oct 30 17:39:03.725650 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:03.725582 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59c44d56bd-vwqs7_85bfc58a-5f99-41ad-a651-05ba06bc2be1/prom-label-proxy/0.log" Oct 30 17:39:03.835564 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:03.835534 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59c44d56bd-vwqs7_85bfc58a-5f99-41ad-a651-05ba06bc2be1/kube-rbac-proxy-rules/0.log" Oct 30 17:39:03.946727 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:03.946701 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59c44d56bd-vwqs7_85bfc58a-5f99-41ad-a651-05ba06bc2be1/kube-rbac-proxy-metrics/0.log" Oct 30 17:39:05.735030 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:05.735003 6202 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-tdvms" Oct 30 17:39:11.479374 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:11.479346 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xrskd_86634960-264a-48a7-969a-5fbe6e58447e/dns/0.log" Oct 30 17:39:11.608640 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:11.608614 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xrskd_86634960-264a-48a7-969a-5fbe6e58447e/kube-rbac-proxy/0.log" Oct 30 17:39:11.957734 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:11.957706 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lr2cd_ca4b38a9-e2b5-43dd-ad2b-6ecf9a080ffc/dns-node-resolver/0.log" Oct 30 17:39:13.200082 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:13.200045 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7w7js_17726533-46eb-4d51-bc1d-b4e9959d7ba5/node-ca/0.log" Oct 30 17:39:13.680890 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:13.680843 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-hz72n_f39a5526-1dde-4513-a5ad-0c29de3b0fd0/discovery/0.log" Oct 30 17:39:13.754920 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:13.754871 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54d485b749-54hv6_785b392b-2225-4f7c-97dd-74ba17073238/kube-auth-proxy/0.log" Oct 30 17:39:13.878967 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:13.878937 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-sm8wx_f4591587-01a3-4160-a8b7-d25450be9054/istio-proxy/0.log" Oct 30 17:39:13.936550 ip-10-0-129-65 kubenswrapper[6202]: I1030 17:39:13.936480 6202 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-57rw9_8a21ffcf-b1ed-4709-934e-f88b301ea0f4/payload-pre-processing/0.log" Oct 30 17:39:14.067732 ip-10-0-129-65 kubenswrapper[6202]: E1030 17:39:14.067692 6202 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fd5ww" containerName="extractor"